Note: This is test shard 1 of 3.
[==========] Running 2 tests from 1 test suite.
[----------] Global test environment set-up.
[----------] 2 tests from CatalogManagerConfigurations/MasterStressTest
[ RUN ] CatalogManagerConfigurations/MasterStressTest.Test/0
2026-01-30T02:57:39Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-01-30T02:57:39Z Disabled control of system clock
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20260130 02:57:39.241080 18833 external_mini_cluster.cc:1366] Running /tmp/dist-test-tasktGe2wI/build/tsan/bin/kudu
/tmp/dist-test-tasktGe2wI/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.100.126:38191
--webserver_interface=127.18.100.126
--webserver_port=0
--builtin_ntp_servers=127.18.100.84:46033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.100.126:38191,127.18.100.125:39711,127.18.100.124:34629
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20260130 02:57:39.561183 18842 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260130 02:57:39.561892 18842 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260130 02:57:39.562378 18842 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260130 02:57:39.596119 18842 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260130 02:57:39.596467 18842 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260130 02:57:39.596741 18842 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260130 02:57:39.596989 18842 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260130 02:57:39.636327 18842 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.100.84:46033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/master-0/wal
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.18.100.126:38191,127.18.100.125:39711,127.18.100.124:34629
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.100.126:38191
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.100.126
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type FASTDEBUG
built by None at 30 Jan 2026 02:43:21 UTC on 5fd53c4cbb9d
build id 10257
TSAN enabled
I20260130 02:57:39.638001 18842 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260130 02:57:39.640020 18842 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260130 02:57:39.652254 18848 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260130 02:57:41.055655 18847 debug-util.cc:398] Leaking SignalData structure 0x7b080002ec60 after lost signal to thread 18842
W20260130 02:57:41.191669 18847 kernel_stack_watchdog.cc:198] Thread 18842 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:642 for 400ms:
Kernel stack:
(could not read kernel stack)
User stack:
<Timed out: thread did not respond: maybe it is blocking signals>
W20260130 02:57:39.652750 18849 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260130 02:57:41.197178 18842 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.545s user 0.000s sys 0.006s
W20260130 02:57:41.197448 18850 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1540 milliseconds
W20260130 02:57:41.197477 18842 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.545s user 0.000s sys 0.006s
W20260130 02:57:41.199687 18851 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260130 02:57:41.199761 18842 server_base.cc:1055] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20260130 02:57:41.200963 18842 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260130 02:57:41.203881 18842 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260130 02:57:41.205314 18842 hybrid_clock.cc:648] HybridClock initialized: now 1769741861205272 us; error 54 us; skew 500 ppm
I20260130 02:57:41.206137 18842 server_base.cc:860] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260130 02:57:41.212219 18842 webserver.cc:492] Webserver started at http://127.18.100.126:35635/ using document root <none> and password file <none>
I20260130 02:57:41.213196 18842 fs_manager.cc:362] Metadata directory not provided
I20260130 02:57:41.213394 18842 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260130 02:57:41.213884 18842 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260130 02:57:41.218206 18842 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/master-0/data/instance:
uuid: "658299197a424f53a37f14fadf089ae4"
format_stamp: "Formatted at 2026-01-30 02:57:41 on dist-test-slave-mn07"
I20260130 02:57:41.219321 18842 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/master-0/wal/instance:
uuid: "658299197a424f53a37f14fadf089ae4"
format_stamp: "Formatted at 2026-01-30 02:57:41 on dist-test-slave-mn07"
I20260130 02:57:41.226657 18842 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.007s sys 0.001s
I20260130 02:57:41.232121 18858 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260130 02:57:41.233158 18842 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.005s sys 0.000s
I20260130 02:57:41.233462 18842 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/master-0/data,/tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/master-0/wal
uuid: "658299197a424f53a37f14fadf089ae4"
format_stamp: "Formatted at 2026-01-30 02:57:41 on dist-test-slave-mn07"
I20260130 02:57:41.233793 18842 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260130 02:57:41.283941 18842 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260130 02:57:41.285423 18842 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260130 02:57:41.285846 18842 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260130 02:57:41.352449 18842 rpc_server.cc:307] RPC server started. Bound to: 127.18.100.126:38191
I20260130 02:57:41.352519 18909 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.100.126:38191 every 8 connection(s)
I20260130 02:57:41.355120 18842 server_base.cc:1192] Dumped server information to /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/master-0/data/info.pb
I20260130 02:57:41.360963 18910 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:57:41.366092 18833 external_mini_cluster.cc:1428] Started /tmp/dist-test-tasktGe2wI/build/tsan/bin/kudu as pid 18842
I20260130 02:57:41.366688 18833 external_mini_cluster.cc:1442] Reading /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/master-0/wal/instance
I20260130 02:57:41.372188 18833 external_mini_cluster.cc:1366] Running /tmp/dist-test-tasktGe2wI/build/tsan/bin/kudu
/tmp/dist-test-tasktGe2wI/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.100.125:39711
--webserver_interface=127.18.100.125
--webserver_port=0
--builtin_ntp_servers=127.18.100.84:46033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.100.126:38191,127.18.100.125:39711,127.18.100.124:34629
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20260130 02:57:41.374890 18910 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.100.126" port: 38191 } has no permanent_uuid. Determining permanent_uuid...
I20260130 02:57:41.399736 18910 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39711 } has no permanent_uuid. Determining permanent_uuid...
W20260130 02:57:41.402108 18860 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.18.100.125:39711: connect: Connection refused (error 111)
W20260130 02:57:41.406054 18910 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.18.100.125:39711: Network error: Client connection negotiation failed: client connection to 127.18.100.125:39711: connect: Connection refused (error 111)
I20260130 02:57:41.455518 18910 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39711 } attempt: 1
W20260130 02:57:41.459530 18910 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.18.100.125:39711: Network error: Client connection negotiation failed: client connection to 127.18.100.125:39711: connect: Connection refused (error 111)
I20260130 02:57:41.527926 18910 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39711 } attempt: 2
W20260130 02:57:41.531849 18910 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.18.100.125:39711: Network error: Client connection negotiation failed: client connection to 127.18.100.125:39711: connect: Connection refused (error 111)
I20260130 02:57:41.623271 18910 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39711 } attempt: 3
W20260130 02:57:41.627094 18910 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.18.100.125:39711: Network error: Client connection negotiation failed: client connection to 127.18.100.125:39711: connect: Connection refused (error 111)
W20260130 02:57:41.673519 18912 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260130 02:57:41.674100 18912 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260130 02:57:41.674542 18912 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260130 02:57:41.705989 18912 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260130 02:57:41.706327 18912 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260130 02:57:41.706591 18912 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260130 02:57:41.706833 18912 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260130 02:57:41.742709 18912 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.100.84:46033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/master-1/wal
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.18.100.126:38191,127.18.100.125:39711,127.18.100.124:34629
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.100.125:39711
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.18.100.125
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type FASTDEBUG
built by None at 30 Jan 2026 02:43:21 UTC on 5fd53c4cbb9d
build id 10257
TSAN enabled
I20260130 02:57:41.744073 18912 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260130 02:57:41.745738 18912 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260130 02:57:41.756412 18921 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260130 02:57:41.770532 18910 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39711 } attempt: 4
W20260130 02:57:41.776779 18910 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.18.100.125:39711: Network error: Client connection negotiation failed: client connection to 127.18.100.125:39711: connect: Connection refused (error 111)
I20260130 02:57:42.076319 18910 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39711 } attempt: 5
W20260130 02:57:42.082432 18910 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.18.100.125:39711: Network error: Client connection negotiation failed: client connection to 127.18.100.125:39711: connect: Connection refused (error 111)
I20260130 02:57:42.629949 18910 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39711 } attempt: 6
W20260130 02:57:42.655630 18910 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.18.100.125:39711: Network error: Client connection negotiation failed: client connection to 127.18.100.125:39711: connect: Connection refused (error 111)
W20260130 02:57:41.757359 18922 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260130 02:57:42.953352 18923 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1195 milliseconds
W20260130 02:57:42.954159 18924 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260130 02:57:42.957034 18912 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.200s user 0.438s sys 0.757s
W20260130 02:57:42.957294 18912 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.201s user 0.438s sys 0.757s
I20260130 02:57:42.957499 18912 server_base.cc:1055] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20260130 02:57:42.958631 18912 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260130 02:57:42.961092 18912 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260130 02:57:42.962447 18912 hybrid_clock.cc:648] HybridClock initialized: now 1769741862962416 us; error 41 us; skew 500 ppm
I20260130 02:57:42.963366 18912 server_base.cc:860] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260130 02:57:42.970476 18912 webserver.cc:492] Webserver started at http://127.18.100.125:36343/ using document root <none> and password file <none>
I20260130 02:57:42.971455 18912 fs_manager.cc:362] Metadata directory not provided
I20260130 02:57:42.971675 18912 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260130 02:57:42.972122 18912 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260130 02:57:42.976467 18912 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/master-1/data/instance:
uuid: "107f274cdda543038832f06a56ab9130"
format_stamp: "Formatted at 2026-01-30 02:57:42 on dist-test-slave-mn07"
I20260130 02:57:42.977586 18912 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/master-1/wal/instance:
uuid: "107f274cdda543038832f06a56ab9130"
format_stamp: "Formatted at 2026-01-30 02:57:42 on dist-test-slave-mn07"
I20260130 02:57:42.985539 18912 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.009s sys 0.001s
I20260130 02:57:42.991693 18933 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260130 02:57:42.992951 18912 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260130 02:57:42.993286 18912 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/master-1/data,/tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/master-1/wal
uuid: "107f274cdda543038832f06a56ab9130"
format_stamp: "Formatted at 2026-01-30 02:57:42 on dist-test-slave-mn07"
I20260130 02:57:42.993628 18912 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260130 02:57:43.059253 18912 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260130 02:57:43.060737 18912 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260130 02:57:43.061291 18912 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260130 02:57:43.128187 18912 rpc_server.cc:307] RPC server started. Bound to: 127.18.100.125:39711
I20260130 02:57:43.128265 18984 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.100.125:39711 every 8 connection(s)
I20260130 02:57:43.130810 18912 server_base.cc:1192] Dumped server information to /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/master-1/data/info.pb
I20260130 02:57:43.136598 18985 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:57:43.140683 18833 external_mini_cluster.cc:1428] Started /tmp/dist-test-tasktGe2wI/build/tsan/bin/kudu as pid 18912
I20260130 02:57:43.141129 18833 external_mini_cluster.cc:1442] Reading /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/master-1/wal/instance
I20260130 02:57:43.144222 18833 external_mini_cluster.cc:1366] Running /tmp/dist-test-tasktGe2wI/build/tsan/bin/kudu
/tmp/dist-test-tasktGe2wI/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.100.124:34629
--webserver_interface=127.18.100.124
--webserver_port=0
--builtin_ntp_servers=127.18.100.84:46033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.100.126:38191,127.18.100.125:39711,127.18.100.124:34629
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20260130 02:57:43.150105 18985 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.100.126" port: 38191 } has no permanent_uuid. Determining permanent_uuid...
I20260130 02:57:43.175359 18985 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39711 } has no permanent_uuid. Determining permanent_uuid...
I20260130 02:57:43.189437 18985 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34629 } has no permanent_uuid. Determining permanent_uuid...
W20260130 02:57:43.191607 18935 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.18.100.124:34629: connect: Connection refused (error 111)
W20260130 02:57:43.195418 18985 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.18.100.124:34629: Network error: Client connection negotiation failed: client connection to 127.18.100.124:34629: connect: Connection refused (error 111)
I20260130 02:57:43.245419 18985 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34629 } attempt: 1
W20260130 02:57:43.249320 18985 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.18.100.124:34629: Network error: Client connection negotiation failed: client connection to 127.18.100.124:34629: connect: Connection refused (error 111)
I20260130 02:57:43.317817 18985 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34629 } attempt: 2
W20260130 02:57:43.321743 18985 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.18.100.124:34629: Network error: Client connection negotiation failed: client connection to 127.18.100.124:34629: connect: Connection refused (error 111)
I20260130 02:57:43.413239 18985 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34629 } attempt: 3
W20260130 02:57:43.417131 18985 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.18.100.124:34629: Network error: Client connection negotiation failed: client connection to 127.18.100.124:34629: connect: Connection refused (error 111)
W20260130 02:57:43.445490 18987 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260130 02:57:43.446033 18987 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260130 02:57:43.446422 18987 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260130 02:57:43.478639 18987 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260130 02:57:43.478933 18987 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260130 02:57:43.479130 18987 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260130 02:57:43.479321 18987 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260130 02:57:43.514820 18987 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.100.84:46033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/master-2/wal
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.18.100.126:38191,127.18.100.125:39711,127.18.100.124:34629
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.100.124:34629
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.18.100.124
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type FASTDEBUG
built by None at 30 Jan 2026 02:43:21 UTC on 5fd53c4cbb9d
build id 10257
TSAN enabled
I20260130 02:57:43.516284 18987 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260130 02:57:43.517849 18987 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260130 02:57:43.528151 18997 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260130 02:57:43.560619 18985 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34629 } attempt: 4
W20260130 02:57:43.567201 18985 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.18.100.124:34629: Network error: Client connection negotiation failed: client connection to 127.18.100.124:34629: connect: Connection refused (error 111)
I20260130 02:57:43.716151 18910 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39711 } attempt: 7
I20260130 02:57:43.748385 18910 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34629 } has no permanent_uuid. Determining permanent_uuid...
W20260130 02:57:43.774617 18910 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.18.100.124:34629: Network error: Client connection negotiation failed: client connection to 127.18.100.124:34629: connect: Connection refused (error 111)
I20260130 02:57:43.833148 18910 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34629 } attempt: 1
W20260130 02:57:43.840808 18910 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.18.100.124:34629: Network error: Client connection negotiation failed: client connection to 127.18.100.124:34629: connect: Connection refused (error 111)
I20260130 02:57:43.866675 18985 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34629 } attempt: 5
W20260130 02:57:43.872804 18985 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.18.100.124:34629: Network error: Client connection negotiation failed: client connection to 127.18.100.124:34629: connect: Connection refused (error 111)
I20260130 02:57:43.922318 18910 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34629 } attempt: 2
W20260130 02:57:43.928297 18910 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.18.100.124:34629: Network error: Client connection negotiation failed: client connection to 127.18.100.124:34629: connect: Connection refused (error 111)
I20260130 02:57:44.013792 18910 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34629 } attempt: 3
W20260130 02:57:44.019728 18910 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.18.100.124:34629: Network error: Client connection negotiation failed: client connection to 127.18.100.124:34629: connect: Connection refused (error 111)
I20260130 02:57:44.160228 18910 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34629 } attempt: 4
W20260130 02:57:44.166627 18910 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.18.100.124:34629: Network error: Client connection negotiation failed: client connection to 127.18.100.124:34629: connect: Connection refused (error 111)
I20260130 02:57:44.420275 18985 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34629 } attempt: 6
W20260130 02:57:44.448493 18985 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.18.100.124:34629: Network error: Client connection negotiation failed: client connection to 127.18.100.124:34629: connect: Connection refused (error 111)
I20260130 02:57:44.450155 18910 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34629 } attempt: 5
W20260130 02:57:44.455848 18910 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.18.100.124:34629: Network error: Client connection negotiation failed: client connection to 127.18.100.124:34629: connect: Connection refused (error 111)
W20260130 02:57:43.529065 18998 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260130 02:57:44.729786 18999 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1199 milliseconds
W20260130 02:57:44.730821 19000 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260130 02:57:44.733284 18987 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.205s user 0.478s sys 0.712s
W20260130 02:57:44.733672 18987 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.206s user 0.480s sys 0.714s
I20260130 02:57:44.733937 18987 server_base.cc:1055] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20260130 02:57:44.739840 18987 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260130 02:57:44.742492 18987 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260130 02:57:44.743913 18987 hybrid_clock.cc:648] HybridClock initialized: now 1769741864743838 us; error 93 us; skew 500 ppm
I20260130 02:57:44.744824 18987 server_base.cc:860] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260130 02:57:44.751750 18987 webserver.cc:492] Webserver started at http://127.18.100.124:37741/ using document root <none> and password file <none>
I20260130 02:57:44.752722 18987 fs_manager.cc:362] Metadata directory not provided
I20260130 02:57:44.752940 18987 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260130 02:57:44.753382 18987 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260130 02:57:44.757720 18987 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/master-2/data/instance:
uuid: "3749baa287ed4f249329a74071781edd"
format_stamp: "Formatted at 2026-01-30 02:57:44 on dist-test-slave-mn07"
I20260130 02:57:44.759037 18987 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/master-2/wal/instance:
uuid: "3749baa287ed4f249329a74071781edd"
format_stamp: "Formatted at 2026-01-30 02:57:44 on dist-test-slave-mn07"
I20260130 02:57:44.766688 18987 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.007s sys 0.000s
I20260130 02:57:44.773152 19013 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260130 02:57:44.774513 18987 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.003s sys 0.003s
I20260130 02:57:44.774899 18987 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/master-2/data,/tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/master-2/wal
uuid: "3749baa287ed4f249329a74071781edd"
format_stamp: "Formatted at 2026-01-30 02:57:44 on dist-test-slave-mn07"
I20260130 02:57:44.775257 18987 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260130 02:57:44.852315 18987 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260130 02:57:44.853794 18987 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260130 02:57:44.854231 18987 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260130 02:57:44.921710 18987 rpc_server.cc:307] RPC server started. Bound to: 127.18.100.124:34629
I20260130 02:57:44.921789 19064 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.100.124:34629 every 8 connection(s)
I20260130 02:57:44.924435 18987 server_base.cc:1192] Dumped server information to /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/master-2/data/info.pb
I20260130 02:57:44.930228 19065 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:57:44.930480 18833 external_mini_cluster.cc:1428] Started /tmp/dist-test-tasktGe2wI/build/tsan/bin/kudu as pid 18987
I20260130 02:57:44.930951 18833 external_mini_cluster.cc:1442] Reading /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/master-2/wal/instance
I20260130 02:57:44.945783 19065 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.100.126" port: 38191 } has no permanent_uuid. Determining permanent_uuid...
I20260130 02:57:44.973273 19065 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39711 } has no permanent_uuid. Determining permanent_uuid...
I20260130 02:57:44.984292 19065 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34629 } has no permanent_uuid. Determining permanent_uuid...
I20260130 02:57:45.002892 19065 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3749baa287ed4f249329a74071781edd: Bootstrap starting.
I20260130 02:57:45.008316 18910 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34629 } attempt: 6
I20260130 02:57:45.010113 19065 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3749baa287ed4f249329a74071781edd: Neither blocks nor log segments found. Creating new log.
I20260130 02:57:45.012403 19065 log.cc:826] T 00000000000000000000000000000000 P 3749baa287ed4f249329a74071781edd: Log is configured to *not* fsync() on all Append() calls
I20260130 02:57:45.017728 19065 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3749baa287ed4f249329a74071781edd: No bootstrap required, opened a new log
I20260130 02:57:45.026068 18910 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4: Bootstrap starting.
I20260130 02:57:45.032460 18910 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4: Neither blocks nor log segments found. Creating new log.
I20260130 02:57:45.034198 18910 log.cc:826] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4: Log is configured to *not* fsync() on all Append() calls
I20260130 02:57:45.038319 18910 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4: No bootstrap required, opened a new log
I20260130 02:57:45.038329 19065 raft_consensus.cc:359] T 00000000000000000000000000000000 P 3749baa287ed4f249329a74071781edd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "658299197a424f53a37f14fadf089ae4" member_type: VOTER last_known_addr { host: "127.18.100.126" port: 38191 } } peers { permanent_uuid: "107f274cdda543038832f06a56ab9130" member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39711 } } peers { permanent_uuid: "3749baa287ed4f249329a74071781edd" member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34629 } }
I20260130 02:57:45.039011 19065 raft_consensus.cc:385] T 00000000000000000000000000000000 P 3749baa287ed4f249329a74071781edd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:57:45.039234 19065 raft_consensus.cc:740] T 00000000000000000000000000000000 P 3749baa287ed4f249329a74071781edd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3749baa287ed4f249329a74071781edd, State: Initialized, Role: FOLLOWER
I20260130 02:57:45.040007 19065 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3749baa287ed4f249329a74071781edd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "658299197a424f53a37f14fadf089ae4" member_type: VOTER last_known_addr { host: "127.18.100.126" port: 38191 } } peers { permanent_uuid: "107f274cdda543038832f06a56ab9130" member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39711 } } peers { permanent_uuid: "3749baa287ed4f249329a74071781edd" member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34629 } }
I20260130 02:57:45.044894 19078 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3749baa287ed4f249329a74071781edd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "658299197a424f53a37f14fadf089ae4" member_type: VOTER last_known_addr { host: "127.18.100.126" port: 38191 } } peers { permanent_uuid: "107f274cdda543038832f06a56ab9130" member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39711 } } peers { permanent_uuid: "3749baa287ed4f249329a74071781edd" member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34629 } } }
I20260130 02:57:45.045979 19078 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3749baa287ed4f249329a74071781edd [sys.catalog]: This master's current role is: FOLLOWER
I20260130 02:57:45.047138 19065 sys_catalog.cc:565] T 00000000000000000000000000000000 P 3749baa287ed4f249329a74071781edd [sys.catalog]: configured and running, proceeding with master startup.
I20260130 02:57:45.060694 18910 raft_consensus.cc:359] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "658299197a424f53a37f14fadf089ae4" member_type: VOTER last_known_addr { host: "127.18.100.126" port: 38191 } } peers { permanent_uuid: "107f274cdda543038832f06a56ab9130" member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39711 } } peers { permanent_uuid: "3749baa287ed4f249329a74071781edd" member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34629 } }
I20260130 02:57:45.061627 18910 raft_consensus.cc:385] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:57:45.061944 18910 raft_consensus.cc:740] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 658299197a424f53a37f14fadf089ae4, State: Initialized, Role: FOLLOWER
I20260130 02:57:45.062917 18910 consensus_queue.cc:260] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "658299197a424f53a37f14fadf089ae4" member_type: VOTER last_known_addr { host: "127.18.100.126" port: 38191 } } peers { permanent_uuid: "107f274cdda543038832f06a56ab9130" member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39711 } } peers { permanent_uuid: "3749baa287ed4f249329a74071781edd" member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34629 } }
I20260130 02:57:45.065814 19088 sys_catalog.cc:455] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "658299197a424f53a37f14fadf089ae4" member_type: VOTER last_known_addr { host: "127.18.100.126" port: 38191 } } peers { permanent_uuid: "107f274cdda543038832f06a56ab9130" member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39711 } } peers { permanent_uuid: "3749baa287ed4f249329a74071781edd" member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34629 } } }
I20260130 02:57:45.066642 19088 sys_catalog.cc:458] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4 [sys.catalog]: This master's current role is: FOLLOWER
I20260130 02:57:45.067631 18910 sys_catalog.cc:565] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4 [sys.catalog]: configured and running, proceeding with master startup.
W20260130 02:57:45.079665 19090 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 3749baa287ed4f249329a74071781edd: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260130 02:57:45.080159 19090 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260130 02:57:45.073050 18987 master_runner.cc:186] Error getting master registration for 127.18.100.126:38191: OK, instance_id { permanent_uuid: "658299197a424f53a37f14fadf089ae4" instance_seqno: 1769741861334545 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260130 02:57:45.102380 18987 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260130 02:57:45.109164 19019 tablet.cc:2378] T 00000000000000000000000000000000 P 3749baa287ed4f249329a74071781edd: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260130 02:57:45.113700 19102 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260130 02:57:45.114179 19102 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260130 02:57:45.108184 18842 master_runner.cc:186] Error getting master registration for 127.18.100.125:39711: OK, instance_id { permanent_uuid: "107f274cdda543038832f06a56ab9130" instance_seqno: 1769741863109848 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260130 02:57:45.141263 18842 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260130 02:57:45.237483 19078 raft_consensus.cc:493] T 00000000000000000000000000000000 P 3749baa287ed4f249329a74071781edd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:57:45.238116 19078 raft_consensus.cc:515] T 00000000000000000000000000000000 P 3749baa287ed4f249329a74071781edd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "658299197a424f53a37f14fadf089ae4" member_type: VOTER last_known_addr { host: "127.18.100.126" port: 38191 } } peers { permanent_uuid: "107f274cdda543038832f06a56ab9130" member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39711 } } peers { permanent_uuid: "3749baa287ed4f249329a74071781edd" member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34629 } }
I20260130 02:57:45.241091 19078 leader_election.cc:290] T 00000000000000000000000000000000 P 3749baa287ed4f249329a74071781edd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 658299197a424f53a37f14fadf089ae4 (127.18.100.126:38191), 107f274cdda543038832f06a56ab9130 (127.18.100.125:39711)
I20260130 02:57:45.241882 18885 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3749baa287ed4f249329a74071781edd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "658299197a424f53a37f14fadf089ae4" is_pre_election: true
I20260130 02:57:45.242820 18885 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3749baa287ed4f249329a74071781edd in term 0.
I20260130 02:57:45.244284 19014 leader_election.cc:304] T 00000000000000000000000000000000 P 3749baa287ed4f249329a74071781edd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3749baa287ed4f249329a74071781edd, 658299197a424f53a37f14fadf089ae4; no voters:
I20260130 02:57:45.245289 19078 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 3749baa287ed4f249329a74071781edd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260130 02:57:45.246032 19078 raft_consensus.cc:493] T 00000000000000000000000000000000 P 3749baa287ed4f249329a74071781edd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260130 02:57:45.246762 19078 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 3749baa287ed4f249329a74071781edd [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:57:45.260931 19078 raft_consensus.cc:515] T 00000000000000000000000000000000 P 3749baa287ed4f249329a74071781edd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "658299197a424f53a37f14fadf089ae4" member_type: VOTER last_known_addr { host: "127.18.100.126" port: 38191 } } peers { permanent_uuid: "107f274cdda543038832f06a56ab9130" member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39711 } } peers { permanent_uuid: "3749baa287ed4f249329a74071781edd" member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34629 } }
I20260130 02:57:45.264901 19078 leader_election.cc:290] T 00000000000000000000000000000000 P 3749baa287ed4f249329a74071781edd [CANDIDATE]: Term 1 election: Requested vote from peers 658299197a424f53a37f14fadf089ae4 (127.18.100.126:38191), 107f274cdda543038832f06a56ab9130 (127.18.100.125:39711)
I20260130 02:57:45.267264 18885 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3749baa287ed4f249329a74071781edd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "658299197a424f53a37f14fadf089ae4"
I20260130 02:57:45.267838 18885 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:57:45.275627 18885 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3749baa287ed4f249329a74071781edd in term 1.
I20260130 02:57:45.276873 19014 leader_election.cc:304] T 00000000000000000000000000000000 P 3749baa287ed4f249329a74071781edd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3749baa287ed4f249329a74071781edd, 658299197a424f53a37f14fadf089ae4; no voters:
I20260130 02:57:45.277861 19078 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 3749baa287ed4f249329a74071781edd [term 1 FOLLOWER]: Leader election won for term 1
I20260130 02:57:45.281014 19078 raft_consensus.cc:697] T 00000000000000000000000000000000 P 3749baa287ed4f249329a74071781edd [term 1 LEADER]: Becoming Leader. State: Replica: 3749baa287ed4f249329a74071781edd, State: Running, Role: LEADER
I20260130 02:57:45.283262 19078 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3749baa287ed4f249329a74071781edd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "658299197a424f53a37f14fadf089ae4" member_type: VOTER last_known_addr { host: "127.18.100.126" port: 38191 } } peers { permanent_uuid: "107f274cdda543038832f06a56ab9130" member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39711 } } peers { permanent_uuid: "3749baa287ed4f249329a74071781edd" member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34629 } }
W20260130 02:57:45.294853 18864 tablet.cc:2378] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260130 02:57:45.299450 19104 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3749baa287ed4f249329a74071781edd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3749baa287ed4f249329a74071781edd. Latest consensus state: current_term: 1 leader_uuid: "3749baa287ed4f249329a74071781edd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "658299197a424f53a37f14fadf089ae4" member_type: VOTER last_known_addr { host: "127.18.100.126" port: 38191 } } peers { permanent_uuid: "107f274cdda543038832f06a56ab9130" member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39711 } } peers { permanent_uuid: "3749baa287ed4f249329a74071781edd" member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34629 } } }
I20260130 02:57:45.300246 19104 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3749baa287ed4f249329a74071781edd [sys.catalog]: This master's current role is: LEADER
I20260130 02:57:45.308193 19106 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260130 02:57:45.316021 19106 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260130 02:57:45.345424 18885 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4 [term 1 FOLLOWER]: Refusing update from remote peer 3749baa287ed4f249329a74071781edd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260130 02:57:45.347420 19078 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 3749baa287ed4f249329a74071781edd [LEADER]: Connected to new peer: Peer: permanent_uuid: "658299197a424f53a37f14fadf089ae4" member_type: VOTER last_known_addr { host: "127.18.100.126" port: 38191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260130 02:57:45.392488 19088 sys_catalog.cc:455] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3749baa287ed4f249329a74071781edd. Latest consensus state: current_term: 1 leader_uuid: "3749baa287ed4f249329a74071781edd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "658299197a424f53a37f14fadf089ae4" member_type: VOTER last_known_addr { host: "127.18.100.126" port: 38191 } } peers { permanent_uuid: "107f274cdda543038832f06a56ab9130" member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39711 } } peers { permanent_uuid: "3749baa287ed4f249329a74071781edd" member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34629 } } }
I20260130 02:57:45.393203 19088 sys_catalog.cc:458] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4 [sys.catalog]: This master's current role is: FOLLOWER
I20260130 02:57:45.401067 19107 mvcc.cc:204] Tried to move back new op lower bound from 7248862680429342720 to 7248862680223113216. Current Snapshot: MvccSnapshot[applied={T|T < 7248862680429342720}]
I20260130 02:57:45.410845 19078 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3749baa287ed4f249329a74071781edd [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3749baa287ed4f249329a74071781edd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "658299197a424f53a37f14fadf089ae4" member_type: VOTER last_known_addr { host: "127.18.100.126" port: 38191 } } peers { permanent_uuid: "107f274cdda543038832f06a56ab9130" member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39711 } } peers { permanent_uuid: "3749baa287ed4f249329a74071781edd" member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34629 } } }
I20260130 02:57:45.411605 19078 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3749baa287ed4f249329a74071781edd [sys.catalog]: This master's current role is: LEADER
I20260130 02:57:45.414539 19088 sys_catalog.cc:455] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3749baa287ed4f249329a74071781edd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "658299197a424f53a37f14fadf089ae4" member_type: VOTER last_known_addr { host: "127.18.100.126" port: 38191 } } peers { permanent_uuid: "107f274cdda543038832f06a56ab9130" member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39711 } } peers { permanent_uuid: "3749baa287ed4f249329a74071781edd" member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34629 } } }
I20260130 02:57:45.415235 19088 sys_catalog.cc:458] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4 [sys.catalog]: This master's current role is: FOLLOWER
I20260130 02:57:45.421811 19106 catalog_manager.cc:1357] Generated new cluster ID: db1c0e7709154c10a8c9a0ea68ffe38b
I20260130 02:57:45.422144 19106 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260130 02:57:45.508991 18985 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34629 } attempt: 7
I20260130 02:57:45.514604 19106 catalog_manager.cc:1380] Generated new certificate authority record
I20260130 02:57:45.516631 19106 catalog_manager.cc:1514] Loading token signing keys...
I20260130 02:57:45.568264 19106 catalog_manager.cc:6027] T 00000000000000000000000000000000 P 3749baa287ed4f249329a74071781edd: Generated new TSK 0
I20260130 02:57:45.569406 19106 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260130 02:57:45.578770 18985 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 107f274cdda543038832f06a56ab9130: Bootstrap starting.
I20260130 02:57:45.586091 18985 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 107f274cdda543038832f06a56ab9130: Neither blocks nor log segments found. Creating new log.
I20260130 02:57:45.588315 18985 log.cc:826] T 00000000000000000000000000000000 P 107f274cdda543038832f06a56ab9130: Log is configured to *not* fsync() on all Append() calls
I20260130 02:57:45.598834 18985 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 107f274cdda543038832f06a56ab9130: No bootstrap required, opened a new log
I20260130 02:57:45.601306 18833 external_mini_cluster.cc:1366] Running /tmp/dist-test-tasktGe2wI/build/tsan/bin/kudu
/tmp/dist-test-tasktGe2wI/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.100.65:0
--local_ip_for_outbound_sockets=127.18.100.65
--webserver_interface=127.18.100.65
--webserver_port=0
--tserver_master_addrs=127.18.100.126:38191,127.18.100.125:39711,127.18.100.124:34629
--builtin_ntp_servers=127.18.100.84:46033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20260130 02:57:45.640314 18985 raft_consensus.cc:359] T 00000000000000000000000000000000 P 107f274cdda543038832f06a56ab9130 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "658299197a424f53a37f14fadf089ae4" member_type: VOTER last_known_addr { host: "127.18.100.126" port: 38191 } } peers { permanent_uuid: "107f274cdda543038832f06a56ab9130" member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39711 } } peers { permanent_uuid: "3749baa287ed4f249329a74071781edd" member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34629 } }
I20260130 02:57:45.641147 18985 raft_consensus.cc:385] T 00000000000000000000000000000000 P 107f274cdda543038832f06a56ab9130 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:57:45.641450 18985 raft_consensus.cc:740] T 00000000000000000000000000000000 P 107f274cdda543038832f06a56ab9130 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 107f274cdda543038832f06a56ab9130, State: Initialized, Role: FOLLOWER
I20260130 02:57:45.642328 18985 consensus_queue.cc:260] T 00000000000000000000000000000000 P 107f274cdda543038832f06a56ab9130 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "658299197a424f53a37f14fadf089ae4" member_type: VOTER last_known_addr { host: "127.18.100.126" port: 38191 } } peers { permanent_uuid: "107f274cdda543038832f06a56ab9130" member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39711 } } peers { permanent_uuid: "3749baa287ed4f249329a74071781edd" member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34629 } }
I20260130 02:57:45.652763 19115 sys_catalog.cc:455] T 00000000000000000000000000000000 P 107f274cdda543038832f06a56ab9130 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "658299197a424f53a37f14fadf089ae4" member_type: VOTER last_known_addr { host: "127.18.100.126" port: 38191 } } peers { permanent_uuid: "107f274cdda543038832f06a56ab9130" member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39711 } } peers { permanent_uuid: "3749baa287ed4f249329a74071781edd" member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34629 } } }
I20260130 02:57:45.655100 19115 sys_catalog.cc:458] T 00000000000000000000000000000000 P 107f274cdda543038832f06a56ab9130 [sys.catalog]: This master's current role is: FOLLOWER
I20260130 02:57:45.661311 18985 sys_catalog.cc:565] T 00000000000000000000000000000000 P 107f274cdda543038832f06a56ab9130 [sys.catalog]: configured and running, proceeding with master startup.
I20260130 02:57:45.661484 18959 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3749baa287ed4f249329a74071781edd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "107f274cdda543038832f06a56ab9130"
I20260130 02:57:45.662293 18959 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 107f274cdda543038832f06a56ab9130 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:57:45.666177 18960 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3749baa287ed4f249329a74071781edd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "107f274cdda543038832f06a56ab9130" is_pre_election: true
I20260130 02:57:45.674526 18959 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 107f274cdda543038832f06a56ab9130 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3749baa287ed4f249329a74071781edd in term 1.
I20260130 02:57:45.675839 18958 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 107f274cdda543038832f06a56ab9130 [term 1 FOLLOWER]: Refusing update from remote peer 3749baa287ed4f249329a74071781edd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260130 02:57:45.681183 19104 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 3749baa287ed4f249329a74071781edd [LEADER]: Connected to new peer: Peer: permanent_uuid: "107f274cdda543038832f06a56ab9130" member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39711 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260130 02:57:45.740857 19115 sys_catalog.cc:455] T 00000000000000000000000000000000 P 107f274cdda543038832f06a56ab9130 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3749baa287ed4f249329a74071781edd. Latest consensus state: current_term: 1 leader_uuid: "3749baa287ed4f249329a74071781edd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "658299197a424f53a37f14fadf089ae4" member_type: VOTER last_known_addr { host: "127.18.100.126" port: 38191 } } peers { permanent_uuid: "107f274cdda543038832f06a56ab9130" member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39711 } } peers { permanent_uuid: "3749baa287ed4f249329a74071781edd" member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34629 } } }
I20260130 02:57:45.741581 19115 sys_catalog.cc:458] T 00000000000000000000000000000000 P 107f274cdda543038832f06a56ab9130 [sys.catalog]: This master's current role is: FOLLOWER
I20260130 02:57:45.743209 19119 sys_catalog.cc:455] T 00000000000000000000000000000000 P 107f274cdda543038832f06a56ab9130 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3749baa287ed4f249329a74071781edd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "658299197a424f53a37f14fadf089ae4" member_type: VOTER last_known_addr { host: "127.18.100.126" port: 38191 } } peers { permanent_uuid: "107f274cdda543038832f06a56ab9130" member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39711 } } peers { permanent_uuid: "3749baa287ed4f249329a74071781edd" member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34629 } } }
I20260130 02:57:45.743965 19119 sys_catalog.cc:458] T 00000000000000000000000000000000 P 107f274cdda543038832f06a56ab9130 [sys.catalog]: This master's current role is: FOLLOWER
I20260130 02:57:45.843451 19131 catalog_manager.cc:1269] Loaded cluster ID: db1c0e7709154c10a8c9a0ea68ffe38b
I20260130 02:57:45.843819 19131 catalog_manager.cc:1562] T 00000000000000000000000000000000 P 107f274cdda543038832f06a56ab9130: loading cluster ID for follower catalog manager: success
I20260130 02:57:45.851397 19131 catalog_manager.cc:1584] T 00000000000000000000000000000000 P 107f274cdda543038832f06a56ab9130: acquiring CA information for follower catalog manager: success
I20260130 02:57:45.865190 19131 catalog_manager.cc:1612] T 00000000000000000000000000000000 P 107f274cdda543038832f06a56ab9130: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260130 02:57:46.120369 19102 catalog_manager.cc:1269] Loaded cluster ID: db1c0e7709154c10a8c9a0ea68ffe38b
I20260130 02:57:46.120766 19102 catalog_manager.cc:1562] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4: loading cluster ID for follower catalog manager: success
I20260130 02:57:46.128379 19102 catalog_manager.cc:1584] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4: acquiring CA information for follower catalog manager: success
I20260130 02:57:46.144501 19102 catalog_manager.cc:1612] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260130 02:57:46.182896 19114 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260130 02:57:46.183554 19114 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260130 02:57:46.184495 19114 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260130 02:57:46.186486 19119 raft_consensus.cc:493] T 00000000000000000000000000000000 P 107f274cdda543038832f06a56ab9130 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3749baa287ed4f249329a74071781edd)
I20260130 02:57:46.187122 19119 raft_consensus.cc:515] T 00000000000000000000000000000000 P 107f274cdda543038832f06a56ab9130 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "658299197a424f53a37f14fadf089ae4" member_type: VOTER last_known_addr { host: "127.18.100.126" port: 38191 } } peers { permanent_uuid: "107f274cdda543038832f06a56ab9130" member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39711 } } peers { permanent_uuid: "3749baa287ed4f249329a74071781edd" member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34629 } }
I20260130 02:57:46.191138 19040 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "107f274cdda543038832f06a56ab9130" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3749baa287ed4f249329a74071781edd" is_pre_election: true
I20260130 02:57:46.193681 18885 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "107f274cdda543038832f06a56ab9130" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "658299197a424f53a37f14fadf089ae4" is_pre_election: true
I20260130 02:57:46.196024 18934 leader_election.cc:304] T 00000000000000000000000000000000 P 107f274cdda543038832f06a56ab9130 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 107f274cdda543038832f06a56ab9130; no voters: 3749baa287ed4f249329a74071781edd, 658299197a424f53a37f14fadf089ae4
I20260130 02:57:46.196918 19115 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 107f274cdda543038832f06a56ab9130 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260130 02:57:46.197625 19119 leader_election.cc:290] T 00000000000000000000000000000000 P 107f274cdda543038832f06a56ab9130 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 658299197a424f53a37f14fadf089ae4 (127.18.100.126:38191), 3749baa287ed4f249329a74071781edd (127.18.100.124:34629)
I20260130 02:57:46.282408 19137 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3749baa287ed4f249329a74071781edd [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3749baa287ed4f249329a74071781edd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "658299197a424f53a37f14fadf089ae4" member_type: VOTER last_known_addr { host: "127.18.100.126" port: 38191 } } peers { permanent_uuid: "107f274cdda543038832f06a56ab9130" member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39711 } } peers { permanent_uuid: "3749baa287ed4f249329a74071781edd" member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34629 } } }
I20260130 02:57:46.284444 19137 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3749baa287ed4f249329a74071781edd [sys.catalog]: This master's current role is: LEADER
W20260130 02:57:46.290109 19114 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260130 02:57:46.291646 19114 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.100.65
I20260130 02:57:46.368588 19114 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.100.84:46033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.100.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.100.65
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.18.100.126:38191,127.18.100.125:39711,127.18.100.124:34629
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.100.65
--log_dir=/tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type FASTDEBUG
built by None at 30 Jan 2026 02:43:21 UTC on 5fd53c4cbb9d
build id 10257
TSAN enabled
I20260130 02:57:46.370241 19114 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260130 02:57:46.372327 19114 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260130 02:57:46.432495 19142 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260130 02:57:46.463107 19146 raft_consensus.cc:493] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3749baa287ed4f249329a74071781edd)
I20260130 02:57:46.463691 19146 raft_consensus.cc:515] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "658299197a424f53a37f14fadf089ae4" member_type: VOTER last_known_addr { host: "127.18.100.126" port: 38191 } } peers { permanent_uuid: "107f274cdda543038832f06a56ab9130" member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39711 } } peers { permanent_uuid: "3749baa287ed4f249329a74071781edd" member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34629 } }
I20260130 02:57:46.466182 19146 leader_election.cc:290] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 107f274cdda543038832f06a56ab9130 (127.18.100.125:39711), 3749baa287ed4f249329a74071781edd (127.18.100.124:34629)
I20260130 02:57:46.467278 18960 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "658299197a424f53a37f14fadf089ae4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "107f274cdda543038832f06a56ab9130" is_pre_election: true
I20260130 02:57:46.468425 19040 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "658299197a424f53a37f14fadf089ae4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3749baa287ed4f249329a74071781edd" is_pre_election: true
I20260130 02:57:46.470770 18860 leader_election.cc:304] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 658299197a424f53a37f14fadf089ae4; no voters: 107f274cdda543038832f06a56ab9130, 3749baa287ed4f249329a74071781edd
I20260130 02:57:46.471889 19146 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260130 02:57:47.628764 19148 raft_consensus.cc:493] T 00000000000000000000000000000000 P 107f274cdda543038832f06a56ab9130 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3749baa287ed4f249329a74071781edd)
I20260130 02:57:47.629540 19148 raft_consensus.cc:515] T 00000000000000000000000000000000 P 107f274cdda543038832f06a56ab9130 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "658299197a424f53a37f14fadf089ae4" member_type: VOTER last_known_addr { host: "127.18.100.126" port: 38191 } } peers { permanent_uuid: "107f274cdda543038832f06a56ab9130" member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39711 } } peers { permanent_uuid: "3749baa287ed4f249329a74071781edd" member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34629 } }
I20260130 02:57:47.632409 18885 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "107f274cdda543038832f06a56ab9130" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "658299197a424f53a37f14fadf089ae4" is_pre_election: true
I20260130 02:57:47.641027 19040 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "107f274cdda543038832f06a56ab9130" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3749baa287ed4f249329a74071781edd" is_pre_election: true
I20260130 02:57:47.642714 18935 leader_election.cc:304] T 00000000000000000000000000000000 P 107f274cdda543038832f06a56ab9130 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 107f274cdda543038832f06a56ab9130; no voters: 3749baa287ed4f249329a74071781edd, 658299197a424f53a37f14fadf089ae4
I20260130 02:57:47.646543 19148 leader_election.cc:290] T 00000000000000000000000000000000 P 107f274cdda543038832f06a56ab9130 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 658299197a424f53a37f14fadf089ae4 (127.18.100.126:38191), 3749baa287ed4f249329a74071781edd (127.18.100.124:34629)
I20260130 02:57:47.647559 19148 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 107f274cdda543038832f06a56ab9130 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260130 02:57:48.241612 19150 raft_consensus.cc:493] T 00000000000000000000000000000000 P 107f274cdda543038832f06a56ab9130 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3749baa287ed4f249329a74071781edd)
I20260130 02:57:48.243388 19150 raft_consensus.cc:515] T 00000000000000000000000000000000 P 107f274cdda543038832f06a56ab9130 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "658299197a424f53a37f14fadf089ae4" member_type: VOTER last_known_addr { host: "127.18.100.126" port: 38191 } } peers { permanent_uuid: "107f274cdda543038832f06a56ab9130" member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39711 } } peers { permanent_uuid: "3749baa287ed4f249329a74071781edd" member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34629 } }
I20260130 02:57:48.251184 18885 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "107f274cdda543038832f06a56ab9130" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "658299197a424f53a37f14fadf089ae4" is_pre_election: true
I20260130 02:57:48.254698 19040 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "107f274cdda543038832f06a56ab9130" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3749baa287ed4f249329a74071781edd" is_pre_election: true
I20260130 02:57:48.256516 18935 leader_election.cc:304] T 00000000000000000000000000000000 P 107f274cdda543038832f06a56ab9130 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 107f274cdda543038832f06a56ab9130; no voters: 3749baa287ed4f249329a74071781edd, 658299197a424f53a37f14fadf089ae4
I20260130 02:57:48.261076 19150 leader_election.cc:290] T 00000000000000000000000000000000 P 107f274cdda543038832f06a56ab9130 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 658299197a424f53a37f14fadf089ae4 (127.18.100.126:38191), 3749baa287ed4f249329a74071781edd (127.18.100.124:34629)
I20260130 02:57:48.263375 19150 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 107f274cdda543038832f06a56ab9130 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260130 02:57:48.302320 19152 raft_consensus.cc:493] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3749baa287ed4f249329a74071781edd)
I20260130 02:57:48.304000 19152 raft_consensus.cc:515] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "658299197a424f53a37f14fadf089ae4" member_type: VOTER last_known_addr { host: "127.18.100.126" port: 38191 } } peers { permanent_uuid: "107f274cdda543038832f06a56ab9130" member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39711 } } peers { permanent_uuid: "3749baa287ed4f249329a74071781edd" member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34629 } }
I20260130 02:57:48.307547 18960 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "658299197a424f53a37f14fadf089ae4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "107f274cdda543038832f06a56ab9130" is_pre_election: true
I20260130 02:57:48.309666 19040 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "658299197a424f53a37f14fadf089ae4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3749baa287ed4f249329a74071781edd" is_pre_election: true
I20260130 02:57:48.310926 18860 leader_election.cc:304] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 658299197a424f53a37f14fadf089ae4; no voters: 107f274cdda543038832f06a56ab9130, 3749baa287ed4f249329a74071781edd
I20260130 02:57:48.311221 19152 leader_election.cc:290] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 107f274cdda543038832f06a56ab9130 (127.18.100.125:39711), 3749baa287ed4f249329a74071781edd (127.18.100.124:34629)
I20260130 02:57:48.317631 19152 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260130 02:57:47.818328 19140 debug-util.cc:398] Leaking SignalData structure 0x7b0800034c40 after lost signal to thread 19114
W20260130 02:57:48.526357 19140 kernel_stack_watchdog.cc:198] Thread 19114 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:642 for 382ms:
Kernel stack:
(could not read kernel stack)
User stack:
<Timed out: thread did not respond: maybe it is blocking signals>
W20260130 02:57:46.433490 19141 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260130 02:57:48.531839 19114 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 2.097s user 0.003s sys 0.000s
W20260130 02:57:48.532215 19114 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 2.098s user 0.003s sys 0.000s
W20260130 02:57:48.533993 19145 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260130 02:57:48.535120 19114 server_base.cc:1055] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
W20260130 02:57:48.535032 19143 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection timed out after 2100 milliseconds
I20260130 02:57:48.536785 19114 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260130 02:57:48.539439 19114 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260130 02:57:48.540874 19114 hybrid_clock.cc:648] HybridClock initialized: now 1769741868540833 us; error 47 us; skew 500 ppm
I20260130 02:57:48.541934 19114 server_base.cc:860] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260130 02:57:48.549427 19114 webserver.cc:492] Webserver started at http://127.18.100.65:45051/ using document root <none> and password file <none>
I20260130 02:57:48.550719 19114 fs_manager.cc:362] Metadata directory not provided
I20260130 02:57:48.551007 19114 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260130 02:57:48.551578 19114 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260130 02:57:48.558648 19114 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-0/data/instance:
uuid: "371a36d219164974abdf848841528152"
format_stamp: "Formatted at 2026-01-30 02:57:48 on dist-test-slave-mn07"
I20260130 02:57:48.560186 19114 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-0/wal/instance:
uuid: "371a36d219164974abdf848841528152"
format_stamp: "Formatted at 2026-01-30 02:57:48 on dist-test-slave-mn07"
I20260130 02:57:48.570221 19114 fs_manager.cc:696] Time spent creating directory manager: real 0.009s user 0.010s sys 0.000s
I20260130 02:57:48.577646 19157 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260130 02:57:48.578958 19114 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260130 02:57:48.579355 19114 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-0/wal
uuid: "371a36d219164974abdf848841528152"
format_stamp: "Formatted at 2026-01-30 02:57:48 on dist-test-slave-mn07"
I20260130 02:57:48.579797 19114 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260130 02:57:48.643613 19114 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260130 02:57:48.645637 19114 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260130 02:57:48.646189 19114 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260130 02:57:48.650466 19114 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260130 02:57:48.654960 19114 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260130 02:57:48.655174 19114 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260130 02:57:48.655457 19114 ts_tablet_manager.cc:616] Registered 0 tablets
I20260130 02:57:48.655619 19114 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260130 02:57:48.821040 19114 rpc_server.cc:307] RPC server started. Bound to: 127.18.100.65:42021
I20260130 02:57:48.821208 19269 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.100.65:42021 every 8 connection(s)
I20260130 02:57:48.823833 19114 server_base.cc:1192] Dumped server information to /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-0/data/info.pb
I20260130 02:57:48.829396 18833 external_mini_cluster.cc:1428] Started /tmp/dist-test-tasktGe2wI/build/tsan/bin/kudu as pid 19114
I20260130 02:57:48.829836 18833 external_mini_cluster.cc:1442] Reading /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-0/wal/instance
I20260130 02:57:48.841624 18833 external_mini_cluster.cc:1366] Running /tmp/dist-test-tasktGe2wI/build/tsan/bin/kudu
/tmp/dist-test-tasktGe2wI/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.100.66:0
--local_ip_for_outbound_sockets=127.18.100.66
--webserver_interface=127.18.100.66
--webserver_port=0
--tserver_master_addrs=127.18.100.126:38191,127.18.100.125:39711,127.18.100.124:34629
--builtin_ntp_servers=127.18.100.84:46033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20260130 02:57:48.875188 19270 heartbeater.cc:344] Connected to a master server at 127.18.100.124:34629
I20260130 02:57:48.875682 19270 heartbeater.cc:461] Registering TS with master...
I20260130 02:57:48.877130 19270 heartbeater.cc:507] Master 127.18.100.124:34629 requested a full tablet report, sending...
I20260130 02:57:48.880612 19030 ts_manager.cc:194] Registered new tserver with Master: 371a36d219164974abdf848841528152 (127.18.100.65:42021)
I20260130 02:57:48.882620 19275 heartbeater.cc:344] Connected to a master server at 127.18.100.126:38191
I20260130 02:57:48.882999 19275 heartbeater.cc:461] Registering TS with master...
I20260130 02:57:48.883885 19275 heartbeater.cc:507] Master 127.18.100.126:38191 requested a full tablet report, sending...
I20260130 02:57:48.886791 18875 ts_manager.cc:194] Registered new tserver with Master: 371a36d219164974abdf848841528152 (127.18.100.65:42021)
I20260130 02:57:48.883634 19030 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.18.100.65:46609
I20260130 02:57:48.894325 19271 heartbeater.cc:344] Connected to a master server at 127.18.100.125:39711
I20260130 02:57:48.894752 19271 heartbeater.cc:461] Registering TS with master...
I20260130 02:57:48.895463 19271 heartbeater.cc:507] Master 127.18.100.125:39711 requested a full tablet report, sending...
I20260130 02:57:48.898427 18950 ts_manager.cc:194] Registered new tserver with Master: 371a36d219164974abdf848841528152 (127.18.100.65:42021)
W20260130 02:57:49.220309 19279 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260130 02:57:49.220811 19279 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260130 02:57:49.221459 19279 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260130 02:57:49.255756 19279 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260130 02:57:49.256729 19279 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.100.66
I20260130 02:57:49.279498 19286 raft_consensus.cc:493] T 00000000000000000000000000000000 P 107f274cdda543038832f06a56ab9130 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3749baa287ed4f249329a74071781edd)
I20260130 02:57:49.280040 19286 raft_consensus.cc:515] T 00000000000000000000000000000000 P 107f274cdda543038832f06a56ab9130 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "658299197a424f53a37f14fadf089ae4" member_type: VOTER last_known_addr { host: "127.18.100.126" port: 38191 } } peers { permanent_uuid: "107f274cdda543038832f06a56ab9130" member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39711 } } peers { permanent_uuid: "3749baa287ed4f249329a74071781edd" member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34629 } }
I20260130 02:57:49.282436 19286 leader_election.cc:290] T 00000000000000000000000000000000 P 107f274cdda543038832f06a56ab9130 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 658299197a424f53a37f14fadf089ae4 (127.18.100.126:38191), 3749baa287ed4f249329a74071781edd (127.18.100.124:34629)
I20260130 02:57:49.283701 19040 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "107f274cdda543038832f06a56ab9130" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3749baa287ed4f249329a74071781edd" is_pre_election: true
I20260130 02:57:49.283926 18885 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "107f274cdda543038832f06a56ab9130" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "658299197a424f53a37f14fadf089ae4" is_pre_election: true
I20260130 02:57:49.285260 18934 leader_election.cc:304] T 00000000000000000000000000000000 P 107f274cdda543038832f06a56ab9130 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 107f274cdda543038832f06a56ab9130; no voters: 3749baa287ed4f249329a74071781edd, 658299197a424f53a37f14fadf089ae4
I20260130 02:57:49.286005 19286 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 107f274cdda543038832f06a56ab9130 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260130 02:57:49.300590 19287 raft_consensus.cc:493] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3749baa287ed4f249329a74071781edd)
I20260130 02:57:49.300979 19287 raft_consensus.cc:515] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "658299197a424f53a37f14fadf089ae4" member_type: VOTER last_known_addr { host: "127.18.100.126" port: 38191 } } peers { permanent_uuid: "107f274cdda543038832f06a56ab9130" member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39711 } } peers { permanent_uuid: "3749baa287ed4f249329a74071781edd" member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34629 } }
I20260130 02:57:49.302599 19287 leader_election.cc:290] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 107f274cdda543038832f06a56ab9130 (127.18.100.125:39711), 3749baa287ed4f249329a74071781edd (127.18.100.124:34629)
I20260130 02:57:49.303736 19040 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "658299197a424f53a37f14fadf089ae4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3749baa287ed4f249329a74071781edd" is_pre_election: true
I20260130 02:57:49.303617 18960 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "658299197a424f53a37f14fadf089ae4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "107f274cdda543038832f06a56ab9130" is_pre_election: true
I20260130 02:57:49.304600 18960 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 107f274cdda543038832f06a56ab9130 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 658299197a424f53a37f14fadf089ae4 in term 1.
I20260130 02:57:49.305568 18860 leader_election.cc:304] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 107f274cdda543038832f06a56ab9130, 658299197a424f53a37f14fadf089ae4; no voters: 3749baa287ed4f249329a74071781edd
I20260130 02:57:49.306157 19287 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260130 02:57:49.306097 19279 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.100.84:46033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.100.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.100.66
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.18.100.126:38191,127.18.100.125:39711,127.18.100.124:34629
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.100.66
--log_dir=/tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type FASTDEBUG
built by None at 30 Jan 2026 02:43:21 UTC on 5fd53c4cbb9d
build id 10257
TSAN enabled
I20260130 02:57:49.306452 19287 raft_consensus.cc:493] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 3749baa287ed4f249329a74071781edd)
I20260130 02:57:49.306797 19287 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4 [term 1 FOLLOWER]: Advancing to term 2
I20260130 02:57:49.307701 19279 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260130 02:57:49.309693 19279 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20260130 02:57:49.311651 19287 raft_consensus.cc:515] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "658299197a424f53a37f14fadf089ae4" member_type: VOTER last_known_addr { host: "127.18.100.126" port: 38191 } } peers { permanent_uuid: "107f274cdda543038832f06a56ab9130" member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39711 } } peers { permanent_uuid: "3749baa287ed4f249329a74071781edd" member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34629 } }
I20260130 02:57:49.313185 19287 leader_election.cc:290] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4 [CANDIDATE]: Term 2 election: Requested vote from peers 107f274cdda543038832f06a56ab9130 (127.18.100.125:39711), 3749baa287ed4f249329a74071781edd (127.18.100.124:34629)
I20260130 02:57:49.314347 19040 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "658299197a424f53a37f14fadf089ae4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3749baa287ed4f249329a74071781edd"
I20260130 02:57:49.314251 18960 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "658299197a424f53a37f14fadf089ae4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "107f274cdda543038832f06a56ab9130"
I20260130 02:57:49.314863 18960 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 107f274cdda543038832f06a56ab9130 [term 1 FOLLOWER]: Advancing to term 2
I20260130 02:57:49.319605 18960 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 107f274cdda543038832f06a56ab9130 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 658299197a424f53a37f14fadf089ae4 in term 2.
I20260130 02:57:49.320909 18860 leader_election.cc:304] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 107f274cdda543038832f06a56ab9130, 658299197a424f53a37f14fadf089ae4; no voters: 3749baa287ed4f249329a74071781edd
I20260130 02:57:49.322031 19287 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4 [term 2 FOLLOWER]: Leader election won for term 2
I20260130 02:57:49.327059 19287 raft_consensus.cc:697] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4 [term 2 LEADER]: Becoming Leader. State: Replica: 658299197a424f53a37f14fadf089ae4, State: Running, Role: LEADER
W20260130 02:57:49.327327 19289 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260130 02:57:49.328155 19287 consensus_queue.cc:237] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "658299197a424f53a37f14fadf089ae4" member_type: VOTER last_known_addr { host: "127.18.100.126" port: 38191 } } peers { permanent_uuid: "107f274cdda543038832f06a56ab9130" member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39711 } } peers { permanent_uuid: "3749baa287ed4f249329a74071781edd" member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34629 } }
I20260130 02:57:49.351733 19291 sys_catalog.cc:455] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 658299197a424f53a37f14fadf089ae4. Latest consensus state: current_term: 2 leader_uuid: "658299197a424f53a37f14fadf089ae4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "658299197a424f53a37f14fadf089ae4" member_type: VOTER last_known_addr { host: "127.18.100.126" port: 38191 } } peers { permanent_uuid: "107f274cdda543038832f06a56ab9130" member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39711 } } peers { permanent_uuid: "3749baa287ed4f249329a74071781edd" member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34629 } } }
I20260130 02:57:49.352450 19291 sys_catalog.cc:458] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4 [sys.catalog]: This master's current role is: LEADER
I20260130 02:57:49.356081 18885 raft_consensus.cc:1240] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4 [term 2 LEADER]: Rejecting Update request from peer 3749baa287ed4f249329a74071781edd for earlier term 1. Current term is 2. Ops: []
I20260130 02:57:49.364015 19293 consensus_queue.cc:1059] T 00000000000000000000000000000000 P 3749baa287ed4f249329a74071781edd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "658299197a424f53a37f14fadf089ae4" member_type: VOTER last_known_addr { host: "127.18.100.126" port: 38191 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260130 02:57:49.366415 19293 consensus_queue.cc:1243] T 00000000000000000000000000000000 P 3749baa287ed4f249329a74071781edd [LEADER]: Peer 658299197a424f53a37f14fadf089ae4 log is divergent from this leader: its last log entry 2.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20260130 02:57:49.367151 19297 raft_consensus.cc:3055] T 00000000000000000000000000000000 P 3749baa287ed4f249329a74071781edd [term 1 LEADER]: Stepping down as leader of term 1
I20260130 02:57:49.367627 19297 raft_consensus.cc:740] T 00000000000000000000000000000000 P 3749baa287ed4f249329a74071781edd [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3749baa287ed4f249329a74071781edd, State: Running, Role: LEADER
I20260130 02:57:49.368077 19296 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260130 02:57:49.368659 19297 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3749baa287ed4f249329a74071781edd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "658299197a424f53a37f14fadf089ae4" member_type: VOTER last_known_addr { host: "127.18.100.126" port: 38191 } } peers { permanent_uuid: "107f274cdda543038832f06a56ab9130" member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39711 } } peers { permanent_uuid: "3749baa287ed4f249329a74071781edd" member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34629 } }
I20260130 02:57:49.374305 19297 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 3749baa287ed4f249329a74071781edd [term 1 FOLLOWER]: Advancing to term 2
I20260130 02:57:49.387558 19296 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260130 02:57:49.391803 19296 catalog_manager.cc:1269] Loaded cluster ID: db1c0e7709154c10a8c9a0ea68ffe38b
I20260130 02:57:49.392212 19296 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260130 02:57:49.392606 19270 heartbeater.cc:499] Master 127.18.100.124:34629 was elected leader, sending a full tablet report...
I20260130 02:57:49.397418 19296 catalog_manager.cc:1514] Loading token signing keys...
I20260130 02:57:49.401834 19296 catalog_manager.cc:6038] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4: Loaded TSK: 0
I20260130 02:57:49.403347 19296 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260130 02:57:49.731460 19040 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 3749baa287ed4f249329a74071781edd [term 2 FOLLOWER]: Refusing update from remote peer 658299197a424f53a37f14fadf089ae4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260130 02:57:49.733502 19291 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3749baa287ed4f249329a74071781edd" member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260130 02:57:49.739306 19297 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3749baa287ed4f249329a74071781edd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 658299197a424f53a37f14fadf089ae4. Latest consensus state: current_term: 2 leader_uuid: "658299197a424f53a37f14fadf089ae4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "658299197a424f53a37f14fadf089ae4" member_type: VOTER last_known_addr { host: "127.18.100.126" port: 38191 } } peers { permanent_uuid: "107f274cdda543038832f06a56ab9130" member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39711 } } peers { permanent_uuid: "3749baa287ed4f249329a74071781edd" member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34629 } } }
I20260130 02:57:49.740101 19297 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3749baa287ed4f249329a74071781edd [sys.catalog]: This master's current role is: FOLLOWER
I20260130 02:57:49.745193 19287 sys_catalog.cc:455] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "658299197a424f53a37f14fadf089ae4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "658299197a424f53a37f14fadf089ae4" member_type: VOTER last_known_addr { host: "127.18.100.126" port: 38191 } } peers { permanent_uuid: "107f274cdda543038832f06a56ab9130" member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39711 } } peers { permanent_uuid: "3749baa287ed4f249329a74071781edd" member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34629 } } }
I20260130 02:57:49.745956 19287 sys_catalog.cc:458] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4 [sys.catalog]: This master's current role is: LEADER
I20260130 02:57:49.747926 18960 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 107f274cdda543038832f06a56ab9130 [term 2 FOLLOWER]: Refusing update from remote peer 658299197a424f53a37f14fadf089ae4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260130 02:57:49.749521 19291 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "107f274cdda543038832f06a56ab9130" member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39711 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260130 02:57:49.757929 19297 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3749baa287ed4f249329a74071781edd [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "658299197a424f53a37f14fadf089ae4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "658299197a424f53a37f14fadf089ae4" member_type: VOTER last_known_addr { host: "127.18.100.126" port: 38191 } } peers { permanent_uuid: "107f274cdda543038832f06a56ab9130" member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39711 } } peers { permanent_uuid: "3749baa287ed4f249329a74071781edd" member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34629 } } }
I20260130 02:57:49.758729 19297 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3749baa287ed4f249329a74071781edd [sys.catalog]: This master's current role is: FOLLOWER
I20260130 02:57:49.764470 19304 sys_catalog.cc:455] T 00000000000000000000000000000000 P 107f274cdda543038832f06a56ab9130 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "658299197a424f53a37f14fadf089ae4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "658299197a424f53a37f14fadf089ae4" member_type: VOTER last_known_addr { host: "127.18.100.126" port: 38191 } } peers { permanent_uuid: "107f274cdda543038832f06a56ab9130" member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39711 } } peers { permanent_uuid: "3749baa287ed4f249329a74071781edd" member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34629 } } }
I20260130 02:57:49.764819 19286 sys_catalog.cc:455] T 00000000000000000000000000000000 P 107f274cdda543038832f06a56ab9130 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 658299197a424f53a37f14fadf089ae4. Latest consensus state: current_term: 2 leader_uuid: "658299197a424f53a37f14fadf089ae4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "658299197a424f53a37f14fadf089ae4" member_type: VOTER last_known_addr { host: "127.18.100.126" port: 38191 } } peers { permanent_uuid: "107f274cdda543038832f06a56ab9130" member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39711 } } peers { permanent_uuid: "3749baa287ed4f249329a74071781edd" member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34629 } } }
I20260130 02:57:49.765416 19304 sys_catalog.cc:458] T 00000000000000000000000000000000 P 107f274cdda543038832f06a56ab9130 [sys.catalog]: This master's current role is: FOLLOWER
I20260130 02:57:49.765616 19286 sys_catalog.cc:458] T 00000000000000000000000000000000 P 107f274cdda543038832f06a56ab9130 [sys.catalog]: This master's current role is: FOLLOWER
I20260130 02:57:50.089229 19090 catalog_manager.cc:1612] T 00000000000000000000000000000000 P 3749baa287ed4f249329a74071781edd: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260130 02:57:50.156222 19287 sys_catalog.cc:455] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "658299197a424f53a37f14fadf089ae4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "658299197a424f53a37f14fadf089ae4" member_type: VOTER last_known_addr { host: "127.18.100.126" port: 38191 } } peers { permanent_uuid: "107f274cdda543038832f06a56ab9130" member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39711 } } peers { permanent_uuid: "3749baa287ed4f249329a74071781edd" member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34629 } } }
I20260130 02:57:50.156957 19287 sys_catalog.cc:458] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4 [sys.catalog]: This master's current role is: LEADER
I20260130 02:57:50.246965 19297 raft_consensus.cc:493] T 00000000000000000000000000000000 P 3749baa287ed4f249329a74071781edd [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 658299197a424f53a37f14fadf089ae4)
I20260130 02:57:50.247542 19297 raft_consensus.cc:515] T 00000000000000000000000000000000 P 3749baa287ed4f249329a74071781edd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "658299197a424f53a37f14fadf089ae4" member_type: VOTER last_known_addr { host: "127.18.100.126" port: 38191 } } peers { permanent_uuid: "107f274cdda543038832f06a56ab9130" member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39711 } } peers { permanent_uuid: "3749baa287ed4f249329a74071781edd" member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34629 } }
I20260130 02:57:50.249601 19297 leader_election.cc:290] T 00000000000000000000000000000000 P 3749baa287ed4f249329a74071781edd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 658299197a424f53a37f14fadf089ae4 (127.18.100.126:38191), 107f274cdda543038832f06a56ab9130 (127.18.100.125:39711)
I20260130 02:57:50.250974 18885 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3749baa287ed4f249329a74071781edd" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "658299197a424f53a37f14fadf089ae4" is_pre_election: true
I20260130 02:57:50.251433 18960 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3749baa287ed4f249329a74071781edd" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "107f274cdda543038832f06a56ab9130" is_pre_election: true
I20260130 02:57:50.253294 19015 leader_election.cc:304] T 00000000000000000000000000000000 P 3749baa287ed4f249329a74071781edd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3749baa287ed4f249329a74071781edd; no voters: 107f274cdda543038832f06a56ab9130, 658299197a424f53a37f14fadf089ae4
I20260130 02:57:50.254199 19297 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 3749baa287ed4f249329a74071781edd [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260130 02:57:50.397982 19275 heartbeater.cc:499] Master 127.18.100.126:38191 was elected leader, sending a full tablet report...
W20260130 02:57:49.328341 19290 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260130 02:57:50.497645 19279 thread.cc:641] GCE (cloud detector) Time spent creating pthread: real 1.173s user 0.487s sys 0.644s
W20260130 02:57:50.498003 19279 thread.cc:608] GCE (cloud detector) Time spent starting thread: real 1.173s user 0.487s sys 0.644s
I20260130 02:57:50.499375 19279 server_base.cc:1060] running on GCE node
W20260130 02:57:50.500859 19306 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260130 02:57:50.501981 19279 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260130 02:57:50.504148 19279 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260130 02:57:50.505487 19279 hybrid_clock.cc:648] HybridClock initialized: now 1769741870505459 us; error 36 us; skew 500 ppm
I20260130 02:57:50.506264 19279 server_base.cc:860] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260130 02:57:50.513307 19279 webserver.cc:492] Webserver started at http://127.18.100.66:43365/ using document root <none> and password file <none>
I20260130 02:57:50.514314 19279 fs_manager.cc:362] Metadata directory not provided
I20260130 02:57:50.514539 19279 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260130 02:57:50.515008 19279 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260130 02:57:50.519342 19279 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-1/data/instance:
uuid: "78c2c46c7520492db35d51c244fb4f74"
format_stamp: "Formatted at 2026-01-30 02:57:50 on dist-test-slave-mn07"
I20260130 02:57:50.520505 19279 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-1/wal/instance:
uuid: "78c2c46c7520492db35d51c244fb4f74"
format_stamp: "Formatted at 2026-01-30 02:57:50 on dist-test-slave-mn07"
I20260130 02:57:50.528571 19279 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.009s sys 0.000s
I20260130 02:57:50.534883 19311 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260130 02:57:50.536326 19279 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260130 02:57:50.536782 19279 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-1/wal
uuid: "78c2c46c7520492db35d51c244fb4f74"
format_stamp: "Formatted at 2026-01-30 02:57:50 on dist-test-slave-mn07"
I20260130 02:57:50.537135 19279 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260130 02:57:50.599032 19279 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260130 02:57:50.600482 19279 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260130 02:57:50.600888 19279 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260130 02:57:50.603394 19279 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260130 02:57:50.607347 19279 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260130 02:57:50.607553 19279 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260130 02:57:50.607795 19279 ts_tablet_manager.cc:616] Registered 0 tablets
I20260130 02:57:50.608049 19279 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260130 02:57:50.661386 19359 raft_consensus.cc:493] T 00000000000000000000000000000000 P 107f274cdda543038832f06a56ab9130 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 658299197a424f53a37f14fadf089ae4)
I20260130 02:57:50.661895 19359 raft_consensus.cc:515] T 00000000000000000000000000000000 P 107f274cdda543038832f06a56ab9130 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "658299197a424f53a37f14fadf089ae4" member_type: VOTER last_known_addr { host: "127.18.100.126" port: 38191 } } peers { permanent_uuid: "107f274cdda543038832f06a56ab9130" member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39711 } } peers { permanent_uuid: "3749baa287ed4f249329a74071781edd" member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34629 } }
I20260130 02:57:50.665021 19359 leader_election.cc:290] T 00000000000000000000000000000000 P 107f274cdda543038832f06a56ab9130 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 658299197a424f53a37f14fadf089ae4 (127.18.100.126:38191), 3749baa287ed4f249329a74071781edd (127.18.100.124:34629)
I20260130 02:57:50.664968 18885 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "107f274cdda543038832f06a56ab9130" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "658299197a424f53a37f14fadf089ae4" is_pre_election: true
I20260130 02:57:50.665071 19040 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "107f274cdda543038832f06a56ab9130" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3749baa287ed4f249329a74071781edd" is_pre_election: true
I20260130 02:57:50.666707 18934 leader_election.cc:304] T 00000000000000000000000000000000 P 107f274cdda543038832f06a56ab9130 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 107f274cdda543038832f06a56ab9130; no voters: 3749baa287ed4f249329a74071781edd, 658299197a424f53a37f14fadf089ae4
I20260130 02:57:50.667474 19359 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 107f274cdda543038832f06a56ab9130 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260130 02:57:50.734030 19279 rpc_server.cc:307] RPC server started. Bound to: 127.18.100.66:40767
I20260130 02:57:50.734169 19424 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.100.66:40767 every 8 connection(s)
I20260130 02:57:50.736497 19279 server_base.cc:1192] Dumped server information to /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-1/data/info.pb
I20260130 02:57:50.745179 18833 external_mini_cluster.cc:1428] Started /tmp/dist-test-tasktGe2wI/build/tsan/bin/kudu as pid 19279
I20260130 02:57:50.745914 18833 external_mini_cluster.cc:1442] Reading /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-1/wal/instance
I20260130 02:57:50.759361 18833 external_mini_cluster.cc:1366] Running /tmp/dist-test-tasktGe2wI/build/tsan/bin/kudu
/tmp/dist-test-tasktGe2wI/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.100.67:0
--local_ip_for_outbound_sockets=127.18.100.67
--webserver_interface=127.18.100.67
--webserver_port=0
--tserver_master_addrs=127.18.100.126:38191,127.18.100.125:39711,127.18.100.124:34629
--builtin_ntp_servers=127.18.100.84:46033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20260130 02:57:50.765988 19435 raft_consensus.cc:493] T 00000000000000000000000000000000 P 3749baa287ed4f249329a74071781edd [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 658299197a424f53a37f14fadf089ae4)
I20260130 02:57:50.766605 19435 raft_consensus.cc:515] T 00000000000000000000000000000000 P 3749baa287ed4f249329a74071781edd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "658299197a424f53a37f14fadf089ae4" member_type: VOTER last_known_addr { host: "127.18.100.126" port: 38191 } } peers { permanent_uuid: "107f274cdda543038832f06a56ab9130" member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39711 } } peers { permanent_uuid: "3749baa287ed4f249329a74071781edd" member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34629 } }
I20260130 02:57:50.769556 18885 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3749baa287ed4f249329a74071781edd" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "658299197a424f53a37f14fadf089ae4" is_pre_election: true
I20260130 02:57:50.772532 18960 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3749baa287ed4f249329a74071781edd" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "107f274cdda543038832f06a56ab9130" is_pre_election: true
I20260130 02:57:50.774144 19015 leader_election.cc:304] T 00000000000000000000000000000000 P 3749baa287ed4f249329a74071781edd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3749baa287ed4f249329a74071781edd; no voters: 107f274cdda543038832f06a56ab9130, 658299197a424f53a37f14fadf089ae4
I20260130 02:57:50.778471 19435 leader_election.cc:290] T 00000000000000000000000000000000 P 3749baa287ed4f249329a74071781edd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 658299197a424f53a37f14fadf089ae4 (127.18.100.126:38191), 107f274cdda543038832f06a56ab9130 (127.18.100.125:39711)
I20260130 02:57:50.779090 19435 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 3749baa287ed4f249329a74071781edd [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260130 02:57:50.793469 19427 heartbeater.cc:344] Connected to a master server at 127.18.100.126:38191
I20260130 02:57:50.793973 19427 heartbeater.cc:461] Registering TS with master...
I20260130 02:57:50.795127 19426 heartbeater.cc:344] Connected to a master server at 127.18.100.125:39711
I20260130 02:57:50.795354 19427 heartbeater.cc:507] Master 127.18.100.126:38191 requested a full tablet report, sending...
I20260130 02:57:50.795486 19426 heartbeater.cc:461] Registering TS with master...
I20260130 02:57:50.796319 19426 heartbeater.cc:507] Master 127.18.100.125:39711 requested a full tablet report, sending...
I20260130 02:57:50.797993 19425 heartbeater.cc:344] Connected to a master server at 127.18.100.124:34629
I20260130 02:57:50.798111 18875 ts_manager.cc:194] Registered new tserver with Master: 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767)
I20260130 02:57:50.798327 19425 heartbeater.cc:461] Registering TS with master...
I20260130 02:57:50.798919 18950 ts_manager.cc:194] Registered new tserver with Master: 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767)
I20260130 02:57:50.799175 19425 heartbeater.cc:507] Master 127.18.100.124:34629 requested a full tablet report, sending...
I20260130 02:57:50.800668 18875 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.18.100.66:59691
I20260130 02:57:50.801506 19030 ts_manager.cc:194] Registered new tserver with Master: 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767)
W20260130 02:57:51.085923 19436 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260130 02:57:51.086414 19436 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260130 02:57:51.087054 19436 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260130 02:57:51.118604 19436 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260130 02:57:51.119537 19436 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.100.67
I20260130 02:57:51.165552 19436 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.100.84:46033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.100.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.100.67
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.18.100.126:38191,127.18.100.125:39711,127.18.100.124:34629
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.100.67
--log_dir=/tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type FASTDEBUG
built by None at 30 Jan 2026 02:43:21 UTC on 5fd53c4cbb9d
build id 10257
TSAN enabled
I20260130 02:57:51.166934 19436 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260130 02:57:51.168599 19436 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260130 02:57:51.180692 19443 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260130 02:57:51.244002 19449 raft_consensus.cc:493] T 00000000000000000000000000000000 P 107f274cdda543038832f06a56ab9130 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 658299197a424f53a37f14fadf089ae4)
I20260130 02:57:51.245751 19449 raft_consensus.cc:515] T 00000000000000000000000000000000 P 107f274cdda543038832f06a56ab9130 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "658299197a424f53a37f14fadf089ae4" member_type: VOTER last_known_addr { host: "127.18.100.126" port: 38191 } } peers { permanent_uuid: "107f274cdda543038832f06a56ab9130" member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39711 } } peers { permanent_uuid: "3749baa287ed4f249329a74071781edd" member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34629 } }
I20260130 02:57:51.249022 18885 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "107f274cdda543038832f06a56ab9130" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "658299197a424f53a37f14fadf089ae4" is_pre_election: true
I20260130 02:57:51.249207 19040 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "107f274cdda543038832f06a56ab9130" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3749baa287ed4f249329a74071781edd" is_pre_election: true
I20260130 02:57:51.252326 18935 leader_election.cc:304] T 00000000000000000000000000000000 P 107f274cdda543038832f06a56ab9130 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 107f274cdda543038832f06a56ab9130; no voters: 3749baa287ed4f249329a74071781edd, 658299197a424f53a37f14fadf089ae4
I20260130 02:57:51.258613 19449 leader_election.cc:290] T 00000000000000000000000000000000 P 107f274cdda543038832f06a56ab9130 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 658299197a424f53a37f14fadf089ae4 (127.18.100.126:38191), 3749baa287ed4f249329a74071781edd (127.18.100.124:34629)
I20260130 02:57:51.259768 19449 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 107f274cdda543038832f06a56ab9130 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260130 02:57:51.304397 19427 heartbeater.cc:499] Master 127.18.100.126:38191 was elected leader, sending a full tablet report...
I20260130 02:57:51.394217 19451 raft_consensus.cc:493] T 00000000000000000000000000000000 P 3749baa287ed4f249329a74071781edd [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 658299197a424f53a37f14fadf089ae4)
I20260130 02:57:51.401691 19451 raft_consensus.cc:515] T 00000000000000000000000000000000 P 3749baa287ed4f249329a74071781edd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "658299197a424f53a37f14fadf089ae4" member_type: VOTER last_known_addr { host: "127.18.100.126" port: 38191 } } peers { permanent_uuid: "107f274cdda543038832f06a56ab9130" member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39711 } } peers { permanent_uuid: "3749baa287ed4f249329a74071781edd" member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34629 } }
I20260130 02:57:51.407332 18885 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3749baa287ed4f249329a74071781edd" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "658299197a424f53a37f14fadf089ae4" is_pre_election: true
I20260130 02:57:51.419345 18960 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3749baa287ed4f249329a74071781edd" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "107f274cdda543038832f06a56ab9130" is_pre_election: true
I20260130 02:57:51.421061 19015 leader_election.cc:304] T 00000000000000000000000000000000 P 3749baa287ed4f249329a74071781edd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3749baa287ed4f249329a74071781edd; no voters: 107f274cdda543038832f06a56ab9130, 658299197a424f53a37f14fadf089ae4
I20260130 02:57:51.426558 19451 leader_election.cc:290] T 00000000000000000000000000000000 P 3749baa287ed4f249329a74071781edd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 658299197a424f53a37f14fadf089ae4 (127.18.100.126:38191), 107f274cdda543038832f06a56ab9130 (127.18.100.125:39711)
I20260130 02:57:51.427690 19451 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 3749baa287ed4f249329a74071781edd [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260130 02:57:52.170125 19454 raft_consensus.cc:493] T 00000000000000000000000000000000 P 107f274cdda543038832f06a56ab9130 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 658299197a424f53a37f14fadf089ae4)
I20260130 02:57:52.182017 19454 raft_consensus.cc:515] T 00000000000000000000000000000000 P 107f274cdda543038832f06a56ab9130 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "658299197a424f53a37f14fadf089ae4" member_type: VOTER last_known_addr { host: "127.18.100.126" port: 38191 } } peers { permanent_uuid: "107f274cdda543038832f06a56ab9130" member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39711 } } peers { permanent_uuid: "3749baa287ed4f249329a74071781edd" member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34629 } }
I20260130 02:57:52.185588 18885 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "107f274cdda543038832f06a56ab9130" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "658299197a424f53a37f14fadf089ae4" is_pre_election: true
I20260130 02:57:52.189509 19040 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "107f274cdda543038832f06a56ab9130" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3749baa287ed4f249329a74071781edd" is_pre_election: true
I20260130 02:57:52.191453 18935 leader_election.cc:304] T 00000000000000000000000000000000 P 107f274cdda543038832f06a56ab9130 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 107f274cdda543038832f06a56ab9130; no voters: 3749baa287ed4f249329a74071781edd, 658299197a424f53a37f14fadf089ae4
I20260130 02:57:52.193023 19454 leader_election.cc:290] T 00000000000000000000000000000000 P 107f274cdda543038832f06a56ab9130 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 658299197a424f53a37f14fadf089ae4 (127.18.100.126:38191), 3749baa287ed4f249329a74071781edd (127.18.100.124:34629)
I20260130 02:57:52.193353 19455 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 107f274cdda543038832f06a56ab9130 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260130 02:57:51.182376 19444 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260130 02:57:52.477171 19446 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260130 02:57:52.480212 19445 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1293 milliseconds
W20260130 02:57:52.480417 19436 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.300s user 0.496s sys 0.759s
W20260130 02:57:52.480772 19436 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.300s user 0.496s sys 0.759s
I20260130 02:57:52.481045 19436 server_base.cc:1055] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20260130 02:57:52.482456 19436 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260130 02:57:52.485090 19436 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260130 02:57:52.486642 19436 hybrid_clock.cc:648] HybridClock initialized: now 1769741872486610 us; error 58 us; skew 500 ppm
I20260130 02:57:52.487684 19436 server_base.cc:860] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260130 02:57:52.495985 19436 webserver.cc:492] Webserver started at http://127.18.100.67:43627/ using document root <none> and password file <none>
I20260130 02:57:52.497296 19436 fs_manager.cc:362] Metadata directory not provided
I20260130 02:57:52.497570 19436 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260130 02:57:52.498142 19436 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260130 02:57:52.504639 19436 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-2/data/instance:
uuid: "d870164263bb44b6b780a557daea3656"
format_stamp: "Formatted at 2026-01-30 02:57:52 on dist-test-slave-mn07"
I20260130 02:57:52.506167 19436 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-2/wal/instance:
uuid: "d870164263bb44b6b780a557daea3656"
format_stamp: "Formatted at 2026-01-30 02:57:52 on dist-test-slave-mn07"
I20260130 02:57:52.515993 19436 fs_manager.cc:696] Time spent creating directory manager: real 0.009s user 0.008s sys 0.000s
I20260130 02:57:52.523422 19460 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260130 02:57:52.524673 19436 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.001s sys 0.004s
I20260130 02:57:52.525079 19436 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-2/wal
uuid: "d870164263bb44b6b780a557daea3656"
format_stamp: "Formatted at 2026-01-30 02:57:52 on dist-test-slave-mn07"
I20260130 02:57:52.525543 19436 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260130 02:57:52.607129 19436 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260130 02:57:52.608636 19436 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260130 02:57:52.609014 19436 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260130 02:57:52.611462 19436 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260130 02:57:52.615451 19436 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260130 02:57:52.615653 19436 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260130 02:57:52.615851 19436 ts_tablet_manager.cc:616] Registered 0 tablets
I20260130 02:57:52.615983 19436 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260130 02:57:52.746381 19436 rpc_server.cc:307] RPC server started. Bound to: 127.18.100.67:39571
I20260130 02:57:52.746492 19572 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.100.67:39571 every 8 connection(s)
I20260130 02:57:52.748978 19436 server_base.cc:1192] Dumped server information to /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-2/data/info.pb
I20260130 02:57:52.749997 18833 external_mini_cluster.cc:1428] Started /tmp/dist-test-tasktGe2wI/build/tsan/bin/kudu as pid 19436
I20260130 02:57:52.750451 18833 external_mini_cluster.cc:1442] Reading /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-2/wal/instance
I20260130 02:57:52.792922 19574 heartbeater.cc:344] Connected to a master server at 127.18.100.125:39711
I20260130 02:57:52.793448 19574 heartbeater.cc:461] Registering TS with master...
I20260130 02:57:52.794754 19574 heartbeater.cc:507] Master 127.18.100.125:39711 requested a full tablet report, sending...
I20260130 02:57:52.796511 19575 heartbeater.cc:344] Connected to a master server at 127.18.100.126:38191
I20260130 02:57:52.796850 19575 heartbeater.cc:461] Registering TS with master...
I20260130 02:57:52.797695 19575 heartbeater.cc:507] Master 127.18.100.126:38191 requested a full tablet report, sending...
I20260130 02:57:52.797658 18949 ts_manager.cc:194] Registered new tserver with Master: d870164263bb44b6b780a557daea3656 (127.18.100.67:39571)
I20260130 02:57:52.797809 19573 heartbeater.cc:344] Connected to a master server at 127.18.100.124:34629
I20260130 02:57:52.798110 19573 heartbeater.cc:461] Registering TS with master...
I20260130 02:57:52.798924 19573 heartbeater.cc:507] Master 127.18.100.124:34629 requested a full tablet report, sending...
I20260130 02:57:52.799612 18874 ts_manager.cc:194] Registered new tserver with Master: d870164263bb44b6b780a557daea3656 (127.18.100.67:39571)
I20260130 02:57:52.801194 19030 ts_manager.cc:194] Registered new tserver with Master: d870164263bb44b6b780a557daea3656 (127.18.100.67:39571)
I20260130 02:57:52.801751 18874 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.18.100.67:47213
I20260130 02:57:52.817989 18833 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20260130 02:57:52.819926 19583 raft_consensus.cc:493] T 00000000000000000000000000000000 P 3749baa287ed4f249329a74071781edd [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 658299197a424f53a37f14fadf089ae4)
I20260130 02:57:52.820410 19583 raft_consensus.cc:515] T 00000000000000000000000000000000 P 3749baa287ed4f249329a74071781edd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "658299197a424f53a37f14fadf089ae4" member_type: VOTER last_known_addr { host: "127.18.100.126" port: 38191 } } peers { permanent_uuid: "107f274cdda543038832f06a56ab9130" member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39711 } } peers { permanent_uuid: "3749baa287ed4f249329a74071781edd" member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34629 } }
I20260130 02:57:52.823016 18885 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3749baa287ed4f249329a74071781edd" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "658299197a424f53a37f14fadf089ae4" is_pre_election: true
I20260130 02:57:52.823292 19583 leader_election.cc:290] T 00000000000000000000000000000000 P 3749baa287ed4f249329a74071781edd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 658299197a424f53a37f14fadf089ae4 (127.18.100.126:38191), 107f274cdda543038832f06a56ab9130 (127.18.100.125:39711)
I20260130 02:57:52.823468 18960 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3749baa287ed4f249329a74071781edd" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "107f274cdda543038832f06a56ab9130" is_pre_election: true
I20260130 02:57:52.824939 19015 leader_election.cc:304] T 00000000000000000000000000000000 P 3749baa287ed4f249329a74071781edd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3749baa287ed4f249329a74071781edd; no voters: 107f274cdda543038832f06a56ab9130, 658299197a424f53a37f14fadf089ae4
I20260130 02:57:52.825732 19583 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 3749baa287ed4f249329a74071781edd [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260130 02:57:52.870549 18874 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50284:
name: "default.Table_bb299970e3d74b969c9d4f7cca2632cf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260130 02:57:52.872797 18872 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50284:
name: "default.Table_55080a98b6b5457daa87e6f365eea715"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260130 02:57:52.874395 18874 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bb299970e3d74b969c9d4f7cca2632cf in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260130 02:57:52.870716 18873 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50284:
name: "default.Table_dfd464c60be84f42a354a58701674700"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260130 02:57:52.877285 18873 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_dfd464c60be84f42a354a58701674700 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260130 02:57:52.874450 18872 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_55080a98b6b5457daa87e6f365eea715 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260130 02:57:52.871001 18875 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50284:
name: "default.Table_c4e56dff52d245fbb15591cfa399a66a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260130 02:57:52.880795 18875 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c4e56dff52d245fbb15591cfa399a66a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260130 02:57:52.901018 18833 external_mini_cluster.cc:1658] Killing /tmp/dist-test-tasktGe2wI/build/tsan/bin/kudu with pid 18987
I20260130 02:57:52.933255 18872 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50284:
table { table_name: "default.Table_55080a98b6b5457daa87e6f365eea715" } new_table_name: "default.Table_eb61588fb4964de39da6127edd2bd501" modify_external_catalogs: true
W20260130 02:57:52.937383 18860 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.18.100.124:34629 (error 108) [suppressed 12 similar messages]
W20260130 02:57:52.938668 19270 heartbeater.cc:646] Failed to heartbeat to 127.18.100.124:34629 (0 consecutive failures): Network error: Failed to send heartbeat to master: recv got EOF from unknown peer (error 108)
W20260130 02:57:52.966295 19616 negotiation.cc:337] Failed RPC negotiation. Trace:
0130 02:57:52.945134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.100.124:34629 (local address 127.0.0.1:40172)
0130 02:57:52.960070 (+ 14936us) negotiation.cc:107] Waiting for socket to connect
0130 02:57:52.960304 (+ 234us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.100.124:34629: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":13506,"thread_start_us":13282,"threads_started":1}
W20260130 02:57:52.967335 18860 consensus_peers.cc:597] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4 -> Peer 3749baa287ed4f249329a74071781edd (127.18.100.124:34629): Couldn't send request to peer 3749baa287ed4f249329a74071781edd. Status: Network error: Client connection negotiation failed: client connection to 127.18.100.124:34629: connect: Connection reset by peer (error 104). This is attempt 1: this message will repeat every 5th retry.
I20260130 02:57:52.968582 18875 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50284:
table { table_name: "default.Table_dfd464c60be84f42a354a58701674700" } new_table_name: "default.Table_d2eb867c8c1f4713b405b04522346ef3" modify_external_catalogs: true
I20260130 02:57:52.972263 18874 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50284:
table { table_name: "default.Table_bb299970e3d74b969c9d4f7cca2632cf" } modify_external_catalogs: true
I20260130 02:57:52.972702 18873 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50284:
table { table_name: "default.Table_c4e56dff52d245fbb15591cfa399a66a" } modify_external_catalogs: true
I20260130 02:57:52.973019 18874 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50284:
table { table_name: "default.Table_bb299970e3d74b969c9d4f7cca2632cf" } modify_external_catalogs: true
I20260130 02:57:52.973210 18873 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50284:
table { table_name: "default.Table_c4e56dff52d245fbb15591cfa399a66a" } modify_external_catalogs: true
W20260130 02:57:52.975848 18872 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3796b2db28ff4bb78c3f225f8ab9baae has no consensus state
I20260130 02:57:53.006021 19205 tablet_service.cc:1505] Processing CreateTablet for tablet 3796b2db28ff4bb78c3f225f8ab9baae (DEFAULT_TABLE table=default.Table_eb61588fb4964de39da6127edd2bd501 [id=acd866a00ee34b529227c16091417594]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:57:53.008364 19205 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3796b2db28ff4bb78c3f225f8ab9baae. 1 dirs total, 0 dirs full, 0 dirs failed
W20260130 02:57:53.014624 18859 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet 3796b2db28ff4bb78c3f225f8ab9baae (table default.Table_eb61588fb4964de39da6127edd2bd501 [id=acd866a00ee34b529227c16091417594], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 3796b2db28ff4bb78c3f225f8ab9baae consensus state has no leader
W20260130 02:57:53.029816 18875 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 96584ca174b140f7ad1a2ca698baca71 has no consensus state
I20260130 02:57:53.031862 18874 catalog_manager.cc:5936] Not sending DeleteTablet requests; no consensus state for tablet 81dcc61cbcf848efb8bc5851a4f0f4aa
I20260130 02:57:53.034782 18873 catalog_manager.cc:5936] Not sending DeleteTablet requests; no consensus state for tablet e860966f68d746d4b32330ddbb75c5f0
I20260130 02:57:53.049175 19360 tablet_service.cc:1505] Processing CreateTablet for tablet 3796b2db28ff4bb78c3f225f8ab9baae (DEFAULT_TABLE table=default.Table_eb61588fb4964de39da6127edd2bd501 [id=acd866a00ee34b529227c16091417594]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:57:53.051170 19360 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3796b2db28ff4bb78c3f225f8ab9baae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:57:53.056628 19508 tablet_service.cc:1505] Processing CreateTablet for tablet 3796b2db28ff4bb78c3f225f8ab9baae (DEFAULT_TABLE table=default.Table_eb61588fb4964de39da6127edd2bd501 [id=acd866a00ee34b529227c16091417594]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:57:53.059144 19508 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3796b2db28ff4bb78c3f225f8ab9baae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:57:53.076639 19204 tablet_service.cc:1505] Processing CreateTablet for tablet 96584ca174b140f7ad1a2ca698baca71 (DEFAULT_TABLE table=default.Table_d2eb867c8c1f4713b405b04522346ef3 [id=1515db389f284e2f9ed1a3cd0061d0f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:57:53.078090 19204 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 96584ca174b140f7ad1a2ca698baca71. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:57:53.079950 19624 tablet_bootstrap.cc:492] T 3796b2db28ff4bb78c3f225f8ab9baae P 371a36d219164974abdf848841528152: Bootstrap starting.
I20260130 02:57:53.080364 19507 tablet_service.cc:1505] Processing CreateTablet for tablet 96584ca174b140f7ad1a2ca698baca71 (DEFAULT_TABLE table=default.Table_d2eb867c8c1f4713b405b04522346ef3 [id=1515db389f284e2f9ed1a3cd0061d0f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:57:53.081550 19507 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 96584ca174b140f7ad1a2ca698baca71. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:57:53.083760 19358 tablet_service.cc:1505] Processing CreateTablet for tablet 96584ca174b140f7ad1a2ca698baca71 (DEFAULT_TABLE table=default.Table_d2eb867c8c1f4713b405b04522346ef3 [id=1515db389f284e2f9ed1a3cd0061d0f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:57:53.084867 19358 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 96584ca174b140f7ad1a2ca698baca71. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:57:53.096221 19625 tablet_bootstrap.cc:492] T 3796b2db28ff4bb78c3f225f8ab9baae P d870164263bb44b6b780a557daea3656: Bootstrap starting.
I20260130 02:57:53.103101 19624 tablet_bootstrap.cc:654] T 3796b2db28ff4bb78c3f225f8ab9baae P 371a36d219164974abdf848841528152: Neither blocks nor log segments found. Creating new log.
I20260130 02:57:53.103276 19625 tablet_bootstrap.cc:654] T 3796b2db28ff4bb78c3f225f8ab9baae P d870164263bb44b6b780a557daea3656: Neither blocks nor log segments found. Creating new log.
I20260130 02:57:53.105615 19625 log.cc:826] T 3796b2db28ff4bb78c3f225f8ab9baae P d870164263bb44b6b780a557daea3656: Log is configured to *not* fsync() on all Append() calls
I20260130 02:57:53.111569 19624 log.cc:826] T 3796b2db28ff4bb78c3f225f8ab9baae P 371a36d219164974abdf848841528152: Log is configured to *not* fsync() on all Append() calls
I20260130 02:57:53.112461 19626 tablet_bootstrap.cc:492] T 3796b2db28ff4bb78c3f225f8ab9baae P 78c2c46c7520492db35d51c244fb4f74: Bootstrap starting.
I20260130 02:57:53.118289 19625 tablet_bootstrap.cc:492] T 3796b2db28ff4bb78c3f225f8ab9baae P d870164263bb44b6b780a557daea3656: No bootstrap required, opened a new log
I20260130 02:57:53.118896 19625 ts_tablet_manager.cc:1403] T 3796b2db28ff4bb78c3f225f8ab9baae P d870164263bb44b6b780a557daea3656: Time spent bootstrapping tablet: real 0.023s user 0.019s sys 0.000s
I20260130 02:57:53.121443 19626 tablet_bootstrap.cc:654] T 3796b2db28ff4bb78c3f225f8ab9baae P 78c2c46c7520492db35d51c244fb4f74: Neither blocks nor log segments found. Creating new log.
I20260130 02:57:53.121603 19624 tablet_bootstrap.cc:492] T 3796b2db28ff4bb78c3f225f8ab9baae P 371a36d219164974abdf848841528152: No bootstrap required, opened a new log
I20260130 02:57:53.122236 19624 ts_tablet_manager.cc:1403] T 3796b2db28ff4bb78c3f225f8ab9baae P 371a36d219164974abdf848841528152: Time spent bootstrapping tablet: real 0.043s user 0.013s sys 0.005s
I20260130 02:57:53.123858 19626 log.cc:826] T 3796b2db28ff4bb78c3f225f8ab9baae P 78c2c46c7520492db35d51c244fb4f74: Log is configured to *not* fsync() on all Append() calls
I20260130 02:57:53.134125 18874 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50284:
name: "default.Table_b2b8ae6f8ee74fc2be70b4cd5d16782e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260130 02:57:53.134860 19626 tablet_bootstrap.cc:492] T 3796b2db28ff4bb78c3f225f8ab9baae P 78c2c46c7520492db35d51c244fb4f74: No bootstrap required, opened a new log
I20260130 02:57:53.135385 19626 ts_tablet_manager.cc:1403] T 3796b2db28ff4bb78c3f225f8ab9baae P 78c2c46c7520492db35d51c244fb4f74: Time spent bootstrapping tablet: real 0.024s user 0.009s sys 0.011s
W20260130 02:57:53.135877 18874 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b2b8ae6f8ee74fc2be70b4cd5d16782e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260130 02:57:53.147630 19625 raft_consensus.cc:359] T 3796b2db28ff4bb78c3f225f8ab9baae P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:57:53.149694 19625 raft_consensus.cc:385] T 3796b2db28ff4bb78c3f225f8ab9baae P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:57:53.150139 19625 raft_consensus.cc:740] T 3796b2db28ff4bb78c3f225f8ab9baae P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d870164263bb44b6b780a557daea3656, State: Initialized, Role: FOLLOWER
I20260130 02:57:53.151295 19625 consensus_queue.cc:260] T 3796b2db28ff4bb78c3f225f8ab9baae P d870164263bb44b6b780a557daea3656 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:57:53.163470 19626 raft_consensus.cc:359] T 3796b2db28ff4bb78c3f225f8ab9baae P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:57:53.164497 19626 raft_consensus.cc:385] T 3796b2db28ff4bb78c3f225f8ab9baae P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:57:53.164844 19626 raft_consensus.cc:740] T 3796b2db28ff4bb78c3f225f8ab9baae P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78c2c46c7520492db35d51c244fb4f74, State: Initialized, Role: FOLLOWER
I20260130 02:57:53.164717 18874 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50284:
name: "default.Table_192f0cbcf82c4a0db6d5e5adc402e01a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260130 02:57:53.165132 18873 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50284:
name: "default.Table_5b8236964cd94924b945ec3d68a500fc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260130 02:57:53.166399 18874 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_192f0cbcf82c4a0db6d5e5adc402e01a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260130 02:57:53.166149 19626 consensus_queue.cc:260] T 3796b2db28ff4bb78c3f225f8ab9baae P 78c2c46c7520492db35d51c244fb4f74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
W20260130 02:57:53.166828 18873 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5b8236964cd94924b945ec3d68a500fc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260130 02:57:53.171926 18875 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50284:
name: "default.Table_418a00b15bd443fcaa3033df74c4f33b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260130 02:57:53.173754 18875 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_418a00b15bd443fcaa3033df74c4f33b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260130 02:57:53.173914 19573 heartbeater.cc:646] Failed to heartbeat to 127.18.100.124:34629 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.18.100.124:34629: connect: Connection refused (error 111)
I20260130 02:57:53.176309 19575 heartbeater.cc:499] Master 127.18.100.126:38191 was elected leader, sending a full tablet report...
I20260130 02:57:53.180570 19625 ts_tablet_manager.cc:1434] T 3796b2db28ff4bb78c3f225f8ab9baae P d870164263bb44b6b780a557daea3656: Time spent starting tablet: real 0.061s user 0.035s sys 0.000s
I20260130 02:57:53.181473 19625 tablet_bootstrap.cc:492] T 96584ca174b140f7ad1a2ca698baca71 P d870164263bb44b6b780a557daea3656: Bootstrap starting.
I20260130 02:57:53.181733 19360 tablet_service.cc:1505] Processing CreateTablet for tablet 00e6ed81b43d4e74aa6d648fdc1f6c65 (DEFAULT_TABLE table=default.Table_b2b8ae6f8ee74fc2be70b4cd5d16782e [id=18cadca4c17f42d4bcbb7db8c12045bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:57:53.182812 19360 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00e6ed81b43d4e74aa6d648fdc1f6c65. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:57:53.183336 19204 tablet_service.cc:1505] Processing CreateTablet for tablet 00e6ed81b43d4e74aa6d648fdc1f6c65 (DEFAULT_TABLE table=default.Table_b2b8ae6f8ee74fc2be70b4cd5d16782e [id=18cadca4c17f42d4bcbb7db8c12045bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:57:53.184396 19204 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00e6ed81b43d4e74aa6d648fdc1f6c65. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:57:53.187229 18871 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50284:
table { table_name: "default.Table_d2eb867c8c1f4713b405b04522346ef3" } new_table_name: "default.Table_16126d9be1d1434783c533e31b977783" modify_external_catalogs: true
I20260130 02:57:53.193476 19507 tablet_service.cc:1505] Processing CreateTablet for tablet 00e6ed81b43d4e74aa6d648fdc1f6c65 (DEFAULT_TABLE table=default.Table_b2b8ae6f8ee74fc2be70b4cd5d16782e [id=18cadca4c17f42d4bcbb7db8c12045bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:57:53.194506 19507 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00e6ed81b43d4e74aa6d648fdc1f6c65. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:57:53.182238 18872 catalog_manager.cc:5746] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571) does not have the latest schema for tablet 3796b2db28ff4bb78c3f225f8ab9baae (table default.Table_eb61588fb4964de39da6127edd2bd501 [id=acd866a00ee34b529227c16091417594]). Expected version 1 got 0
I20260130 02:57:53.195463 18872 catalog_manager.cc:5746] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571) does not have the latest schema for tablet 96584ca174b140f7ad1a2ca698baca71 (table default.Table_d2eb867c8c1f4713b405b04522346ef3 [id=1515db389f284e2f9ed1a3cd0061d0f1]). Expected version 1 got 0
I20260130 02:57:53.198881 19625 tablet_bootstrap.cc:654] T 96584ca174b140f7ad1a2ca698baca71 P d870164263bb44b6b780a557daea3656: Neither blocks nor log segments found. Creating new log.
I20260130 02:57:53.221333 19626 ts_tablet_manager.cc:1434] T 3796b2db28ff4bb78c3f225f8ab9baae P 78c2c46c7520492db35d51c244fb4f74: Time spent starting tablet: real 0.086s user 0.030s sys 0.005s
I20260130 02:57:53.223826 19624 raft_consensus.cc:359] T 3796b2db28ff4bb78c3f225f8ab9baae P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:57:53.224887 19624 raft_consensus.cc:385] T 3796b2db28ff4bb78c3f225f8ab9baae P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:57:53.225217 19624 raft_consensus.cc:740] T 3796b2db28ff4bb78c3f225f8ab9baae P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 371a36d219164974abdf848841528152, State: Initialized, Role: FOLLOWER
I20260130 02:57:53.226317 19624 consensus_queue.cc:260] T 3796b2db28ff4bb78c3f225f8ab9baae P 371a36d219164974abdf848841528152 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:57:53.229624 19630 raft_consensus.cc:493] T 3796b2db28ff4bb78c3f225f8ab9baae P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:57:53.230131 19630 raft_consensus.cc:515] T 3796b2db28ff4bb78c3f225f8ab9baae P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:57:53.232685 19626 tablet_bootstrap.cc:492] T 96584ca174b140f7ad1a2ca698baca71 P 78c2c46c7520492db35d51c244fb4f74: Bootstrap starting.
I20260130 02:57:53.236552 19630 leader_election.cc:290] T 3796b2db28ff4bb78c3f225f8ab9baae P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767), 371a36d219164974abdf848841528152 (127.18.100.65:42021)
I20260130 02:57:53.241735 18870 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50284:
table { table_name: "default.Table_192f0cbcf82c4a0db6d5e5adc402e01a" } new_table_name: "default.Table_76c537f09c384533b1c632de353f4dbb" modify_external_catalogs: true
I20260130 02:57:53.244625 18875 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50284:
table { table_name: "default.Table_418a00b15bd443fcaa3033df74c4f33b" } modify_external_catalogs: true
I20260130 02:57:53.245184 18875 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50284:
table { table_name: "default.Table_418a00b15bd443fcaa3033df74c4f33b" } modify_external_catalogs: true
I20260130 02:57:53.247123 18869 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50284:
table { table_name: "default.Table_eb61588fb4964de39da6127edd2bd501" } modify_external_catalogs: true
I20260130 02:57:53.247757 18869 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50284:
table { table_name: "default.Table_eb61588fb4964de39da6127edd2bd501" } modify_external_catalogs: true
I20260130 02:57:53.250406 19624 ts_tablet_manager.cc:1434] T 3796b2db28ff4bb78c3f225f8ab9baae P 371a36d219164974abdf848841528152: Time spent starting tablet: real 0.128s user 0.030s sys 0.007s
W20260130 02:57:53.253908 19430 tablet.cc:2378] T 3796b2db28ff4bb78c3f225f8ab9baae P 78c2c46c7520492db35d51c244fb4f74: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260130 02:57:53.255218 19624 tablet_bootstrap.cc:492] T 96584ca174b140f7ad1a2ca698baca71 P 371a36d219164974abdf848841528152: Bootstrap starting.
I20260130 02:57:53.261149 19624 tablet_bootstrap.cc:654] T 96584ca174b140f7ad1a2ca698baca71 P 371a36d219164974abdf848841528152: Neither blocks nor log segments found. Creating new log.
I20260130 02:57:53.263016 19225 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3796b2db28ff4bb78c3f225f8ab9baae" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152" is_pre_election: true
I20260130 02:57:53.263903 19225 raft_consensus.cc:2468] T 3796b2db28ff4bb78c3f225f8ab9baae P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d870164263bb44b6b780a557daea3656 in term 0.
W20260130 02:57:53.265723 18871 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 96584ca174b140f7ad1a2ca698baca71 consensus state has no leader
I20260130 02:57:53.266297 19625 tablet_bootstrap.cc:492] T 96584ca174b140f7ad1a2ca698baca71 P d870164263bb44b6b780a557daea3656: No bootstrap required, opened a new log
I20260130 02:57:53.266738 19625 ts_tablet_manager.cc:1403] T 96584ca174b140f7ad1a2ca698baca71 P d870164263bb44b6b780a557daea3656: Time spent bootstrapping tablet: real 0.085s user 0.007s sys 0.008s
I20260130 02:57:53.271988 19463 leader_election.cc:304] T 3796b2db28ff4bb78c3f225f8ab9baae P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 371a36d219164974abdf848841528152, d870164263bb44b6b780a557daea3656; no voters:
I20260130 02:57:53.272255 19625 raft_consensus.cc:359] T 96584ca174b140f7ad1a2ca698baca71 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:57:53.272905 19625 raft_consensus.cc:385] T 96584ca174b140f7ad1a2ca698baca71 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:57:53.272902 19630 raft_consensus.cc:2804] T 3796b2db28ff4bb78c3f225f8ab9baae P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260130 02:57:53.273182 19625 raft_consensus.cc:740] T 96584ca174b140f7ad1a2ca698baca71 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d870164263bb44b6b780a557daea3656, State: Initialized, Role: FOLLOWER
I20260130 02:57:53.273370 19630 raft_consensus.cc:493] T 3796b2db28ff4bb78c3f225f8ab9baae P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260130 02:57:53.273689 19630 raft_consensus.cc:3060] T 3796b2db28ff4bb78c3f225f8ab9baae P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:57:53.273844 19625 consensus_queue.cc:260] T 96584ca174b140f7ad1a2ca698baca71 P d870164263bb44b6b780a557daea3656 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:57:53.274456 19204 tablet_service.cc:1505] Processing CreateTablet for tablet dd64001040074682875a236453d4974f (DEFAULT_TABLE table=default.Table_192f0cbcf82c4a0db6d5e5adc402e01a [id=e86c9f7d95b34a96baeb24e965c2f582]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:57:53.275557 19204 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dd64001040074682875a236453d4974f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:57:53.280536 19360 tablet_service.cc:1505] Processing CreateTablet for tablet dd64001040074682875a236453d4974f (DEFAULT_TABLE table=default.Table_192f0cbcf82c4a0db6d5e5adc402e01a [id=e86c9f7d95b34a96baeb24e965c2f582]), partition=RANGE (key) PARTITION UNBOUNDED
W20260130 02:57:53.281566 19580 tablet.cc:2378] T 3796b2db28ff4bb78c3f225f8ab9baae P d870164263bb44b6b780a557daea3656: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260130 02:57:53.281776 19360 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dd64001040074682875a236453d4974f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:57:53.283649 19624 tablet_bootstrap.cc:492] T 96584ca174b140f7ad1a2ca698baca71 P 371a36d219164974abdf848841528152: No bootstrap required, opened a new log
I20260130 02:57:53.284228 19624 ts_tablet_manager.cc:1403] T 96584ca174b140f7ad1a2ca698baca71 P 371a36d219164974abdf848841528152: Time spent bootstrapping tablet: real 0.029s user 0.006s sys 0.008s
I20260130 02:57:53.283994 19507 tablet_service.cc:1505] Processing CreateTablet for tablet dd64001040074682875a236453d4974f (DEFAULT_TABLE table=default.Table_192f0cbcf82c4a0db6d5e5adc402e01a [id=e86c9f7d95b34a96baeb24e965c2f582]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:57:53.285072 19507 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dd64001040074682875a236453d4974f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:57:53.293398 19630 raft_consensus.cc:515] T 3796b2db28ff4bb78c3f225f8ab9baae P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:57:53.294718 19624 raft_consensus.cc:359] T 96584ca174b140f7ad1a2ca698baca71 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:57:53.295495 19624 raft_consensus.cc:385] T 96584ca174b140f7ad1a2ca698baca71 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:57:53.295814 19624 raft_consensus.cc:740] T 96584ca174b140f7ad1a2ca698baca71 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 371a36d219164974abdf848841528152, State: Initialized, Role: FOLLOWER
I20260130 02:57:53.296612 19624 consensus_queue.cc:260] T 96584ca174b140f7ad1a2ca698baca71 P 371a36d219164974abdf848841528152 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:57:53.299255 19624 ts_tablet_manager.cc:1434] T 96584ca174b140f7ad1a2ca698baca71 P 371a36d219164974abdf848841528152: Time spent starting tablet: real 0.015s user 0.005s sys 0.001s
I20260130 02:57:53.299878 19625 ts_tablet_manager.cc:1434] T 96584ca174b140f7ad1a2ca698baca71 P d870164263bb44b6b780a557daea3656: Time spent starting tablet: real 0.033s user 0.007s sys 0.000s
I20260130 02:57:53.300206 19624 tablet_bootstrap.cc:492] T 00e6ed81b43d4e74aa6d648fdc1f6c65 P 371a36d219164974abdf848841528152: Bootstrap starting.
I20260130 02:57:53.303052 19635 raft_consensus.cc:493] T 96584ca174b140f7ad1a2ca698baca71 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:57:53.307155 19225 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3796b2db28ff4bb78c3f225f8ab9baae" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152"
I20260130 02:57:53.307701 19225 raft_consensus.cc:3060] T 3796b2db28ff4bb78c3f225f8ab9baae P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:57:53.314952 19225 raft_consensus.cc:2468] T 3796b2db28ff4bb78c3f225f8ab9baae P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d870164263bb44b6b780a557daea3656 in term 1.
W20260130 02:57:53.316927 18870 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet dd64001040074682875a236453d4974f consensus state has no leader
I20260130 02:57:53.305133 19630 leader_election.cc:290] T 3796b2db28ff4bb78c3f225f8ab9baae P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 election: Requested vote from peers 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767), 371a36d219164974abdf848841528152 (127.18.100.65:42021)
I20260130 02:57:53.316200 19463 leader_election.cc:304] T 3796b2db28ff4bb78c3f225f8ab9baae P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 371a36d219164974abdf848841528152, d870164263bb44b6b780a557daea3656; no voters:
I20260130 02:57:53.318483 19630 raft_consensus.cc:2804] T 3796b2db28ff4bb78c3f225f8ab9baae P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Leader election won for term 1
I20260130 02:57:53.318969 19630 raft_consensus.cc:697] T 3796b2db28ff4bb78c3f225f8ab9baae P d870164263bb44b6b780a557daea3656 [term 1 LEADER]: Becoming Leader. State: Replica: d870164263bb44b6b780a557daea3656, State: Running, Role: LEADER
W20260130 02:57:53.320142 19425 heartbeater.cc:646] Failed to heartbeat to 127.18.100.124:34629 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.18.100.124:34629: connect: Connection refused (error 111)
I20260130 02:57:53.322751 19630 consensus_queue.cc:237] T 3796b2db28ff4bb78c3f225f8ab9baae P d870164263bb44b6b780a557daea3656 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:57:53.303571 19635 raft_consensus.cc:515] T 96584ca174b140f7ad1a2ca698baca71 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:57:53.330963 19626 tablet_bootstrap.cc:654] T 96584ca174b140f7ad1a2ca698baca71 P 78c2c46c7520492db35d51c244fb4f74: Neither blocks nor log segments found. Creating new log.
I20260130 02:57:53.333274 19635 leader_election.cc:290] T 96584ca174b140f7ad1a2ca698baca71 P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767), d870164263bb44b6b780a557daea3656 (127.18.100.67:39571)
I20260130 02:57:53.333475 18875 catalog_manager.cc:5936] Not sending DeleteTablet requests; no consensus state for tablet 913f3e727b44405c838bef698f4ceac2
I20260130 02:57:53.336360 19624 tablet_bootstrap.cc:654] T 00e6ed81b43d4e74aa6d648fdc1f6c65 P 371a36d219164974abdf848841528152: Neither blocks nor log segments found. Creating new log.
I20260130 02:57:53.325969 19625 tablet_bootstrap.cc:492] T 00e6ed81b43d4e74aa6d648fdc1f6c65 P d870164263bb44b6b780a557daea3656: Bootstrap starting.
I20260130 02:57:53.346041 19380 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3796b2db28ff4bb78c3f225f8ab9baae" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74" is_pre_election: true
I20260130 02:57:53.346920 19380 raft_consensus.cc:2468] T 3796b2db28ff4bb78c3f225f8ab9baae P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d870164263bb44b6b780a557daea3656 in term 0.
I20260130 02:57:53.348410 19380 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3796b2db28ff4bb78c3f225f8ab9baae" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74"
I20260130 02:57:53.348883 19380 raft_consensus.cc:3060] T 3796b2db28ff4bb78c3f225f8ab9baae P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:57:53.355909 19380 raft_consensus.cc:2468] T 3796b2db28ff4bb78c3f225f8ab9baae P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d870164263bb44b6b780a557daea3656 in term 1.
I20260130 02:57:53.363629 19626 tablet_bootstrap.cc:492] T 96584ca174b140f7ad1a2ca698baca71 P 78c2c46c7520492db35d51c244fb4f74: No bootstrap required, opened a new log
I20260130 02:57:53.364161 19626 ts_tablet_manager.cc:1403] T 96584ca174b140f7ad1a2ca698baca71 P 78c2c46c7520492db35d51c244fb4f74: Time spent bootstrapping tablet: real 0.132s user 0.001s sys 0.012s
I20260130 02:57:53.367136 19626 raft_consensus.cc:359] T 96584ca174b140f7ad1a2ca698baca71 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:57:53.367931 19626 raft_consensus.cc:385] T 96584ca174b140f7ad1a2ca698baca71 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:57:53.367980 19205 tablet_service.cc:1505] Processing CreateTablet for tablet df2c8989d6c1480c97baa23db01141eb (DEFAULT_TABLE table=default.Table_5b8236964cd94924b945ec3d68a500fc [id=747af828147149efb7b7b8d33c49286d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:57:53.369194 19205 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet df2c8989d6c1480c97baa23db01141eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:57:53.369469 19507 tablet_service.cc:1505] Processing CreateTablet for tablet df2c8989d6c1480c97baa23db01141eb (DEFAULT_TABLE table=default.Table_5b8236964cd94924b945ec3d68a500fc [id=747af828147149efb7b7b8d33c49286d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:57:53.370442 19507 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet df2c8989d6c1480c97baa23db01141eb. 1 dirs total, 0 dirs full, 0 dirs failed
W20260130 02:57:53.370784 19280 tablet.cc:2378] T 3796b2db28ff4bb78c3f225f8ab9baae P 371a36d219164974abdf848841528152: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260130 02:57:53.370973 19360 tablet_service.cc:1505] Processing CreateTablet for tablet df2c8989d6c1480c97baa23db01141eb (DEFAULT_TABLE table=default.Table_5b8236964cd94924b945ec3d68a500fc [id=747af828147149efb7b7b8d33c49286d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:57:53.371359 18875 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50284:
name: "default.Table_d2a79a5f15f344d28d66324210f97757"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260130 02:57:53.372975 18875 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d2a79a5f15f344d28d66324210f97757 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260130 02:57:53.376363 19625 tablet_bootstrap.cc:654] T 00e6ed81b43d4e74aa6d648fdc1f6c65 P d870164263bb44b6b780a557daea3656: Neither blocks nor log segments found. Creating new log.
I20260130 02:57:53.378095 19360 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet df2c8989d6c1480c97baa23db01141eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:57:53.380906 19624 tablet_bootstrap.cc:492] T 00e6ed81b43d4e74aa6d648fdc1f6c65 P 371a36d219164974abdf848841528152: No bootstrap required, opened a new log
I20260130 02:57:53.381347 19624 ts_tablet_manager.cc:1403] T 00e6ed81b43d4e74aa6d648fdc1f6c65 P 371a36d219164974abdf848841528152: Time spent bootstrapping tablet: real 0.081s user 0.006s sys 0.004s
I20260130 02:57:53.379648 19626 raft_consensus.cc:740] T 96584ca174b140f7ad1a2ca698baca71 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78c2c46c7520492db35d51c244fb4f74, State: Initialized, Role: FOLLOWER
I20260130 02:57:53.383872 19624 raft_consensus.cc:359] T 00e6ed81b43d4e74aa6d648fdc1f6c65 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:57:53.384621 19624 raft_consensus.cc:385] T 00e6ed81b43d4e74aa6d648fdc1f6c65 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:57:53.384608 19626 consensus_queue.cc:260] T 96584ca174b140f7ad1a2ca698baca71 P 78c2c46c7520492db35d51c244fb4f74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:57:53.384933 19624 raft_consensus.cc:740] T 00e6ed81b43d4e74aa6d648fdc1f6c65 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 371a36d219164974abdf848841528152, State: Initialized, Role: FOLLOWER
I20260130 02:57:53.385637 19624 consensus_queue.cc:260] T 00e6ed81b43d4e74aa6d648fdc1f6c65 P 371a36d219164974abdf848841528152 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:57:53.388185 19624 ts_tablet_manager.cc:1434] T 00e6ed81b43d4e74aa6d648fdc1f6c65 P 371a36d219164974abdf848841528152: Time spent starting tablet: real 0.007s user 0.006s sys 0.001s
I20260130 02:57:53.388733 19626 ts_tablet_manager.cc:1434] T 96584ca174b140f7ad1a2ca698baca71 P 78c2c46c7520492db35d51c244fb4f74: Time spent starting tablet: real 0.024s user 0.006s sys 0.001s
I20260130 02:57:53.389153 19624 tablet_bootstrap.cc:492] T dd64001040074682875a236453d4974f P 371a36d219164974abdf848841528152: Bootstrap starting.
I20260130 02:57:53.389593 19626 tablet_bootstrap.cc:492] T 00e6ed81b43d4e74aa6d648fdc1f6c65 P 78c2c46c7520492db35d51c244fb4f74: Bootstrap starting.
I20260130 02:57:53.396490 19624 tablet_bootstrap.cc:654] T dd64001040074682875a236453d4974f P 371a36d219164974abdf848841528152: Neither blocks nor log segments found. Creating new log.
I20260130 02:57:53.406602 18869 catalog_manager.cc:5941] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4: Sending DeleteTablet for 3 replicas of tablet 3796b2db28ff4bb78c3f225f8ab9baae
I20260130 02:57:53.408538 19358 tablet_service.cc:1552] Processing DeleteTablet for tablet 3796b2db28ff4bb78c3f225f8ab9baae with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:57:53 UTC) from {username='slave'} at 127.0.0.1:60110
I20260130 02:57:53.412878 19204 tablet_service.cc:1552] Processing DeleteTablet for tablet 3796b2db28ff4bb78c3f225f8ab9baae with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:57:53 UTC) from {username='slave'} at 127.0.0.1:42870
I20260130 02:57:53.416949 18874 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50284:
name: "default.Table_0f26dbdebb8c49f6997d31582bbab3d1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260130 02:57:53.418056 19626 tablet_bootstrap.cc:654] T 00e6ed81b43d4e74aa6d648fdc1f6c65 P 78c2c46c7520492db35d51c244fb4f74: Neither blocks nor log segments found. Creating new log.
I20260130 02:57:53.420037 19624 tablet_bootstrap.cc:492] T dd64001040074682875a236453d4974f P 371a36d219164974abdf848841528152: No bootstrap required, opened a new log
I20260130 02:57:53.420531 19624 ts_tablet_manager.cc:1403] T dd64001040074682875a236453d4974f P 371a36d219164974abdf848841528152: Time spent bootstrapping tablet: real 0.032s user 0.016s sys 0.001s
I20260130 02:57:53.418910 19507 tablet_service.cc:1552] Processing DeleteTablet for tablet 3796b2db28ff4bb78c3f225f8ab9baae with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:57:53 UTC) from {username='slave'} at 127.0.0.1:59068
W20260130 02:57:53.423167 18874 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0f26dbdebb8c49f6997d31582bbab3d1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260130 02:57:53.423496 19624 raft_consensus.cc:359] T dd64001040074682875a236453d4974f P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:57:53.424228 19624 raft_consensus.cc:385] T dd64001040074682875a236453d4974f P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:57:53.424576 19624 raft_consensus.cc:740] T dd64001040074682875a236453d4974f P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 371a36d219164974abdf848841528152, State: Initialized, Role: FOLLOWER
I20260130 02:57:53.425366 19624 consensus_queue.cc:260] T dd64001040074682875a236453d4974f P 371a36d219164974abdf848841528152 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:57:53.417944 18872 catalog_manager.cc:5746] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571) does not have the latest schema for tablet 96584ca174b140f7ad1a2ca698baca71 (table default.Table_16126d9be1d1434783c533e31b977783 [id=1515db389f284e2f9ed1a3cd0061d0f1]). Expected version 2 got 0
I20260130 02:57:53.427846 19624 ts_tablet_manager.cc:1434] T dd64001040074682875a236453d4974f P 371a36d219164974abdf848841528152: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260130 02:57:53.429776 18869 catalog_manager.cc:5746] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021) does not have the latest schema for tablet 96584ca174b140f7ad1a2ca698baca71 (table default.Table_16126d9be1d1434783c533e31b977783 [id=1515db389f284e2f9ed1a3cd0061d0f1]). Expected version 2 got 0
I20260130 02:57:53.435523 18868 catalog_manager.cc:5746] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767) does not have the latest schema for tablet 96584ca174b140f7ad1a2ca698baca71 (table default.Table_16126d9be1d1434783c533e31b977783 [id=1515db389f284e2f9ed1a3cd0061d0f1]). Expected version 2 got 0
I20260130 02:57:53.442502 19624 tablet_bootstrap.cc:492] T df2c8989d6c1480c97baa23db01141eb P 371a36d219164974abdf848841528152: Bootstrap starting.
I20260130 02:57:53.443739 18868 catalog_manager.cc:5746] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021) does not have the latest schema for tablet dd64001040074682875a236453d4974f (table default.Table_76c537f09c384533b1c632de353f4dbb [id=e86c9f7d95b34a96baeb24e965c2f582]). Expected version 1 got 0
I20260130 02:57:53.446233 19380 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "96584ca174b140f7ad1a2ca698baca71" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74" is_pre_election: true
I20260130 02:57:53.446920 19380 raft_consensus.cc:2468] T 96584ca174b140f7ad1a2ca698baca71 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 371a36d219164974abdf848841528152 in term 0.
I20260130 02:57:53.448500 19624 tablet_bootstrap.cc:654] T df2c8989d6c1480c97baa23db01141eb P 371a36d219164974abdf848841528152: Neither blocks nor log segments found. Creating new log.
I20260130 02:57:53.448496 19625 tablet_bootstrap.cc:492] T 00e6ed81b43d4e74aa6d648fdc1f6c65 P d870164263bb44b6b780a557daea3656: No bootstrap required, opened a new log
I20260130 02:57:53.448911 19625 ts_tablet_manager.cc:1403] T 00e6ed81b43d4e74aa6d648fdc1f6c65 P d870164263bb44b6b780a557daea3656: Time spent bootstrapping tablet: real 0.123s user 0.010s sys 0.000s
I20260130 02:57:53.449561 19645 tablet_replica.cc:333] T 3796b2db28ff4bb78c3f225f8ab9baae P 78c2c46c7520492db35d51c244fb4f74: stopping tablet replica
I20260130 02:57:53.450377 19645 raft_consensus.cc:2243] T 3796b2db28ff4bb78c3f225f8ab9baae P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260130 02:57:53.451012 19645 raft_consensus.cc:2272] T 3796b2db28ff4bb78c3f225f8ab9baae P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260130 02:57:53.460026 19159 leader_election.cc:304] T 96584ca174b140f7ad1a2ca698baca71 P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 371a36d219164974abdf848841528152, 78c2c46c7520492db35d51c244fb4f74; no voters:
I20260130 02:57:53.452661 19646 tablet_replica.cc:333] T 3796b2db28ff4bb78c3f225f8ab9baae P 371a36d219164974abdf848841528152: stopping tablet replica
I20260130 02:57:53.461531 19646 raft_consensus.cc:2243] T 3796b2db28ff4bb78c3f225f8ab9baae P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260130 02:57:53.461999 19635 raft_consensus.cc:2804] T 96584ca174b140f7ad1a2ca698baca71 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260130 02:57:53.462467 19635 raft_consensus.cc:493] T 96584ca174b140f7ad1a2ca698baca71 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260130 02:57:53.462826 19635 raft_consensus.cc:3060] T 96584ca174b140f7ad1a2ca698baca71 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:57:53.465736 19625 raft_consensus.cc:359] T 00e6ed81b43d4e74aa6d648fdc1f6c65 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:57:53.458678 19528 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "96584ca174b140f7ad1a2ca698baca71" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
I20260130 02:57:53.469000 19528 raft_consensus.cc:2468] T 96584ca174b140f7ad1a2ca698baca71 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 371a36d219164974abdf848841528152 in term 0.
I20260130 02:57:53.469599 19635 raft_consensus.cc:515] T 96584ca174b140f7ad1a2ca698baca71 P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:57:53.457123 19647 tablet_replica.cc:333] T 3796b2db28ff4bb78c3f225f8ab9baae P d870164263bb44b6b780a557daea3656: stopping tablet replica
I20260130 02:57:53.470849 19647 raft_consensus.cc:2243] T 3796b2db28ff4bb78c3f225f8ab9baae P d870164263bb44b6b780a557daea3656 [term 1 LEADER]: Raft consensus shutting down.
I20260130 02:57:53.471539 19635 leader_election.cc:290] T 96584ca174b140f7ad1a2ca698baca71 P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 election: Requested vote from peers 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767), d870164263bb44b6b780a557daea3656 (127.18.100.67:39571)
I20260130 02:57:53.471693 19647 pending_rounds.cc:70] T 3796b2db28ff4bb78c3f225f8ab9baae P d870164263bb44b6b780a557daea3656: Trying to abort 1 pending ops.
I20260130 02:57:53.471941 19647 pending_rounds.cc:77] T 3796b2db28ff4bb78c3f225f8ab9baae P d870164263bb44b6b780a557daea3656: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7248862713178947584 op_type: NO_OP noop_request { }
I20260130 02:57:53.472381 19647 raft_consensus.cc:2889] T 3796b2db28ff4bb78c3f225f8ab9baae P d870164263bb44b6b780a557daea3656 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20260130 02:57:53.468425 19626 tablet_bootstrap.cc:492] T 00e6ed81b43d4e74aa6d648fdc1f6c65 P 78c2c46c7520492db35d51c244fb4f74: No bootstrap required, opened a new log
I20260130 02:57:53.472910 19647 raft_consensus.cc:2272] T 3796b2db28ff4bb78c3f225f8ab9baae P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260130 02:57:53.473055 19646 raft_consensus.cc:2272] T 3796b2db28ff4bb78c3f225f8ab9baae P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260130 02:57:53.473687 19626 ts_tablet_manager.cc:1403] T 00e6ed81b43d4e74aa6d648fdc1f6c65 P 78c2c46c7520492db35d51c244fb4f74: Time spent bootstrapping tablet: real 0.084s user 0.012s sys 0.000s
I20260130 02:57:53.474009 19611 sys_catalog.cc:455] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "658299197a424f53a37f14fadf089ae4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "658299197a424f53a37f14fadf089ae4" member_type: VOTER last_known_addr { host: "127.18.100.126" port: 38191 } } peers { permanent_uuid: "107f274cdda543038832f06a56ab9130" member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39711 } } peers { permanent_uuid: "3749baa287ed4f249329a74071781edd" member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34629 } } }
I20260130 02:57:53.474786 19611 sys_catalog.cc:458] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4 [sys.catalog]: This master's current role is: LEADER
I20260130 02:57:53.476150 19626 raft_consensus.cc:359] T 00e6ed81b43d4e74aa6d648fdc1f6c65 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:57:53.476624 19647 ts_tablet_manager.cc:1916] T 3796b2db28ff4bb78c3f225f8ab9baae P d870164263bb44b6b780a557daea3656: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260130 02:57:53.476819 19626 raft_consensus.cc:385] T 00e6ed81b43d4e74aa6d648fdc1f6c65 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:57:53.477065 19626 raft_consensus.cc:740] T 00e6ed81b43d4e74aa6d648fdc1f6c65 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78c2c46c7520492db35d51c244fb4f74, State: Initialized, Role: FOLLOWER
I20260130 02:57:53.477671 19626 consensus_queue.cc:260] T 00e6ed81b43d4e74aa6d648fdc1f6c65 P 78c2c46c7520492db35d51c244fb4f74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:57:53.479334 18872 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50284:
table { table_name: "default.Table_16126d9be1d1434783c533e31b977783" } new_table_name: "default.Table_3732a807f13b4ccea8c73b3a61c8c2e5" modify_external_catalogs: true
I20260130 02:57:53.479701 19626 ts_tablet_manager.cc:1434] T 00e6ed81b43d4e74aa6d648fdc1f6c65 P 78c2c46c7520492db35d51c244fb4f74: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260130 02:57:53.480981 19528 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "96584ca174b140f7ad1a2ca698baca71" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656"
I20260130 02:57:53.481500 19528 raft_consensus.cc:3060] T 96584ca174b140f7ad1a2ca698baca71 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:57:53.479626 19380 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "96584ca174b140f7ad1a2ca698baca71" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74"
I20260130 02:57:53.482277 19380 raft_consensus.cc:3060] T 96584ca174b140f7ad1a2ca698baca71 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:57:53.478377 18868 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50284:
name: "default.Table_b8f4e9cf27f44a479e1adad0b7ce4e33"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260130 02:57:53.484064 18868 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b8f4e9cf27f44a479e1adad0b7ce4e33 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260130 02:57:53.488305 19528 raft_consensus.cc:2468] T 96584ca174b140f7ad1a2ca698baca71 P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 371a36d219164974abdf848841528152 in term 1.
I20260130 02:57:53.490031 19159 leader_election.cc:304] T 96584ca174b140f7ad1a2ca698baca71 P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 371a36d219164974abdf848841528152, d870164263bb44b6b780a557daea3656; no voters:
I20260130 02:57:53.492631 19380 raft_consensus.cc:2468] T 96584ca174b140f7ad1a2ca698baca71 P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 371a36d219164974abdf848841528152 in term 1.
I20260130 02:57:53.481878 19645 ts_tablet_manager.cc:1916] T 3796b2db28ff4bb78c3f225f8ab9baae P 78c2c46c7520492db35d51c244fb4f74: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260130 02:57:53.445737 18869 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50284:
name: "default.Table_8e5ff21c1d224aab962d02463ea05a49"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260130 02:57:53.499992 18869 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8e5ff21c1d224aab962d02463ea05a49 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260130 02:57:53.503399 19626 tablet_bootstrap.cc:492] T dd64001040074682875a236453d4974f P 78c2c46c7520492db35d51c244fb4f74: Bootstrap starting.
I20260130 02:57:53.506381 19625 raft_consensus.cc:385] T 00e6ed81b43d4e74aa6d648fdc1f6c65 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:57:53.506755 19625 raft_consensus.cc:740] T 00e6ed81b43d4e74aa6d648fdc1f6c65 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d870164263bb44b6b780a557daea3656, State: Initialized, Role: FOLLOWER
I20260130 02:57:53.507505 19625 consensus_queue.cc:260] T 00e6ed81b43d4e74aa6d648fdc1f6c65 P d870164263bb44b6b780a557daea3656 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:57:53.508678 19635 raft_consensus.cc:2804] T 96584ca174b140f7ad1a2ca698baca71 P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Leader election won for term 1
W20260130 02:57:53.520994 18872 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 96584ca174b140f7ad1a2ca698baca71 consensus state has no leader
I20260130 02:57:53.526103 18868 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50284:
table { table_name: "default.Table_d2a79a5f15f344d28d66324210f97757" } new_table_name: "default.Table_5dc02861dbc440528863f809c8300429" modify_external_catalogs: true
I20260130 02:57:53.537246 19646 ts_tablet_manager.cc:1916] T 3796b2db28ff4bb78c3f225f8ab9baae P 371a36d219164974abdf848841528152: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260130 02:57:53.537312 19625 ts_tablet_manager.cc:1434] T 00e6ed81b43d4e74aa6d648fdc1f6c65 P d870164263bb44b6b780a557daea3656: Time spent starting tablet: real 0.074s user 0.005s sys 0.000s
I20260130 02:57:53.538233 19625 tablet_bootstrap.cc:492] T dd64001040074682875a236453d4974f P d870164263bb44b6b780a557daea3656: Bootstrap starting.
I20260130 02:57:53.540809 18874 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50284:
table { table_name: "default.Table_b2b8ae6f8ee74fc2be70b4cd5d16782e" } modify_external_catalogs: true
I20260130 02:57:53.541437 18874 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50284:
table { table_name: "default.Table_b2b8ae6f8ee74fc2be70b4cd5d16782e" } modify_external_catalogs: true
I20260130 02:57:53.547588 19624 tablet_bootstrap.cc:492] T df2c8989d6c1480c97baa23db01141eb P 371a36d219164974abdf848841528152: No bootstrap required, opened a new log
I20260130 02:57:53.548074 19624 ts_tablet_manager.cc:1403] T df2c8989d6c1480c97baa23db01141eb P 371a36d219164974abdf848841528152: Time spent bootstrapping tablet: real 0.106s user 0.037s sys 0.027s
I20260130 02:57:53.551379 19624 raft_consensus.cc:359] T df2c8989d6c1480c97baa23db01141eb P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:57:53.551878 19635 raft_consensus.cc:697] T 96584ca174b140f7ad1a2ca698baca71 P 371a36d219164974abdf848841528152 [term 1 LEADER]: Becoming Leader. State: Replica: 371a36d219164974abdf848841528152, State: Running, Role: LEADER
I20260130 02:57:53.552135 19624 raft_consensus.cc:385] T df2c8989d6c1480c97baa23db01141eb P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:57:53.552508 19624 raft_consensus.cc:740] T df2c8989d6c1480c97baa23db01141eb P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 371a36d219164974abdf848841528152, State: Initialized, Role: FOLLOWER
I20260130 02:57:53.552825 19635 consensus_queue.cc:237] T 96584ca174b140f7ad1a2ca698baca71 P 371a36d219164974abdf848841528152 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:57:53.553165 19624 consensus_queue.cc:260] T df2c8989d6c1480c97baa23db01141eb P 371a36d219164974abdf848841528152 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:57:53.555433 19624 ts_tablet_manager.cc:1434] T df2c8989d6c1480c97baa23db01141eb P 371a36d219164974abdf848841528152: Time spent starting tablet: real 0.007s user 0.005s sys 0.001s
W20260130 02:57:53.561883 18868 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 06b31d020b864c418476e6dcc1e4eafb has no consensus state
I20260130 02:57:53.567489 19626 tablet_bootstrap.cc:654] T dd64001040074682875a236453d4974f P 78c2c46c7520492db35d51c244fb4f74: Neither blocks nor log segments found. Creating new log.
I20260130 02:57:53.569619 19204 tablet_service.cc:1505] Processing CreateTablet for tablet 06b31d020b864c418476e6dcc1e4eafb (DEFAULT_TABLE table=default.Table_5dc02861dbc440528863f809c8300429 [id=b78b791f363d4c0f906e2a4b21dfacc4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:57:53.570837 19204 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 06b31d020b864c418476e6dcc1e4eafb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:57:53.572234 19360 tablet_service.cc:1505] Processing CreateTablet for tablet 06b31d020b864c418476e6dcc1e4eafb (DEFAULT_TABLE table=default.Table_5dc02861dbc440528863f809c8300429 [id=b78b791f363d4c0f906e2a4b21dfacc4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:57:53.572474 19507 tablet_service.cc:1505] Processing CreateTablet for tablet 06b31d020b864c418476e6dcc1e4eafb (DEFAULT_TABLE table=default.Table_5dc02861dbc440528863f809c8300429 [id=b78b791f363d4c0f906e2a4b21dfacc4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:57:53.573262 19360 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 06b31d020b864c418476e6dcc1e4eafb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:57:53.573501 19507 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 06b31d020b864c418476e6dcc1e4eafb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:57:53.575353 19645 ts_tablet_manager.cc:1929] T 3796b2db28ff4bb78c3f225f8ab9baae P 78c2c46c7520492db35d51c244fb4f74: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260130 02:57:53.575764 19645 log.cc:1199] T 3796b2db28ff4bb78c3f225f8ab9baae P 78c2c46c7520492db35d51c244fb4f74: Deleting WAL directory at /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-1/wal/wals/3796b2db28ff4bb78c3f225f8ab9baae
I20260130 02:57:53.576602 19645 ts_tablet_manager.cc:1950] T 3796b2db28ff4bb78c3f225f8ab9baae P 78c2c46c7520492db35d51c244fb4f74: Deleting consensus metadata
I20260130 02:57:53.580616 19625 tablet_bootstrap.cc:654] T dd64001040074682875a236453d4974f P d870164263bb44b6b780a557daea3656: Neither blocks nor log segments found. Creating new log.
I20260130 02:57:53.589433 19624 tablet_bootstrap.cc:492] T 06b31d020b864c418476e6dcc1e4eafb P 371a36d219164974abdf848841528152: Bootstrap starting.
I20260130 02:57:53.592690 19631 raft_consensus.cc:493] T 00e6ed81b43d4e74aa6d648fdc1f6c65 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:57:53.593551 19625 tablet_bootstrap.cc:492] T dd64001040074682875a236453d4974f P d870164263bb44b6b780a557daea3656: No bootstrap required, opened a new log
I20260130 02:57:53.593164 19631 raft_consensus.cc:515] T 00e6ed81b43d4e74aa6d648fdc1f6c65 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:57:53.593988 19625 ts_tablet_manager.cc:1403] T dd64001040074682875a236453d4974f P d870164263bb44b6b780a557daea3656: Time spent bootstrapping tablet: real 0.056s user 0.010s sys 0.000s
I20260130 02:57:53.596930 19624 tablet_bootstrap.cc:654] T 06b31d020b864c418476e6dcc1e4eafb P 371a36d219164974abdf848841528152: Neither blocks nor log segments found. Creating new log.
I20260130 02:57:53.599408 19625 raft_consensus.cc:359] T dd64001040074682875a236453d4974f P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:57:53.600070 19625 raft_consensus.cc:385] T dd64001040074682875a236453d4974f P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:57:53.600356 19625 raft_consensus.cc:740] T dd64001040074682875a236453d4974f P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d870164263bb44b6b780a557daea3656, State: Initialized, Role: FOLLOWER
I20260130 02:57:53.600963 19625 consensus_queue.cc:260] T dd64001040074682875a236453d4974f P d870164263bb44b6b780a557daea3656 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:57:53.601873 19631 leader_election.cc:290] T 00e6ed81b43d4e74aa6d648fdc1f6c65 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), d870164263bb44b6b780a557daea3656 (127.18.100.67:39571)
I20260130 02:57:53.606704 19625 ts_tablet_manager.cc:1434] T dd64001040074682875a236453d4974f P d870164263bb44b6b780a557daea3656: Time spent starting tablet: real 0.012s user 0.003s sys 0.005s
I20260130 02:57:53.607661 18869 catalog_manager.cc:5654] T 96584ca174b140f7ad1a2ca698baca71 P 371a36d219164974abdf848841528152 reported cstate change: term changed from 0 to 1, leader changed from <none> to 371a36d219164974abdf848841528152 (127.18.100.65). New cstate: current_term: 1 leader_uuid: "371a36d219164974abdf848841528152" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } health_report { overall_health: UNKNOWN } } }
I20260130 02:57:53.609329 18869 catalog_manager.cc:5746] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021) does not have the latest schema for tablet 96584ca174b140f7ad1a2ca698baca71 (table default.Table_3732a807f13b4ccea8c73b3a61c8c2e5 [id=1515db389f284e2f9ed1a3cd0061d0f1]). Expected version 3 got 0
I20260130 02:57:53.611057 19625 tablet_bootstrap.cc:492] T df2c8989d6c1480c97baa23db01141eb P d870164263bb44b6b780a557daea3656: Bootstrap starting.
I20260130 02:57:53.612923 19624 tablet_bootstrap.cc:492] T 06b31d020b864c418476e6dcc1e4eafb P 371a36d219164974abdf848841528152: No bootstrap required, opened a new log
I20260130 02:57:53.613442 19624 ts_tablet_manager.cc:1403] T 06b31d020b864c418476e6dcc1e4eafb P 371a36d219164974abdf848841528152: Time spent bootstrapping tablet: real 0.024s user 0.008s sys 0.009s
I20260130 02:57:53.613473 19225 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00e6ed81b43d4e74aa6d648fdc1f6c65" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152" is_pre_election: true
I20260130 02:57:53.614125 19225 raft_consensus.cc:2468] T 00e6ed81b43d4e74aa6d648fdc1f6c65 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78c2c46c7520492db35d51c244fb4f74 in term 0.
I20260130 02:57:53.615676 19314 leader_election.cc:304] T 00e6ed81b43d4e74aa6d648fdc1f6c65 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 371a36d219164974abdf848841528152, 78c2c46c7520492db35d51c244fb4f74; no voters:
I20260130 02:57:53.616641 19631 raft_consensus.cc:2804] T 00e6ed81b43d4e74aa6d648fdc1f6c65 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260130 02:57:53.616906 19625 tablet_bootstrap.cc:654] T df2c8989d6c1480c97baa23db01141eb P d870164263bb44b6b780a557daea3656: Neither blocks nor log segments found. Creating new log.
I20260130 02:57:53.616974 19631 raft_consensus.cc:493] T 00e6ed81b43d4e74aa6d648fdc1f6c65 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260130 02:57:53.617286 19631 raft_consensus.cc:3060] T 00e6ed81b43d4e74aa6d648fdc1f6c65 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:57:53.616870 19624 raft_consensus.cc:359] T 06b31d020b864c418476e6dcc1e4eafb P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } }
I20260130 02:57:53.618036 19624 raft_consensus.cc:385] T 06b31d020b864c418476e6dcc1e4eafb P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:57:53.618382 19624 raft_consensus.cc:740] T 06b31d020b864c418476e6dcc1e4eafb P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 371a36d219164974abdf848841528152, State: Initialized, Role: FOLLOWER
I20260130 02:57:53.583801 19647 ts_tablet_manager.cc:1929] T 3796b2db28ff4bb78c3f225f8ab9baae P d870164263bb44b6b780a557daea3656: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260130 02:57:53.619140 19624 consensus_queue.cc:260] T 06b31d020b864c418476e6dcc1e4eafb P 371a36d219164974abdf848841528152 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } }
I20260130 02:57:53.619676 19647 log.cc:1199] T 3796b2db28ff4bb78c3f225f8ab9baae P d870164263bb44b6b780a557daea3656: Deleting WAL directory at /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-2/wal/wals/3796b2db28ff4bb78c3f225f8ab9baae
I20260130 02:57:53.621536 19624 ts_tablet_manager.cc:1434] T 06b31d020b864c418476e6dcc1e4eafb P 371a36d219164974abdf848841528152: Time spent starting tablet: real 0.008s user 0.005s sys 0.000s
I20260130 02:57:53.624866 18860 catalog_manager.cc:4985] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767): tablet 3796b2db28ff4bb78c3f225f8ab9baae (table default.Table_eb61588fb4964de39da6127edd2bd501 [id=acd866a00ee34b529227c16091417594]) successfully deleted
I20260130 02:57:53.625097 18868 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50284:
table { table_name: "default.Table_76c537f09c384533b1c632de353f4dbb" } modify_external_catalogs: true
I20260130 02:57:53.625828 19647 ts_tablet_manager.cc:1950] T 3796b2db28ff4bb78c3f225f8ab9baae P d870164263bb44b6b780a557daea3656: Deleting consensus metadata
I20260130 02:57:53.625784 18868 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50284:
table { table_name: "default.Table_76c537f09c384533b1c632de353f4dbb" } modify_external_catalogs: true
I20260130 02:57:53.628980 18860 catalog_manager.cc:4985] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): tablet 3796b2db28ff4bb78c3f225f8ab9baae (table default.Table_eb61588fb4964de39da6127edd2bd501 [id=acd866a00ee34b529227c16091417594]) successfully deleted
I20260130 02:57:53.642820 19646 ts_tablet_manager.cc:1929] T 3796b2db28ff4bb78c3f225f8ab9baae P 371a36d219164974abdf848841528152: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260130 02:57:53.643343 19646 log.cc:1199] T 3796b2db28ff4bb78c3f225f8ab9baae P 371a36d219164974abdf848841528152: Deleting WAL directory at /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-0/wal/wals/3796b2db28ff4bb78c3f225f8ab9baae
I20260130 02:57:53.644316 19646 ts_tablet_manager.cc:1950] T 3796b2db28ff4bb78c3f225f8ab9baae P 371a36d219164974abdf848841528152: Deleting consensus metadata
I20260130 02:57:53.645701 19626 tablet_bootstrap.cc:492] T dd64001040074682875a236453d4974f P 78c2c46c7520492db35d51c244fb4f74: No bootstrap required, opened a new log
I20260130 02:57:53.646193 19626 ts_tablet_manager.cc:1403] T dd64001040074682875a236453d4974f P 78c2c46c7520492db35d51c244fb4f74: Time spent bootstrapping tablet: real 0.143s user 0.010s sys 0.000s
I20260130 02:57:53.647568 18861 catalog_manager.cc:4985] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021): tablet 3796b2db28ff4bb78c3f225f8ab9baae (table default.Table_eb61588fb4964de39da6127edd2bd501 [id=acd866a00ee34b529227c16091417594]) successfully deleted
I20260130 02:57:53.650421 19626 raft_consensus.cc:359] T dd64001040074682875a236453d4974f P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:57:53.651237 19626 raft_consensus.cc:385] T dd64001040074682875a236453d4974f P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:57:53.651527 19626 raft_consensus.cc:740] T dd64001040074682875a236453d4974f P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78c2c46c7520492db35d51c244fb4f74, State: Initialized, Role: FOLLOWER
I20260130 02:57:53.653766 19631 raft_consensus.cc:515] T 00e6ed81b43d4e74aa6d648fdc1f6c65 P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:57:53.654906 19626 consensus_queue.cc:260] T dd64001040074682875a236453d4974f P 78c2c46c7520492db35d51c244fb4f74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:57:53.655725 19631 leader_election.cc:290] T 00e6ed81b43d4e74aa6d648fdc1f6c65 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 election: Requested vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), d870164263bb44b6b780a557daea3656 (127.18.100.67:39571)
I20260130 02:57:53.656445 19225 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00e6ed81b43d4e74aa6d648fdc1f6c65" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152"
I20260130 02:57:53.656978 19225 raft_consensus.cc:3060] T 00e6ed81b43d4e74aa6d648fdc1f6c65 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:57:53.663671 18870 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50284:
name: "default.Table_38b6b66b6a5d41a297953249f24045ce"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260130 02:57:53.665526 18870 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_38b6b66b6a5d41a297953249f24045ce in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260130 02:57:53.668304 19625 tablet_bootstrap.cc:492] T df2c8989d6c1480c97baa23db01141eb P d870164263bb44b6b780a557daea3656: No bootstrap required, opened a new log
I20260130 02:57:53.668890 19625 ts_tablet_manager.cc:1403] T df2c8989d6c1480c97baa23db01141eb P d870164263bb44b6b780a557daea3656: Time spent bootstrapping tablet: real 0.059s user 0.010s sys 0.029s
I20260130 02:57:53.669166 19528 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00e6ed81b43d4e74aa6d648fdc1f6c65" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
I20260130 02:57:53.669785 19528 raft_consensus.cc:2468] T 00e6ed81b43d4e74aa6d648fdc1f6c65 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78c2c46c7520492db35d51c244fb4f74 in term 0.
I20260130 02:57:53.670796 19527 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00e6ed81b43d4e74aa6d648fdc1f6c65" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656"
I20260130 02:57:53.671399 19527 raft_consensus.cc:3060] T 00e6ed81b43d4e74aa6d648fdc1f6c65 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:57:53.674140 19626 ts_tablet_manager.cc:1434] T dd64001040074682875a236453d4974f P 78c2c46c7520492db35d51c244fb4f74: Time spent starting tablet: real 0.028s user 0.008s sys 0.003s
I20260130 02:57:53.675156 19626 tablet_bootstrap.cc:492] T df2c8989d6c1480c97baa23db01141eb P 78c2c46c7520492db35d51c244fb4f74: Bootstrap starting.
I20260130 02:57:53.679802 18868 catalog_manager.cc:5941] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4: Sending DeleteTablet for 3 replicas of tablet dd64001040074682875a236453d4974f
I20260130 02:57:53.681061 19626 tablet_bootstrap.cc:654] T df2c8989d6c1480c97baa23db01141eb P 78c2c46c7520492db35d51c244fb4f74: Neither blocks nor log segments found. Creating new log.
I20260130 02:57:53.682258 19360 tablet_service.cc:1552] Processing DeleteTablet for tablet dd64001040074682875a236453d4974f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:57:53 UTC) from {username='slave'} at 127.0.0.1:60110
I20260130 02:57:53.682485 19507 tablet_service.cc:1552] Processing DeleteTablet for tablet dd64001040074682875a236453d4974f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:57:53 UTC) from {username='slave'} at 127.0.0.1:59068
I20260130 02:57:53.682988 19645 tablet_replica.cc:333] T dd64001040074682875a236453d4974f P 78c2c46c7520492db35d51c244fb4f74: stopping tablet replica
I20260130 02:57:53.683248 19647 tablet_replica.cc:333] T dd64001040074682875a236453d4974f P d870164263bb44b6b780a557daea3656: stopping tablet replica
I20260130 02:57:53.683727 19645 raft_consensus.cc:2243] T dd64001040074682875a236453d4974f P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260130 02:57:53.684041 19647 raft_consensus.cc:2243] T dd64001040074682875a236453d4974f P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260130 02:57:53.684226 19645 raft_consensus.cc:2272] T dd64001040074682875a236453d4974f P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260130 02:57:53.684624 18874 catalog_manager.cc:5941] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4: Sending DeleteTablet for 3 replicas of tablet 00e6ed81b43d4e74aa6d648fdc1f6c65
I20260130 02:57:53.684566 19647 raft_consensus.cc:2272] T dd64001040074682875a236453d4974f P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260130 02:57:53.684211 19225 raft_consensus.cc:2468] T 00e6ed81b43d4e74aa6d648fdc1f6c65 P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78c2c46c7520492db35d51c244fb4f74 in term 1.
I20260130 02:57:53.686652 19314 leader_election.cc:304] T 00e6ed81b43d4e74aa6d648fdc1f6c65 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 371a36d219164974abdf848841528152, 78c2c46c7520492db35d51c244fb4f74; no voters:
I20260130 02:57:53.686925 19647 ts_tablet_manager.cc:1916] T dd64001040074682875a236453d4974f P d870164263bb44b6b780a557daea3656: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260130 02:57:53.688271 19203 tablet_service.cc:1552] Processing DeleteTablet for tablet 00e6ed81b43d4e74aa6d648fdc1f6c65 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:57:53 UTC) from {username='slave'} at 127.0.0.1:42870
I20260130 02:57:53.689134 19360 tablet_service.cc:1552] Processing DeleteTablet for tablet 00e6ed81b43d4e74aa6d648fdc1f6c65 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:57:53 UTC) from {username='slave'} at 127.0.0.1:60110
I20260130 02:57:53.690222 19205 tablet_service.cc:1552] Processing DeleteTablet for tablet dd64001040074682875a236453d4974f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:57:53 UTC) from {username='slave'} at 127.0.0.1:42870
I20260130 02:57:53.690496 19507 tablet_service.cc:1552] Processing DeleteTablet for tablet 00e6ed81b43d4e74aa6d648fdc1f6c65 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:57:53 UTC) from {username='slave'} at 127.0.0.1:59068
I20260130 02:57:53.690946 19646 tablet_replica.cc:333] T 00e6ed81b43d4e74aa6d648fdc1f6c65 P 371a36d219164974abdf848841528152: stopping tablet replica
I20260130 02:57:53.691670 19646 raft_consensus.cc:2243] T 00e6ed81b43d4e74aa6d648fdc1f6c65 P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260130 02:57:53.692018 19645 ts_tablet_manager.cc:1916] T dd64001040074682875a236453d4974f P 78c2c46c7520492db35d51c244fb4f74: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260130 02:57:53.692104 19646 raft_consensus.cc:2272] T 00e6ed81b43d4e74aa6d648fdc1f6c65 P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260130 02:57:53.692135 19663 raft_consensus.cc:2804] T 00e6ed81b43d4e74aa6d648fdc1f6c65 P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Leader election won for term 1
I20260130 02:57:53.692867 19663 raft_consensus.cc:697] T 00e6ed81b43d4e74aa6d648fdc1f6c65 P 78c2c46c7520492db35d51c244fb4f74 [term 1 LEADER]: Becoming Leader. State: Replica: 78c2c46c7520492db35d51c244fb4f74, State: Running, Role: LEADER
I20260130 02:57:53.693826 19663 consensus_queue.cc:237] T 00e6ed81b43d4e74aa6d648fdc1f6c65 P 78c2c46c7520492db35d51c244fb4f74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:57:53.694988 19646 ts_tablet_manager.cc:1916] T 00e6ed81b43d4e74aa6d648fdc1f6c65 P 371a36d219164974abdf848841528152: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260130 02:57:53.701438 19527 raft_consensus.cc:2468] T 00e6ed81b43d4e74aa6d648fdc1f6c65 P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78c2c46c7520492db35d51c244fb4f74 in term 1.
I20260130 02:57:53.711818 19625 raft_consensus.cc:359] T df2c8989d6c1480c97baa23db01141eb P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:57:53.719245 19625 raft_consensus.cc:385] T df2c8989d6c1480c97baa23db01141eb P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:57:53.719995 18870 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50284:
name: "default.Table_69ffd00165c6422292d9976b685d5d0d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260130 02:57:53.721757 18870 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_69ffd00165c6422292d9976b685d5d0d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260130 02:57:53.722669 19625 raft_consensus.cc:740] T df2c8989d6c1480c97baa23db01141eb P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d870164263bb44b6b780a557daea3656, State: Initialized, Role: FOLLOWER
I20260130 02:57:53.723237 19646 ts_tablet_manager.cc:1929] T 00e6ed81b43d4e74aa6d648fdc1f6c65 P 371a36d219164974abdf848841528152: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260130 02:57:53.723626 19646 log.cc:1199] T 00e6ed81b43d4e74aa6d648fdc1f6c65 P 371a36d219164974abdf848841528152: Deleting WAL directory at /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-0/wal/wals/00e6ed81b43d4e74aa6d648fdc1f6c65
I20260130 02:57:53.724274 19646 ts_tablet_manager.cc:1950] T 00e6ed81b43d4e74aa6d648fdc1f6c65 P 371a36d219164974abdf848841528152: Deleting consensus metadata
I20260130 02:57:53.724162 19625 consensus_queue.cc:260] T df2c8989d6c1480c97baa23db01141eb P d870164263bb44b6b780a557daea3656 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:57:53.727017 18861 catalog_manager.cc:4985] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021): tablet 00e6ed81b43d4e74aa6d648fdc1f6c65 (table default.Table_b2b8ae6f8ee74fc2be70b4cd5d16782e [id=18cadca4c17f42d4bcbb7db8c12045bc]) successfully deleted
I20260130 02:57:53.727968 18869 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50284:
table { table_name: "default.Table_3732a807f13b4ccea8c73b3a61c8c2e5" } new_table_name: "default.Table_39345d64a2bc4f46b0136c6c4155ea8b" modify_external_catalogs: true
I20260130 02:57:53.731084 19646 tablet_replica.cc:333] T dd64001040074682875a236453d4974f P 371a36d219164974abdf848841528152: stopping tablet replica
I20260130 02:57:53.731056 18872 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50284:
name: "default.Table_963e520f5c774402ab1b268cdfac33c9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260130 02:57:53.731797 19646 raft_consensus.cc:2243] T dd64001040074682875a236453d4974f P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Raft consensus shutting down.
W20260130 02:57:53.732632 18872 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_963e520f5c774402ab1b268cdfac33c9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260130 02:57:53.732224 19646 raft_consensus.cc:2272] T dd64001040074682875a236453d4974f P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260130 02:57:53.738982 19646 ts_tablet_manager.cc:1916] T dd64001040074682875a236453d4974f P 371a36d219164974abdf848841528152: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260130 02:57:53.741297 19625 ts_tablet_manager.cc:1434] T df2c8989d6c1480c97baa23db01141eb P d870164263bb44b6b780a557daea3656: Time spent starting tablet: real 0.061s user 0.005s sys 0.000s
I20260130 02:57:53.747810 19625 tablet_bootstrap.cc:492] T 06b31d020b864c418476e6dcc1e4eafb P d870164263bb44b6b780a557daea3656: Bootstrap starting.
I20260130 02:57:53.766455 19380 raft_consensus.cc:1275] T 96584ca174b140f7ad1a2ca698baca71 P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Refusing update from remote peer 371a36d219164974abdf848841528152: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260130 02:57:53.767987 19527 raft_consensus.cc:1275] T 96584ca174b140f7ad1a2ca698baca71 P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Refusing update from remote peer 371a36d219164974abdf848841528152: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260130 02:57:53.768250 19635 consensus_queue.cc:1048] T 96584ca174b140f7ad1a2ca698baca71 P 371a36d219164974abdf848841528152 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260130 02:57:53.772001 18874 catalog_manager.cc:5746] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021) does not have the latest schema for tablet 06b31d020b864c418476e6dcc1e4eafb (table default.Table_5dc02861dbc440528863f809c8300429 [id=b78b791f363d4c0f906e2a4b21dfacc4]). Expected version 1 got 0
I20260130 02:57:53.776993 19635 consensus_queue.cc:1048] T 96584ca174b140f7ad1a2ca698baca71 P 371a36d219164974abdf848841528152 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260130 02:57:53.776954 18868 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50284:
name: "default.Table_97c3def679ff49dcb74a992828c65cde"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260130 02:57:53.778997 18868 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_97c3def679ff49dcb74a992828c65cde in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260130 02:57:53.782330 18870 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50284:
table { table_name: "default.Table_5b8236964cd94924b945ec3d68a500fc" } new_table_name: "default.Table_f03fffd6ca4c48cdaf9b143f1c753710" modify_external_catalogs: true
I20260130 02:57:53.792723 19626 tablet_bootstrap.cc:492] T df2c8989d6c1480c97baa23db01141eb P 78c2c46c7520492db35d51c244fb4f74: No bootstrap required, opened a new log
I20260130 02:57:53.793147 19626 ts_tablet_manager.cc:1403] T df2c8989d6c1480c97baa23db01141eb P 78c2c46c7520492db35d51c244fb4f74: Time spent bootstrapping tablet: real 0.118s user 0.007s sys 0.004s
I20260130 02:57:53.795677 19626 raft_consensus.cc:359] T df2c8989d6c1480c97baa23db01141eb P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:57:53.796345 19626 raft_consensus.cc:385] T df2c8989d6c1480c97baa23db01141eb P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:57:53.796772 19626 raft_consensus.cc:740] T df2c8989d6c1480c97baa23db01141eb P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78c2c46c7520492db35d51c244fb4f74, State: Initialized, Role: FOLLOWER
I20260130 02:57:53.797457 19626 consensus_queue.cc:260] T df2c8989d6c1480c97baa23db01141eb P 78c2c46c7520492db35d51c244fb4f74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:57:53.799628 19626 ts_tablet_manager.cc:1434] T df2c8989d6c1480c97baa23db01141eb P 78c2c46c7520492db35d51c244fb4f74: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260130 02:57:53.800473 19626 tablet_bootstrap.cc:492] T 06b31d020b864c418476e6dcc1e4eafb P 78c2c46c7520492db35d51c244fb4f74: Bootstrap starting.
I20260130 02:57:53.808440 19360 tablet_service.cc:1505] Processing CreateTablet for tablet 927c90b84d7449efa36bc5e2035bf0da (DEFAULT_TABLE table=default.Table_0f26dbdebb8c49f6997d31582bbab3d1 [id=28156e0843674fce88f5914a3e4195ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:57:53.809633 19360 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 927c90b84d7449efa36bc5e2035bf0da. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:57:53.812809 19625 tablet_bootstrap.cc:654] T 06b31d020b864c418476e6dcc1e4eafb P d870164263bb44b6b780a557daea3656: Neither blocks nor log segments found. Creating new log.
I20260130 02:57:53.819002 19204 tablet_service.cc:1505] Processing CreateTablet for tablet 927c90b84d7449efa36bc5e2035bf0da (DEFAULT_TABLE table=default.Table_0f26dbdebb8c49f6997d31582bbab3d1 [id=28156e0843674fce88f5914a3e4195ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:57:53.819921 19507 tablet_service.cc:1505] Processing CreateTablet for tablet 927c90b84d7449efa36bc5e2035bf0da (DEFAULT_TABLE table=default.Table_0f26dbdebb8c49f6997d31582bbab3d1 [id=28156e0843674fce88f5914a3e4195ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:57:53.822122 19507 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 927c90b84d7449efa36bc5e2035bf0da. 1 dirs total, 0 dirs full, 0 dirs failed
W20260130 02:57:53.822898 18870 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet df2c8989d6c1480c97baa23db01141eb consensus state has no leader
I20260130 02:57:53.831641 19358 tablet_service.cc:1505] Processing CreateTablet for tablet f73338d4ce584880a9bec751d45ae753 (DEFAULT_TABLE table=default.Table_8e5ff21c1d224aab962d02463ea05a49 [id=74a7608fe84149d9bf37bcea9fe47b00]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:57:53.832809 19358 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f73338d4ce584880a9bec751d45ae753. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:57:53.834008 19205 tablet_service.cc:1505] Processing CreateTablet for tablet f73338d4ce584880a9bec751d45ae753 (DEFAULT_TABLE table=default.Table_8e5ff21c1d224aab962d02463ea05a49 [id=74a7608fe84149d9bf37bcea9fe47b00]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:57:53.835610 19204 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 927c90b84d7449efa36bc5e2035bf0da. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:57:53.836808 19508 tablet_service.cc:1505] Processing CreateTablet for tablet f73338d4ce584880a9bec751d45ae753 (DEFAULT_TABLE table=default.Table_8e5ff21c1d224aab962d02463ea05a49 [id=74a7608fe84149d9bf37bcea9fe47b00]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:57:53.837646 19205 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f73338d4ce584880a9bec751d45ae753. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:57:53.841446 19203 tablet_service.cc:1505] Processing CreateTablet for tablet 79e28c3a566c4521ac2453bbaa3567ed (DEFAULT_TABLE table=default.Table_b8f4e9cf27f44a479e1adad0b7ce4e33 [id=26422816af0c430880cb44e1b0cce534]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:57:53.842800 19203 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 79e28c3a566c4521ac2453bbaa3567ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:57:53.834945 19645 ts_tablet_manager.cc:1929] T dd64001040074682875a236453d4974f P 78c2c46c7520492db35d51c244fb4f74: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260130 02:57:53.844657 19357 tablet_service.cc:1505] Processing CreateTablet for tablet 79e28c3a566c4521ac2453bbaa3567ed (DEFAULT_TABLE table=default.Table_b8f4e9cf27f44a479e1adad0b7ce4e33 [id=26422816af0c430880cb44e1b0cce534]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:57:53.845933 19357 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 79e28c3a566c4521ac2453bbaa3567ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:57:53.851258 19646 ts_tablet_manager.cc:1929] T dd64001040074682875a236453d4974f P 371a36d219164974abdf848841528152: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260130 02:57:53.851657 19646 log.cc:1199] T dd64001040074682875a236453d4974f P 371a36d219164974abdf848841528152: Deleting WAL directory at /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-0/wal/wals/dd64001040074682875a236453d4974f
I20260130 02:57:53.852300 19646 ts_tablet_manager.cc:1950] T dd64001040074682875a236453d4974f P 371a36d219164974abdf848841528152: Deleting consensus metadata
I20260130 02:57:53.844322 19645 log.cc:1199] T dd64001040074682875a236453d4974f P 78c2c46c7520492db35d51c244fb4f74: Deleting WAL directory at /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-1/wal/wals/dd64001040074682875a236453d4974f
I20260130 02:57:53.855279 19645 ts_tablet_manager.cc:1950] T dd64001040074682875a236453d4974f P 78c2c46c7520492db35d51c244fb4f74: Deleting consensus metadata
I20260130 02:57:53.856029 18861 catalog_manager.cc:4985] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021): tablet dd64001040074682875a236453d4974f (table default.Table_76c537f09c384533b1c632de353f4dbb [id=e86c9f7d95b34a96baeb24e965c2f582]) successfully deleted
I20260130 02:57:53.857156 19508 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f73338d4ce584880a9bec751d45ae753. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:57:53.858295 19626 tablet_bootstrap.cc:654] T 06b31d020b864c418476e6dcc1e4eafb P 78c2c46c7520492db35d51c244fb4f74: Neither blocks nor log segments found. Creating new log.
I20260130 02:57:53.859426 19506 tablet_service.cc:1505] Processing CreateTablet for tablet 79e28c3a566c4521ac2453bbaa3567ed (DEFAULT_TABLE table=default.Table_b8f4e9cf27f44a479e1adad0b7ce4e33 [id=26422816af0c430880cb44e1b0cce534]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:57:53.860527 19506 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 79e28c3a566c4521ac2453bbaa3567ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:57:53.861745 19647 ts_tablet_manager.cc:1929] T dd64001040074682875a236453d4974f P d870164263bb44b6b780a557daea3656: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260130 02:57:53.875366 18860 catalog_manager.cc:4985] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767): tablet dd64001040074682875a236453d4974f (table default.Table_76c537f09c384533b1c632de353f4dbb [id=e86c9f7d95b34a96baeb24e965c2f582]) successfully deleted
I20260130 02:57:53.875754 19645 tablet_replica.cc:333] T 00e6ed81b43d4e74aa6d648fdc1f6c65 P 78c2c46c7520492db35d51c244fb4f74: stopping tablet replica
I20260130 02:57:53.876497 19645 raft_consensus.cc:2243] T 00e6ed81b43d4e74aa6d648fdc1f6c65 P 78c2c46c7520492db35d51c244fb4f74 [term 1 LEADER]: Raft consensus shutting down.
I20260130 02:57:53.877275 19645 pending_rounds.cc:70] T 00e6ed81b43d4e74aa6d648fdc1f6c65 P 78c2c46c7520492db35d51c244fb4f74: Trying to abort 1 pending ops.
I20260130 02:57:53.877579 19645 pending_rounds.cc:77] T 00e6ed81b43d4e74aa6d648fdc1f6c65 P 78c2c46c7520492db35d51c244fb4f74: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7248862714660212736 op_type: NO_OP noop_request { }
I20260130 02:57:53.878011 19645 raft_consensus.cc:2889] T 00e6ed81b43d4e74aa6d648fdc1f6c65 P 78c2c46c7520492db35d51c244fb4f74 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20260130 02:57:53.885304 18870 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50284:
table { table_name: "default.Table_8e5ff21c1d224aab962d02463ea05a49" } modify_external_catalogs: true
I20260130 02:57:53.886019 18870 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50284:
table { table_name: "default.Table_8e5ff21c1d224aab962d02463ea05a49" } modify_external_catalogs: true
I20260130 02:57:53.862139 19647 log.cc:1199] T dd64001040074682875a236453d4974f P d870164263bb44b6b780a557daea3656: Deleting WAL directory at /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-2/wal/wals/dd64001040074682875a236453d4974f
I20260130 02:57:53.892001 19647 ts_tablet_manager.cc:1950] T dd64001040074682875a236453d4974f P d870164263bb44b6b780a557daea3656: Deleting consensus metadata
I20260130 02:57:53.893448 18868 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50284:
table { table_name: "default.Table_b8f4e9cf27f44a479e1adad0b7ce4e33" } modify_external_catalogs: true
I20260130 02:57:53.893996 18868 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50284:
table { table_name: "default.Table_b8f4e9cf27f44a479e1adad0b7ce4e33" } modify_external_catalogs: true
I20260130 02:57:53.909906 18860 catalog_manager.cc:4985] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): tablet dd64001040074682875a236453d4974f (table default.Table_76c537f09c384533b1c632de353f4dbb [id=e86c9f7d95b34a96baeb24e965c2f582]) successfully deleted
I20260130 02:57:53.911135 18869 catalog_manager.cc:5746] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767) does not have the latest schema for tablet 96584ca174b140f7ad1a2ca698baca71 (table default.Table_39345d64a2bc4f46b0136c6c4155ea8b [id=1515db389f284e2f9ed1a3cd0061d0f1]). Expected version 4 got 0
I20260130 02:57:53.914208 18872 catalog_manager.cc:5746] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571) does not have the latest schema for tablet 96584ca174b140f7ad1a2ca698baca71 (table default.Table_39345d64a2bc4f46b0136c6c4155ea8b [id=1515db389f284e2f9ed1a3cd0061d0f1]). Expected version 4 got 0
I20260130 02:57:53.878384 19645 raft_consensus.cc:2272] T 00e6ed81b43d4e74aa6d648fdc1f6c65 P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260130 02:57:53.932353 19645 ts_tablet_manager.cc:1916] T 00e6ed81b43d4e74aa6d648fdc1f6c65 P 78c2c46c7520492db35d51c244fb4f74: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260130 02:57:53.932973 18869 catalog_manager.cc:5746] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021) does not have the latest schema for tablet 96584ca174b140f7ad1a2ca698baca71 (table default.Table_39345d64a2bc4f46b0136c6c4155ea8b [id=1515db389f284e2f9ed1a3cd0061d0f1]). Expected version 4 got 0
I20260130 02:57:53.936533 19624 tablet_bootstrap.cc:492] T 927c90b84d7449efa36bc5e2035bf0da P 371a36d219164974abdf848841528152: Bootstrap starting.
I20260130 02:57:53.938840 19673 tablet.cc:1722] T 96584ca174b140f7ad1a2ca698baca71 P 371a36d219164974abdf848841528152: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20260130 02:57:53.942317 19624 tablet_bootstrap.cc:654] T 927c90b84d7449efa36bc5e2035bf0da P 371a36d219164974abdf848841528152: Neither blocks nor log segments found. Creating new log.
I20260130 02:57:53.937954 18872 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50284:
name: "default.Table_d1b81d6a8e704777b0fa35cf23dc91e0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260130 02:57:53.944885 19674 tablet.cc:1722] T 96584ca174b140f7ad1a2ca698baca71 P 78c2c46c7520492db35d51c244fb4f74: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
W20260130 02:57:53.945633 18872 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d1b81d6a8e704777b0fa35cf23dc91e0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260130 02:57:53.965039 19626 tablet_bootstrap.cc:492] T 06b31d020b864c418476e6dcc1e4eafb P 78c2c46c7520492db35d51c244fb4f74: No bootstrap required, opened a new log
I20260130 02:57:53.965517 19626 ts_tablet_manager.cc:1403] T 06b31d020b864c418476e6dcc1e4eafb P 78c2c46c7520492db35d51c244fb4f74: Time spent bootstrapping tablet: real 0.165s user 0.031s sys 0.048s
I20260130 02:57:53.968030 19626 raft_consensus.cc:359] T 06b31d020b864c418476e6dcc1e4eafb P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } }
I20260130 02:57:53.969643 18869 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50284:
table { table_name: "default.Table_963e520f5c774402ab1b268cdfac33c9" } new_table_name: "default.Table_1a735ba1c9994f81ab61385e277b300c" modify_external_catalogs: true
I20260130 02:57:53.968741 19626 raft_consensus.cc:385] T 06b31d020b864c418476e6dcc1e4eafb P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:57:53.972698 19626 raft_consensus.cc:740] T 06b31d020b864c418476e6dcc1e4eafb P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78c2c46c7520492db35d51c244fb4f74, State: Initialized, Role: FOLLOWER
I20260130 02:57:53.973378 19626 consensus_queue.cc:260] T 06b31d020b864c418476e6dcc1e4eafb P 78c2c46c7520492db35d51c244fb4f74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } }
I20260130 02:57:53.978529 18875 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50284:
name: "default.Table_9428a7751159470793188f91cdcba4f3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260130 02:57:53.984714 18871 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50284:
name: "default.Table_1c77605822574b10a05074a093ad30a3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260130 02:57:53.981678 19630 raft_consensus.cc:493] T df2c8989d6c1480c97baa23db01141eb P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:57:53.986130 19630 raft_consensus.cc:515] T df2c8989d6c1480c97baa23db01141eb P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:57:53.987922 19630 leader_election.cc:290] T df2c8989d6c1480c97baa23db01141eb P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767), 371a36d219164974abdf848841528152 (127.18.100.65:42021)
I20260130 02:57:53.993157 19624 tablet_bootstrap.cc:492] T 927c90b84d7449efa36bc5e2035bf0da P 371a36d219164974abdf848841528152: No bootstrap required, opened a new log
I20260130 02:57:53.993570 19624 ts_tablet_manager.cc:1403] T 927c90b84d7449efa36bc5e2035bf0da P 371a36d219164974abdf848841528152: Time spent bootstrapping tablet: real 0.057s user 0.013s sys 0.001s
I20260130 02:57:53.994316 19669 mvcc.cc:204] Tried to move back new op lower bound from 7248862714724294656 to 7248862714117087232. Current Snapshot: MvccSnapshot[applied={T|T < 7248862714724294656}]
I20260130 02:57:53.996232 19380 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "df2c8989d6c1480c97baa23db01141eb" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74" is_pre_election: true
I20260130 02:57:53.996832 19380 raft_consensus.cc:2468] T df2c8989d6c1480c97baa23db01141eb P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d870164263bb44b6b780a557daea3656 in term 0.
I20260130 02:57:53.998250 19462 leader_election.cc:304] T df2c8989d6c1480c97baa23db01141eb P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78c2c46c7520492db35d51c244fb4f74, d870164263bb44b6b780a557daea3656; no voters:
I20260130 02:57:53.997612 19626 ts_tablet_manager.cc:1434] T 06b31d020b864c418476e6dcc1e4eafb P 78c2c46c7520492db35d51c244fb4f74: Time spent starting tablet: real 0.032s user 0.009s sys 0.001s
I20260130 02:57:53.998646 19624 raft_consensus.cc:359] T 927c90b84d7449efa36bc5e2035bf0da P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } }
I20260130 02:57:54.000075 19624 raft_consensus.cc:385] T 927c90b84d7449efa36bc5e2035bf0da P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:57:54.000739 19645 ts_tablet_manager.cc:1929] T 00e6ed81b43d4e74aa6d648fdc1f6c65 P 78c2c46c7520492db35d51c244fb4f74: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260130 02:57:54.001137 19645 log.cc:1199] T 00e6ed81b43d4e74aa6d648fdc1f6c65 P 78c2c46c7520492db35d51c244fb4f74: Deleting WAL directory at /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-1/wal/wals/00e6ed81b43d4e74aa6d648fdc1f6c65
I20260130 02:57:54.001859 19645 ts_tablet_manager.cc:1950] T 00e6ed81b43d4e74aa6d648fdc1f6c65 P 78c2c46c7520492db35d51c244fb4f74: Deleting consensus metadata
I20260130 02:57:54.003849 19626 tablet_bootstrap.cc:492] T 927c90b84d7449efa36bc5e2035bf0da P 78c2c46c7520492db35d51c244fb4f74: Bootstrap starting.
I20260130 02:57:54.008004 19630 raft_consensus.cc:2804] T df2c8989d6c1480c97baa23db01141eb P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260130 02:57:54.009491 19630 raft_consensus.cc:493] T df2c8989d6c1480c97baa23db01141eb P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260130 02:57:54.009799 19630 raft_consensus.cc:3060] T df2c8989d6c1480c97baa23db01141eb P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:57:54.012178 19635 raft_consensus.cc:493] T 06b31d020b864c418476e6dcc1e4eafb P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:57:54.012701 19635 raft_consensus.cc:515] T 06b31d020b864c418476e6dcc1e4eafb P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } }
I20260130 02:57:54.014457 19635 leader_election.cc:290] T 06b31d020b864c418476e6dcc1e4eafb P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767), d870164263bb44b6b780a557daea3656 (127.18.100.67:39571)
I20260130 02:57:54.016237 19380 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "06b31d020b864c418476e6dcc1e4eafb" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74" is_pre_election: true
I20260130 02:57:54.016834 19380 raft_consensus.cc:2468] T 06b31d020b864c418476e6dcc1e4eafb P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 371a36d219164974abdf848841528152 in term 0.
I20260130 02:57:54.017374 18873 catalog_manager.cc:5746] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021) does not have the latest schema for tablet 96584ca174b140f7ad1a2ca698baca71 (table default.Table_39345d64a2bc4f46b0136c6c4155ea8b [id=1515db389f284e2f9ed1a3cd0061d0f1]). Expected version 4 got 3
I20260130 02:57:54.017896 19159 leader_election.cc:304] T 06b31d020b864c418476e6dcc1e4eafb P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 371a36d219164974abdf848841528152, 78c2c46c7520492db35d51c244fb4f74; no voters:
I20260130 02:57:54.018675 19635 raft_consensus.cc:2804] T 06b31d020b864c418476e6dcc1e4eafb P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260130 02:57:54.018982 19635 raft_consensus.cc:493] T 06b31d020b864c418476e6dcc1e4eafb P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260130 02:57:54.019272 19635 raft_consensus.cc:3060] T 06b31d020b864c418476e6dcc1e4eafb P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:57:54.019203 18867 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50284:
name: "default.Table_23120efe2754472c9f41c54904af4081"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260130 02:57:54.021431 18862 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet df2c8989d6c1480c97baa23db01141eb (table default.Table_f03fffd6ca4c48cdaf9b143f1c753710 [id=747af828147149efb7b7b8d33c49286d], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet df2c8989d6c1480c97baa23db01141eb consensus state has no leader
I20260130 02:57:54.028200 18873 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50284:
table { table_name: "default.Table_5dc02861dbc440528863f809c8300429" } new_table_name: "default.Table_e9d466c5c67e4df1a06a8eb665da9570" modify_external_catalogs: true
I20260130 02:57:54.032672 19625 tablet_bootstrap.cc:492] T 06b31d020b864c418476e6dcc1e4eafb P d870164263bb44b6b780a557daea3656: No bootstrap required, opened a new log
I20260130 02:57:54.033628 19630 raft_consensus.cc:515] T df2c8989d6c1480c97baa23db01141eb P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:57:54.033947 19625 ts_tablet_manager.cc:1403] T 06b31d020b864c418476e6dcc1e4eafb P d870164263bb44b6b780a557daea3656: Time spent bootstrapping tablet: real 0.286s user 0.010s sys 0.016s
I20260130 02:57:54.036293 19380 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "df2c8989d6c1480c97baa23db01141eb" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74"
I20260130 02:57:54.036865 19380 raft_consensus.cc:3060] T df2c8989d6c1480c97baa23db01141eb P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:57:54.038684 19626 tablet_bootstrap.cc:654] T 927c90b84d7449efa36bc5e2035bf0da P 78c2c46c7520492db35d51c244fb4f74: Neither blocks nor log segments found. Creating new log.
I20260130 02:57:54.041101 19653 raft_consensus.cc:493] T df2c8989d6c1480c97baa23db01141eb P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:57:54.041397 19647 tablet_replica.cc:333] T 00e6ed81b43d4e74aa6d648fdc1f6c65 P d870164263bb44b6b780a557daea3656: stopping tablet replica
I20260130 02:57:54.041586 19653 raft_consensus.cc:515] T df2c8989d6c1480c97baa23db01141eb P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:57:54.042104 19647 raft_consensus.cc:2243] T 00e6ed81b43d4e74aa6d648fdc1f6c65 P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260130 02:57:54.042537 19647 raft_consensus.cc:2272] T 00e6ed81b43d4e74aa6d648fdc1f6c65 P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260130 02:57:54.044492 19647 ts_tablet_manager.cc:1916] T 00e6ed81b43d4e74aa6d648fdc1f6c65 P d870164263bb44b6b780a557daea3656: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260130 02:57:54.048179 19380 raft_consensus.cc:2468] T df2c8989d6c1480c97baa23db01141eb P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d870164263bb44b6b780a557daea3656 in term 1.
I20260130 02:57:54.000326 19624 raft_consensus.cc:740] T 927c90b84d7449efa36bc5e2035bf0da P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 371a36d219164974abdf848841528152, State: Initialized, Role: FOLLOWER
I20260130 02:57:54.052325 19635 raft_consensus.cc:515] T 06b31d020b864c418476e6dcc1e4eafb P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } }
I20260130 02:57:54.054960 19379 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "df2c8989d6c1480c97baa23db01141eb" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74" is_pre_election: true
I20260130 02:57:54.066357 19379 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "06b31d020b864c418476e6dcc1e4eafb" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74"
I20260130 02:57:54.066941 19379 raft_consensus.cc:3060] T 06b31d020b864c418476e6dcc1e4eafb P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:57:54.073298 19379 raft_consensus.cc:2468] T 06b31d020b864c418476e6dcc1e4eafb P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 371a36d219164974abdf848841528152 in term 1.
I20260130 02:57:54.056165 19653 leader_election.cc:290] T df2c8989d6c1480c97baa23db01141eb P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767), d870164263bb44b6b780a557daea3656 (127.18.100.67:39571)
I20260130 02:57:54.032649 18860 catalog_manager.cc:4985] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767): tablet 00e6ed81b43d4e74aa6d648fdc1f6c65 (table default.Table_b2b8ae6f8ee74fc2be70b4cd5d16782e [id=18cadca4c17f42d4bcbb7db8c12045bc]) successfully deleted
W20260130 02:57:54.077559 18875 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9428a7751159470793188f91cdcba4f3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260130 02:57:54.077750 18860 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.18.100.66:40767, user_credentials={real_user=slave}} blocked reactor thread for 45134.5us
W20260130 02:57:54.078729 18871 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1c77605822574b10a05074a093ad30a3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260130 02:57:54.079665 18867 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_23120efe2754472c9f41c54904af4081 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260130 02:57:54.096763 19675 tablet.cc:1722] T 96584ca174b140f7ad1a2ca698baca71 P d870164263bb44b6b780a557daea3656: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20260130 02:57:54.088421 19626 tablet_bootstrap.cc:492] T 927c90b84d7449efa36bc5e2035bf0da P 78c2c46c7520492db35d51c244fb4f74: No bootstrap required, opened a new log
I20260130 02:57:54.102720 19674 tablet.cc:1717] T 96584ca174b140f7ad1a2ca698baca71 P 78c2c46c7520492db35d51c244fb4f74: Skipping requested alter to schema version 3, tablet already version 3
I20260130 02:57:54.104374 19626 ts_tablet_manager.cc:1403] T 927c90b84d7449efa36bc5e2035bf0da P 78c2c46c7520492db35d51c244fb4f74: Time spent bootstrapping tablet: real 0.102s user 0.018s sys 0.013s
I20260130 02:57:54.106287 19673 tablet.cc:1717] T 96584ca174b140f7ad1a2ca698baca71 P 371a36d219164974abdf848841528152: Skipping requested alter to schema version 3, tablet already version 3
I20260130 02:57:54.107061 19626 raft_consensus.cc:359] T 927c90b84d7449efa36bc5e2035bf0da P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } }
I20260130 02:57:54.107585 19626 raft_consensus.cc:385] T 927c90b84d7449efa36bc5e2035bf0da P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:57:54.107826 19626 raft_consensus.cc:740] T 927c90b84d7449efa36bc5e2035bf0da P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78c2c46c7520492db35d51c244fb4f74, State: Initialized, Role: FOLLOWER
I20260130 02:57:54.108323 19626 consensus_queue.cc:260] T 927c90b84d7449efa36bc5e2035bf0da P 78c2c46c7520492db35d51c244fb4f74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } }
W20260130 02:57:54.108681 18869 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 38a7a10e36874183bf9fe55a38d829f6 has no consensus state
I20260130 02:57:54.110036 19626 ts_tablet_manager.cc:1434] T 927c90b84d7449efa36bc5e2035bf0da P 78c2c46c7520492db35d51c244fb4f74: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20260130 02:57:54.110740 19626 tablet_bootstrap.cc:492] T f73338d4ce584880a9bec751d45ae753 P 78c2c46c7520492db35d51c244fb4f74: Bootstrap starting.
W20260130 02:57:54.112603 18873 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 06b31d020b864c418476e6dcc1e4eafb consensus state has no leader
I20260130 02:57:54.119786 19625 raft_consensus.cc:359] T 06b31d020b864c418476e6dcc1e4eafb P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } }
I20260130 02:57:54.120476 19625 raft_consensus.cc:385] T 06b31d020b864c418476e6dcc1e4eafb P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:57:54.120759 19625 raft_consensus.cc:740] T 06b31d020b864c418476e6dcc1e4eafb P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d870164263bb44b6b780a557daea3656, State: Initialized, Role: FOLLOWER
I20260130 02:57:54.123010 19626 tablet_bootstrap.cc:654] T f73338d4ce584880a9bec751d45ae753 P 78c2c46c7520492db35d51c244fb4f74: Neither blocks nor log segments found. Creating new log.
I20260130 02:57:54.123327 19625 consensus_queue.cc:260] T 06b31d020b864c418476e6dcc1e4eafb P d870164263bb44b6b780a557daea3656 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } }
I20260130 02:57:54.130367 19462 leader_election.cc:304] T df2c8989d6c1480c97baa23db01141eb P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78c2c46c7520492db35d51c244fb4f74, d870164263bb44b6b780a557daea3656; no voters:
I20260130 02:57:54.144440 19626 tablet_bootstrap.cc:492] T f73338d4ce584880a9bec751d45ae753 P 78c2c46c7520492db35d51c244fb4f74: No bootstrap required, opened a new log
I20260130 02:57:54.144891 19626 ts_tablet_manager.cc:1403] T f73338d4ce584880a9bec751d45ae753 P 78c2c46c7520492db35d51c244fb4f74: Time spent bootstrapping tablet: real 0.034s user 0.010s sys 0.000s
I20260130 02:57:54.056685 19624 consensus_queue.cc:260] T 927c90b84d7449efa36bc5e2035bf0da P 371a36d219164974abdf848841528152 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } }
I20260130 02:57:54.146292 19225 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "df2c8989d6c1480c97baa23db01141eb" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152" is_pre_election: true
I20260130 02:57:54.146903 19225 raft_consensus.cc:2468] T df2c8989d6c1480c97baa23db01141eb P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d870164263bb44b6b780a557daea3656 in term 0.
I20260130 02:57:54.147372 19624 ts_tablet_manager.cc:1434] T 927c90b84d7449efa36bc5e2035bf0da P 371a36d219164974abdf848841528152: Time spent starting tablet: real 0.154s user 0.007s sys 0.000s
I20260130 02:57:54.147178 19626 raft_consensus.cc:359] T f73338d4ce584880a9bec751d45ae753 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } }
I20260130 02:57:54.147753 19626 raft_consensus.cc:385] T f73338d4ce584880a9bec751d45ae753 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:57:54.147985 19626 raft_consensus.cc:740] T f73338d4ce584880a9bec751d45ae753 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78c2c46c7520492db35d51c244fb4f74, State: Initialized, Role: FOLLOWER
I20260130 02:57:54.148591 19624 tablet_bootstrap.cc:492] T f73338d4ce584880a9bec751d45ae753 P 371a36d219164974abdf848841528152: Bootstrap starting.
I20260130 02:57:54.148528 19626 consensus_queue.cc:260] T f73338d4ce584880a9bec751d45ae753 P 78c2c46c7520492db35d51c244fb4f74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } }
I20260130 02:57:54.150319 19626 ts_tablet_manager.cc:1434] T f73338d4ce584880a9bec751d45ae753 P 78c2c46c7520492db35d51c244fb4f74: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20260130 02:57:54.151104 19626 tablet_bootstrap.cc:492] T 79e28c3a566c4521ac2453bbaa3567ed P 78c2c46c7520492db35d51c244fb4f74: Bootstrap starting.
I20260130 02:57:54.154141 19527 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "df2c8989d6c1480c97baa23db01141eb" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
I20260130 02:57:54.154932 19527 raft_consensus.cc:2393] T df2c8989d6c1480c97baa23db01141eb P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 371a36d219164974abdf848841528152 in current term 1: Already voted for candidate d870164263bb44b6b780a557daea3656 in this term.
I20260130 02:57:54.158607 19677 raft_consensus.cc:2804] T df2c8989d6c1480c97baa23db01141eb P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Leader election won for term 1
I20260130 02:57:54.159045 19677 raft_consensus.cc:697] T df2c8989d6c1480c97baa23db01141eb P d870164263bb44b6b780a557daea3656 [term 1 LEADER]: Becoming Leader. State: Replica: d870164263bb44b6b780a557daea3656, State: Running, Role: LEADER
I20260130 02:57:54.159842 19677 consensus_queue.cc:237] T df2c8989d6c1480c97baa23db01141eb P d870164263bb44b6b780a557daea3656 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:57:54.154700 19528 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "06b31d020b864c418476e6dcc1e4eafb" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
I20260130 02:57:54.162859 19528 raft_consensus.cc:2468] T 06b31d020b864c418476e6dcc1e4eafb P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 371a36d219164974abdf848841528152 in term 0.
I20260130 02:57:54.166328 19626 tablet_bootstrap.cc:654] T 79e28c3a566c4521ac2453bbaa3567ed P 78c2c46c7520492db35d51c244fb4f74: Neither blocks nor log segments found. Creating new log.
I20260130 02:57:54.169924 19635 leader_election.cc:290] T 06b31d020b864c418476e6dcc1e4eafb P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 election: Requested vote from peers 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767), d870164263bb44b6b780a557daea3656 (127.18.100.67:39571)
I20260130 02:57:54.170749 19630 leader_election.cc:290] T df2c8989d6c1480c97baa23db01141eb P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 election: Requested vote from peers 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767), 371a36d219164974abdf848841528152 (127.18.100.65:42021)
I20260130 02:57:54.171774 19225 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "df2c8989d6c1480c97baa23db01141eb" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152"
I20260130 02:57:54.172322 19225 raft_consensus.cc:3060] T df2c8989d6c1480c97baa23db01141eb P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Advancing to term 1
W20260130 02:57:54.172709 19159 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.100.66:40767, user_credentials={real_user=slave}} blocked reactor thread for 69459us
I20260130 02:57:54.177945 19159 leader_election.cc:304] T df2c8989d6c1480c97baa23db01141eb P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 371a36d219164974abdf848841528152; no voters: 78c2c46c7520492db35d51c244fb4f74, d870164263bb44b6b780a557daea3656
I20260130 02:57:54.179390 19528 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "06b31d020b864c418476e6dcc1e4eafb" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656"
I20260130 02:57:54.179915 19528 raft_consensus.cc:3060] T 06b31d020b864c418476e6dcc1e4eafb P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:57:54.179566 19159 leader_election.cc:304] T 06b31d020b864c418476e6dcc1e4eafb P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 371a36d219164974abdf848841528152, 78c2c46c7520492db35d51c244fb4f74; no voters:
I20260130 02:57:54.185242 19635 raft_consensus.cc:2804] T 06b31d020b864c418476e6dcc1e4eafb P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Leader election won for term 1
I20260130 02:57:54.185750 19635 raft_consensus.cc:697] T 06b31d020b864c418476e6dcc1e4eafb P 371a36d219164974abdf848841528152 [term 1 LEADER]: Becoming Leader. State: Replica: 371a36d219164974abdf848841528152, State: Running, Role: LEADER
I20260130 02:57:54.185848 19528 raft_consensus.cc:2468] T 06b31d020b864c418476e6dcc1e4eafb P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 371a36d219164974abdf848841528152 in term 1.
I20260130 02:57:54.178323 19225 raft_consensus.cc:2468] T df2c8989d6c1480c97baa23db01141eb P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d870164263bb44b6b780a557daea3656 in term 1.
I20260130 02:57:54.187969 19680 raft_consensus.cc:2749] T df2c8989d6c1480c97baa23db01141eb P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260130 02:57:54.186602 19635 consensus_queue.cc:237] T 06b31d020b864c418476e6dcc1e4eafb P 371a36d219164974abdf848841528152 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } }
I20260130 02:57:54.197798 19626 tablet_bootstrap.cc:492] T 79e28c3a566c4521ac2453bbaa3567ed P 78c2c46c7520492db35d51c244fb4f74: No bootstrap required, opened a new log
I20260130 02:57:54.198154 19626 ts_tablet_manager.cc:1403] T 79e28c3a566c4521ac2453bbaa3567ed P 78c2c46c7520492db35d51c244fb4f74: Time spent bootstrapping tablet: real 0.047s user 0.008s sys 0.001s
I20260130 02:57:54.200393 19625 ts_tablet_manager.cc:1434] T 06b31d020b864c418476e6dcc1e4eafb P d870164263bb44b6b780a557daea3656: Time spent starting tablet: real 0.166s user 0.006s sys 0.000s
I20260130 02:57:54.200207 19626 raft_consensus.cc:359] T 79e28c3a566c4521ac2453bbaa3567ed P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } }
I20260130 02:57:54.200846 19626 raft_consensus.cc:385] T 79e28c3a566c4521ac2453bbaa3567ed P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:57:54.201089 19626 raft_consensus.cc:740] T 79e28c3a566c4521ac2453bbaa3567ed P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78c2c46c7520492db35d51c244fb4f74, State: Initialized, Role: FOLLOWER
I20260130 02:57:54.201308 19625 tablet_bootstrap.cc:492] T f73338d4ce584880a9bec751d45ae753 P d870164263bb44b6b780a557daea3656: Bootstrap starting.
I20260130 02:57:54.201618 19626 consensus_queue.cc:260] T 79e28c3a566c4521ac2453bbaa3567ed P 78c2c46c7520492db35d51c244fb4f74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } }
I20260130 02:57:54.203441 19626 ts_tablet_manager.cc:1434] T 79e28c3a566c4521ac2453bbaa3567ed P 78c2c46c7520492db35d51c244fb4f74: Time spent starting tablet: real 0.005s user 0.002s sys 0.004s
I20260130 02:57:54.211664 18874 catalog_manager.cc:5746] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767) does not have the latest schema for tablet 96584ca174b140f7ad1a2ca698baca71 (table default.Table_39345d64a2bc4f46b0136c6c4155ea8b [id=1515db389f284e2f9ed1a3cd0061d0f1]). Expected version 4 got 0
I20260130 02:57:54.212261 18874 catalog_manager.cc:5746] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767) does not have the latest schema for tablet df2c8989d6c1480c97baa23db01141eb (table default.Table_f03fffd6ca4c48cdaf9b143f1c753710 [id=747af828147149efb7b7b8d33c49286d]). Expected version 1 got 0
I20260130 02:57:54.214947 19357 tablet_service.cc:1505] Processing CreateTablet for tablet 38a7a10e36874183bf9fe55a38d829f6 (DEFAULT_TABLE table=default.Table_1a735ba1c9994f81ab61385e277b300c [id=27adf356fd004597a54c7a9887dd3ca8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:57:54.216020 19357 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 38a7a10e36874183bf9fe55a38d829f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:57:54.218807 19203 tablet_service.cc:1505] Processing CreateTablet for tablet 38a7a10e36874183bf9fe55a38d829f6 (DEFAULT_TABLE table=default.Table_1a735ba1c9994f81ab61385e277b300c [id=27adf356fd004597a54c7a9887dd3ca8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:57:54.219852 19203 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 38a7a10e36874183bf9fe55a38d829f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:57:54.228370 19647 ts_tablet_manager.cc:1929] T 00e6ed81b43d4e74aa6d648fdc1f6c65 P d870164263bb44b6b780a557daea3656: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260130 02:57:54.228760 19647 log.cc:1199] T 00e6ed81b43d4e74aa6d648fdc1f6c65 P d870164263bb44b6b780a557daea3656: Deleting WAL directory at /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-2/wal/wals/00e6ed81b43d4e74aa6d648fdc1f6c65
I20260130 02:57:54.229382 19647 ts_tablet_manager.cc:1950] T 00e6ed81b43d4e74aa6d648fdc1f6c65 P d870164263bb44b6b780a557daea3656: Deleting consensus metadata
I20260130 02:57:54.232653 19673 tablet.cc:1722] T 96584ca174b140f7ad1a2ca698baca71 P 371a36d219164974abdf848841528152: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20260130 02:57:54.238155 18869 catalog_manager.cc:5746] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571) does not have the latest schema for tablet 96584ca174b140f7ad1a2ca698baca71 (table default.Table_39345d64a2bc4f46b0136c6c4155ea8b [id=1515db389f284e2f9ed1a3cd0061d0f1]). Expected version 4 got 0
I20260130 02:57:54.242784 18868 catalog_manager.cc:5941] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4: Sending DeleteTablet for 3 replicas of tablet 79e28c3a566c4521ac2453bbaa3567ed
I20260130 02:57:54.244686 19664 mvcc.cc:204] Tried to move back new op lower bound from 7248862716586196992 to 7248862714117087232. Current Snapshot: MvccSnapshot[applied={T|T < 7248862716586196992}]
I20260130 02:57:54.246470 19506 tablet_service.cc:1552] Processing DeleteTablet for tablet 79e28c3a566c4521ac2453bbaa3567ed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:57:53 UTC) from {username='slave'} at 127.0.0.1:59068
I20260130 02:57:54.230685 19624 tablet_bootstrap.cc:654] T f73338d4ce584880a9bec751d45ae753 P 371a36d219164974abdf848841528152: Neither blocks nor log segments found. Creating new log.
I20260130 02:57:54.247625 19358 tablet_service.cc:1552] Processing DeleteTablet for tablet 79e28c3a566c4521ac2453bbaa3567ed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:57:53 UTC) from {username='slave'} at 127.0.0.1:60110
I20260130 02:57:54.248320 19645 tablet_replica.cc:333] T 79e28c3a566c4521ac2453bbaa3567ed P 78c2c46c7520492db35d51c244fb4f74: stopping tablet replica
I20260130 02:57:54.249043 19645 raft_consensus.cc:2243] T 79e28c3a566c4521ac2453bbaa3567ed P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260130 02:57:54.249493 19645 raft_consensus.cc:2272] T 79e28c3a566c4521ac2453bbaa3567ed P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260130 02:57:54.250140 19205 tablet_service.cc:1552] Processing DeleteTablet for tablet 79e28c3a566c4521ac2453bbaa3567ed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:57:53 UTC) from {username='slave'} at 127.0.0.1:42870
I20260130 02:57:54.253014 19645 ts_tablet_manager.cc:1916] T 79e28c3a566c4521ac2453bbaa3567ed P 78c2c46c7520492db35d51c244fb4f74: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260130 02:57:54.253288 19674 tablet.cc:1722] T 96584ca174b140f7ad1a2ca698baca71 P 78c2c46c7520492db35d51c244fb4f74: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
W20260130 02:57:54.255112 18861 catalog_manager.cc:4965] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021): delete failed for tablet 79e28c3a566c4521ac2453bbaa3567ed because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 79e28c3a566c4521ac2453bbaa3567ed already in progress: creating tablet
I20260130 02:57:54.260601 19506 tablet_service.cc:1505] Processing CreateTablet for tablet 38a7a10e36874183bf9fe55a38d829f6 (DEFAULT_TABLE table=default.Table_1a735ba1c9994f81ab61385e277b300c [id=27adf356fd004597a54c7a9887dd3ca8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:57:54.261754 19506 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 38a7a10e36874183bf9fe55a38d829f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:57:54.261953 19205 tablet_service.cc:1505] Processing CreateTablet for tablet c2a9675427754d0db1578637a37440c6 (DEFAULT_TABLE table=default.Table_69ffd00165c6422292d9976b685d5d0d [id=f4829d67e1394959b7a7335fd067a826]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:57:54.263284 19205 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c2a9675427754d0db1578637a37440c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:57:54.263729 19358 tablet_service.cc:1505] Processing CreateTablet for tablet c2a9675427754d0db1578637a37440c6 (DEFAULT_TABLE table=default.Table_69ffd00165c6422292d9976b685d5d0d [id=f4829d67e1394959b7a7335fd067a826]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:57:54.264737 19358 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c2a9675427754d0db1578637a37440c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:57:54.275487 18860 catalog_manager.cc:4985] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): tablet 00e6ed81b43d4e74aa6d648fdc1f6c65 (table default.Table_b2b8ae6f8ee74fc2be70b4cd5d16782e [id=18cadca4c17f42d4bcbb7db8c12045bc]) successfully deleted
W20260130 02:57:54.276316 18860 catalog_manager.cc:4965] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): delete failed for tablet 79e28c3a566c4521ac2453bbaa3567ed because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 79e28c3a566c4521ac2453bbaa3567ed already in progress: creating tablet
I20260130 02:57:54.276818 19675 tablet.cc:1717] T 96584ca174b140f7ad1a2ca698baca71 P d870164263bb44b6b780a557daea3656: Skipping requested alter to schema version 3, tablet already version 3
I20260130 02:57:54.278002 19360 tablet_service.cc:1505] Processing CreateTablet for tablet b078934be4bb4fd1915c691eeea9d00f (DEFAULT_TABLE table=default.Table_38b6b66b6a5d41a297953249f24045ce [id=720decfb55c24f34983ef636f06e3169]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:57:54.279126 19360 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b078934be4bb4fd1915c691eeea9d00f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:57:54.282614 19507 tablet_service.cc:1505] Processing CreateTablet for tablet c2a9675427754d0db1578637a37440c6 (DEFAULT_TABLE table=default.Table_69ffd00165c6422292d9976b685d5d0d [id=f4829d67e1394959b7a7335fd067a826]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:57:54.283679 19507 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c2a9675427754d0db1578637a37440c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:57:54.284955 19625 tablet_bootstrap.cc:654] T f73338d4ce584880a9bec751d45ae753 P d870164263bb44b6b780a557daea3656: Neither blocks nor log segments found. Creating new log.
I20260130 02:57:54.285355 19663 raft_consensus.cc:493] T f73338d4ce584880a9bec751d45ae753 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:57:54.285879 19508 tablet_service.cc:1505] Processing CreateTablet for tablet b078934be4bb4fd1915c691eeea9d00f (DEFAULT_TABLE table=default.Table_38b6b66b6a5d41a297953249f24045ce [id=720decfb55c24f34983ef636f06e3169]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:57:54.285787 19663 raft_consensus.cc:515] T f73338d4ce584880a9bec751d45ae753 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } }
I20260130 02:57:54.286926 19508 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b078934be4bb4fd1915c691eeea9d00f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:57:54.287606 19663 leader_election.cc:290] T f73338d4ce584880a9bec751d45ae753 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d870164263bb44b6b780a557daea3656 (127.18.100.67:39571), 371a36d219164974abdf848841528152 (127.18.100.65:42021)
I20260130 02:57:54.288920 19225 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f73338d4ce584880a9bec751d45ae753" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152" is_pre_election: true
I20260130 02:57:54.290333 19626 tablet_bootstrap.cc:492] T 38a7a10e36874183bf9fe55a38d829f6 P 78c2c46c7520492db35d51c244fb4f74: Bootstrap starting.
I20260130 02:57:54.290866 19675 tablet.cc:1722] T 96584ca174b140f7ad1a2ca698baca71 P d870164263bb44b6b780a557daea3656: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
W20260130 02:57:54.295418 19314 leader_election.cc:343] T f73338d4ce584880a9bec751d45ae753 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:57:54.297865 19626 tablet_bootstrap.cc:654] T 38a7a10e36874183bf9fe55a38d829f6 P 78c2c46c7520492db35d51c244fb4f74: Neither blocks nor log segments found. Creating new log.
I20260130 02:57:54.306491 19205 tablet_service.cc:1505] Processing CreateTablet for tablet b078934be4bb4fd1915c691eeea9d00f (DEFAULT_TABLE table=default.Table_38b6b66b6a5d41a297953249f24045ce [id=720decfb55c24f34983ef636f06e3169]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:57:54.308965 19673 tablet.cc:1717] T 96584ca174b140f7ad1a2ca698baca71 P 371a36d219164974abdf848841528152: Skipping requested alter to schema version 4, tablet already version 4
I20260130 02:57:54.309058 19204 tablet_service.cc:1505] Processing CreateTablet for tablet a46702bef0e6442d85e4ca49a5f7cfbb (DEFAULT_TABLE table=default.Table_97c3def679ff49dcb74a992828c65cde [id=59eb236fbb134702a61a33d9b15c046c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:57:54.310364 19205 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b078934be4bb4fd1915c691eeea9d00f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:57:54.312269 19204 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a46702bef0e6442d85e4ca49a5f7cfbb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:57:54.320067 19645 ts_tablet_manager.cc:1929] T 79e28c3a566c4521ac2453bbaa3567ed P 78c2c46c7520492db35d51c244fb4f74: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260130 02:57:54.320470 19645 log.cc:1199] T 79e28c3a566c4521ac2453bbaa3567ed P 78c2c46c7520492db35d51c244fb4f74: Deleting WAL directory at /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-1/wal/wals/79e28c3a566c4521ac2453bbaa3567ed
I20260130 02:57:54.321130 19645 ts_tablet_manager.cc:1950] T 79e28c3a566c4521ac2453bbaa3567ed P 78c2c46c7520492db35d51c244fb4f74: Deleting consensus metadata
I20260130 02:57:54.325107 19505 tablet_service.cc:1505] Processing CreateTablet for tablet a46702bef0e6442d85e4ca49a5f7cfbb (DEFAULT_TABLE table=default.Table_97c3def679ff49dcb74a992828c65cde [id=59eb236fbb134702a61a33d9b15c046c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:57:54.326295 19505 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a46702bef0e6442d85e4ca49a5f7cfbb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:57:54.330159 18869 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50284:
name: "default.Table_e67bd1859b5846f386097cf33c5f1bcb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260130 02:57:54.331959 18869 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e67bd1859b5846f386097cf33c5f1bcb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260130 02:57:54.333443 19674 tablet.cc:1717] T 96584ca174b140f7ad1a2ca698baca71 P 78c2c46c7520492db35d51c244fb4f74: Skipping requested alter to schema version 4, tablet already version 4
I20260130 02:57:54.335409 19357 tablet_service.cc:1505] Processing CreateTablet for tablet a46702bef0e6442d85e4ca49a5f7cfbb (DEFAULT_TABLE table=default.Table_97c3def679ff49dcb74a992828c65cde [id=59eb236fbb134702a61a33d9b15c046c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:57:54.336433 19357 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a46702bef0e6442d85e4ca49a5f7cfbb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:57:54.346781 19679 raft_consensus.cc:493] T 927c90b84d7449efa36bc5e2035bf0da P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:57:54.347254 19679 raft_consensus.cc:515] T 927c90b84d7449efa36bc5e2035bf0da P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } }
I20260130 02:57:54.348225 19527 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f73338d4ce584880a9bec751d45ae753" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
W20260130 02:57:54.349375 19313 leader_election.cc:343] T f73338d4ce584880a9bec751d45ae753 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:57:54.349792 19313 leader_election.cc:304] T f73338d4ce584880a9bec751d45ae753 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78c2c46c7520492db35d51c244fb4f74; no voters: 371a36d219164974abdf848841528152, d870164263bb44b6b780a557daea3656
I20260130 02:57:54.350891 19663 raft_consensus.cc:2749] T f73338d4ce584880a9bec751d45ae753 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260130 02:57:54.351579 19679 leader_election.cc:290] T 927c90b84d7449efa36bc5e2035bf0da P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767), d870164263bb44b6b780a557daea3656 (127.18.100.67:39571)
I20260130 02:57:54.355785 19380 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "927c90b84d7449efa36bc5e2035bf0da" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74" is_pre_election: true
I20260130 02:57:54.355955 18867 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50284:
table { table_name: "default.Table_38b6b66b6a5d41a297953249f24045ce" } new_table_name: "default.Table_fcce594e0ed443158293215c25e4e5c3" modify_external_catalogs: true
I20260130 02:57:54.356406 19380 raft_consensus.cc:2468] T 927c90b84d7449efa36bc5e2035bf0da P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 371a36d219164974abdf848841528152 in term 0.
I20260130 02:57:54.358161 19159 leader_election.cc:304] T 927c90b84d7449efa36bc5e2035bf0da P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 371a36d219164974abdf848841528152, 78c2c46c7520492db35d51c244fb4f74; no voters:
I20260130 02:57:54.358646 19527 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "927c90b84d7449efa36bc5e2035bf0da" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
I20260130 02:57:54.359361 18875 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50284:
table { table_name: "default.Table_39345d64a2bc4f46b0136c6c4155ea8b" } new_table_name: "default.Table_4f2d6e4d9c8e465087d65efc9c2de442" modify_external_catalogs: true
I20260130 02:57:54.372493 18873 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50284:
name: "default.Table_a8b2bef1c4564242b8d5fc95a3a35f85"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260130 02:57:54.374187 18873 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a8b2bef1c4564242b8d5fc95a3a35f85 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260130 02:57:54.379070 19635 raft_consensus.cc:2804] T 927c90b84d7449efa36bc5e2035bf0da P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260130 02:57:54.379415 19635 raft_consensus.cc:493] T 927c90b84d7449efa36bc5e2035bf0da P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260130 02:57:54.379761 19635 raft_consensus.cc:3060] T 927c90b84d7449efa36bc5e2035bf0da P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:57:54.385383 18872 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50284:
name: "default.Table_49d7b6a1e24c433eb9103003ed2dfc8b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260130 02:57:54.385982 19159 leader_election.cc:343] T 927c90b84d7449efa36bc5e2035bf0da P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): Illegal state: must be running to vote when last-logged opid is not known
W20260130 02:57:54.387118 18872 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_49d7b6a1e24c433eb9103003ed2dfc8b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260130 02:57:54.391686 18866 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50284:
name: "default.Table_4a74769a488b4760b12165bbffa7cac8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260130 02:57:54.392418 19625 tablet_bootstrap.cc:492] T f73338d4ce584880a9bec751d45ae753 P d870164263bb44b6b780a557daea3656: No bootstrap required, opened a new log
I20260130 02:57:54.392797 19625 ts_tablet_manager.cc:1403] T f73338d4ce584880a9bec751d45ae753 P d870164263bb44b6b780a557daea3656: Time spent bootstrapping tablet: real 0.192s user 0.011s sys 0.009s
W20260130 02:57:54.393437 18866 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4a74769a488b4760b12165bbffa7cac8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260130 02:57:54.394956 19625 raft_consensus.cc:359] T f73338d4ce584880a9bec751d45ae753 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } }
I20260130 02:57:54.395562 19625 raft_consensus.cc:385] T f73338d4ce584880a9bec751d45ae753 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:57:54.395807 19625 raft_consensus.cc:740] T f73338d4ce584880a9bec751d45ae753 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d870164263bb44b6b780a557daea3656, State: Initialized, Role: FOLLOWER
I20260130 02:57:54.396449 19625 consensus_queue.cc:260] T f73338d4ce584880a9bec751d45ae753 P d870164263bb44b6b780a557daea3656 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } }
I20260130 02:57:54.398336 19625 ts_tablet_manager.cc:1434] T f73338d4ce584880a9bec751d45ae753 P d870164263bb44b6b780a557daea3656: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260130 02:57:54.399121 19625 tablet_bootstrap.cc:492] T 927c90b84d7449efa36bc5e2035bf0da P d870164263bb44b6b780a557daea3656: Bootstrap starting.
I20260130 02:57:54.401309 19624 tablet_bootstrap.cc:492] T f73338d4ce584880a9bec751d45ae753 P 371a36d219164974abdf848841528152: No bootstrap required, opened a new log
I20260130 02:57:54.401723 19624 ts_tablet_manager.cc:1403] T f73338d4ce584880a9bec751d45ae753 P 371a36d219164974abdf848841528152: Time spent bootstrapping tablet: real 0.253s user 0.004s sys 0.009s
I20260130 02:57:54.402194 18860 catalog_manager.cc:4985] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767): tablet 79e28c3a566c4521ac2453bbaa3567ed (table default.Table_b8f4e9cf27f44a479e1adad0b7ce4e33 [id=26422816af0c430880cb44e1b0cce534]) successfully deleted
I20260130 02:57:54.404081 19625 tablet_bootstrap.cc:654] T 927c90b84d7449efa36bc5e2035bf0da P d870164263bb44b6b780a557daea3656: Neither blocks nor log segments found. Creating new log.
I20260130 02:57:54.404376 19624 raft_consensus.cc:359] T f73338d4ce584880a9bec751d45ae753 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } }
I20260130 02:57:54.404992 19624 raft_consensus.cc:385] T f73338d4ce584880a9bec751d45ae753 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:57:54.405225 19624 raft_consensus.cc:740] T f73338d4ce584880a9bec751d45ae753 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 371a36d219164974abdf848841528152, State: Initialized, Role: FOLLOWER
I20260130 02:57:54.406364 19675 tablet.cc:1717] T 96584ca174b140f7ad1a2ca698baca71 P d870164263bb44b6b780a557daea3656: Skipping requested alter to schema version 4, tablet already version 4
I20260130 02:57:54.414834 19635 raft_consensus.cc:515] T 927c90b84d7449efa36bc5e2035bf0da P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } }
I20260130 02:57:54.415351 19625 tablet_bootstrap.cc:492] T 927c90b84d7449efa36bc5e2035bf0da P d870164263bb44b6b780a557daea3656: No bootstrap required, opened a new log
I20260130 02:57:54.415799 19625 ts_tablet_manager.cc:1403] T 927c90b84d7449efa36bc5e2035bf0da P d870164263bb44b6b780a557daea3656: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.005s
I20260130 02:57:54.416679 19635 leader_election.cc:290] T 927c90b84d7449efa36bc5e2035bf0da P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 election: Requested vote from peers 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767), d870164263bb44b6b780a557daea3656 (127.18.100.67:39571)
I20260130 02:57:54.418108 19380 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "927c90b84d7449efa36bc5e2035bf0da" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74"
I20260130 02:57:54.418256 19625 raft_consensus.cc:359] T 927c90b84d7449efa36bc5e2035bf0da P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } }
I20260130 02:57:54.418675 19380 raft_consensus.cc:3060] T 927c90b84d7449efa36bc5e2035bf0da P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:57:54.418906 19625 raft_consensus.cc:385] T 927c90b84d7449efa36bc5e2035bf0da P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:57:54.419128 19625 raft_consensus.cc:740] T 927c90b84d7449efa36bc5e2035bf0da P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d870164263bb44b6b780a557daea3656, State: Initialized, Role: FOLLOWER
I20260130 02:57:54.425493 19625 consensus_queue.cc:260] T 927c90b84d7449efa36bc5e2035bf0da P d870164263bb44b6b780a557daea3656 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } }
I20260130 02:57:54.426198 19380 raft_consensus.cc:2468] T 927c90b84d7449efa36bc5e2035bf0da P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 371a36d219164974abdf848841528152 in term 1.
I20260130 02:57:54.428200 19527 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "927c90b84d7449efa36bc5e2035bf0da" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656"
I20260130 02:57:54.428740 19527 raft_consensus.cc:3060] T 927c90b84d7449efa36bc5e2035bf0da P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:57:54.428247 19159 leader_election.cc:304] T 927c90b84d7449efa36bc5e2035bf0da P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 371a36d219164974abdf848841528152, 78c2c46c7520492db35d51c244fb4f74; no voters:
I20260130 02:57:54.430831 19624 consensus_queue.cc:260] T f73338d4ce584880a9bec751d45ae753 P 371a36d219164974abdf848841528152 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } }
I20260130 02:57:54.434973 19679 raft_consensus.cc:2804] T 927c90b84d7449efa36bc5e2035bf0da P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Leader election won for term 1
I20260130 02:57:54.434927 19527 raft_consensus.cc:2468] T 927c90b84d7449efa36bc5e2035bf0da P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 371a36d219164974abdf848841528152 in term 1.
I20260130 02:57:54.435398 19679 raft_consensus.cc:697] T 927c90b84d7449efa36bc5e2035bf0da P 371a36d219164974abdf848841528152 [term 1 LEADER]: Becoming Leader. State: Replica: 371a36d219164974abdf848841528152, State: Running, Role: LEADER
I20260130 02:57:54.436141 19679 consensus_queue.cc:237] T 927c90b84d7449efa36bc5e2035bf0da P 371a36d219164974abdf848841528152 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } }
I20260130 02:57:54.437929 19625 ts_tablet_manager.cc:1434] T 927c90b84d7449efa36bc5e2035bf0da P d870164263bb44b6b780a557daea3656: Time spent starting tablet: real 0.022s user 0.005s sys 0.001s
I20260130 02:57:54.438891 19625 tablet_bootstrap.cc:492] T 79e28c3a566c4521ac2453bbaa3567ed P d870164263bb44b6b780a557daea3656: Bootstrap starting.
I20260130 02:57:54.444592 19624 ts_tablet_manager.cc:1434] T f73338d4ce584880a9bec751d45ae753 P 371a36d219164974abdf848841528152: Time spent starting tablet: real 0.043s user 0.002s sys 0.003s
I20260130 02:57:54.452642 19625 tablet_bootstrap.cc:654] T 79e28c3a566c4521ac2453bbaa3567ed P d870164263bb44b6b780a557daea3656: Neither blocks nor log segments found. Creating new log.
I20260130 02:57:54.453220 19624 tablet_bootstrap.cc:492] T 79e28c3a566c4521ac2453bbaa3567ed P 371a36d219164974abdf848841528152: Bootstrap starting.
I20260130 02:57:54.455308 19677 raft_consensus.cc:493] T f73338d4ce584880a9bec751d45ae753 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:57:54.455807 19677 raft_consensus.cc:515] T f73338d4ce584880a9bec751d45ae753 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } }
I20260130 02:57:54.457828 19677 leader_election.cc:290] T f73338d4ce584880a9bec751d45ae753 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767), 371a36d219164974abdf848841528152 (127.18.100.65:42021)
I20260130 02:57:54.459347 19225 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f73338d4ce584880a9bec751d45ae753" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152" is_pre_election: true
I20260130 02:57:54.459952 19225 raft_consensus.cc:2468] T f73338d4ce584880a9bec751d45ae753 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d870164263bb44b6b780a557daea3656 in term 0.
I20260130 02:57:54.461405 19463 leader_election.cc:304] T f73338d4ce584880a9bec751d45ae753 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 371a36d219164974abdf848841528152, d870164263bb44b6b780a557daea3656; no voters:
I20260130 02:57:54.462143 19677 raft_consensus.cc:2804] T f73338d4ce584880a9bec751d45ae753 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260130 02:57:54.462487 19677 raft_consensus.cc:493] T f73338d4ce584880a9bec751d45ae753 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260130 02:57:54.463176 19677 raft_consensus.cc:3060] T f73338d4ce584880a9bec751d45ae753 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:57:54.469959 19677 raft_consensus.cc:515] T f73338d4ce584880a9bec751d45ae753 P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } }
I20260130 02:57:54.474010 19225 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f73338d4ce584880a9bec751d45ae753" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152"
I20260130 02:57:54.474746 19624 tablet_bootstrap.cc:654] T 79e28c3a566c4521ac2453bbaa3567ed P 371a36d219164974abdf848841528152: Neither blocks nor log segments found. Creating new log.
I20260130 02:57:54.474658 19225 raft_consensus.cc:3060] T f73338d4ce584880a9bec751d45ae753 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:57:54.475333 19677 leader_election.cc:290] T f73338d4ce584880a9bec751d45ae753 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 election: Requested vote from peers 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767), 371a36d219164974abdf848841528152 (127.18.100.65:42021)
I20260130 02:57:54.481320 19225 raft_consensus.cc:2468] T f73338d4ce584880a9bec751d45ae753 P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d870164263bb44b6b780a557daea3656 in term 1.
I20260130 02:57:54.482455 19463 leader_election.cc:304] T f73338d4ce584880a9bec751d45ae753 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 371a36d219164974abdf848841528152, d870164263bb44b6b780a557daea3656; no voters:
I20260130 02:57:54.483336 19677 raft_consensus.cc:2804] T f73338d4ce584880a9bec751d45ae753 P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Leader election won for term 1
I20260130 02:57:54.483784 19677 raft_consensus.cc:697] T f73338d4ce584880a9bec751d45ae753 P d870164263bb44b6b780a557daea3656 [term 1 LEADER]: Becoming Leader. State: Replica: d870164263bb44b6b780a557daea3656, State: Running, Role: LEADER
I20260130 02:57:54.484512 19677 consensus_queue.cc:237] T f73338d4ce584880a9bec751d45ae753 P d870164263bb44b6b780a557daea3656 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } }
I20260130 02:57:54.496198 19379 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f73338d4ce584880a9bec751d45ae753" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74"
I20260130 02:57:54.496755 19379 raft_consensus.cc:3060] T f73338d4ce584880a9bec751d45ae753 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:57:54.498310 19380 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f73338d4ce584880a9bec751d45ae753" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74" is_pre_election: true
I20260130 02:57:54.505203 19625 tablet_bootstrap.cc:492] T 79e28c3a566c4521ac2453bbaa3567ed P d870164263bb44b6b780a557daea3656: No bootstrap required, opened a new log
I20260130 02:57:54.505759 19625 ts_tablet_manager.cc:1403] T 79e28c3a566c4521ac2453bbaa3567ed P d870164263bb44b6b780a557daea3656: Time spent bootstrapping tablet: real 0.067s user 0.010s sys 0.004s
I20260130 02:57:54.508234 19625 raft_consensus.cc:359] T 79e28c3a566c4521ac2453bbaa3567ed P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } }
I20260130 02:57:54.512889 19625 raft_consensus.cc:385] T 79e28c3a566c4521ac2453bbaa3567ed P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:57:54.513530 19625 raft_consensus.cc:740] T 79e28c3a566c4521ac2453bbaa3567ed P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d870164263bb44b6b780a557daea3656, State: Initialized, Role: FOLLOWER
I20260130 02:57:54.514843 19625 consensus_queue.cc:260] T 79e28c3a566c4521ac2453bbaa3567ed P d870164263bb44b6b780a557daea3656 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } }
W20260130 02:57:54.523108 18867 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b078934be4bb4fd1915c691eeea9d00f consensus state has no leader
I20260130 02:57:54.523698 19625 ts_tablet_manager.cc:1434] T 79e28c3a566c4521ac2453bbaa3567ed P d870164263bb44b6b780a557daea3656: Time spent starting tablet: real 0.017s user 0.002s sys 0.003s
I20260130 02:57:54.530396 19625 tablet_bootstrap.cc:492] T a46702bef0e6442d85e4ca49a5f7cfbb P d870164263bb44b6b780a557daea3656: Bootstrap starting.
I20260130 02:57:54.533355 18867 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50284:
table { table_name: "default.Table_9428a7751159470793188f91cdcba4f3" } modify_external_catalogs: true
I20260130 02:57:54.533908 18867 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50284:
table { table_name: "default.Table_9428a7751159470793188f91cdcba4f3" } modify_external_catalogs: true
I20260130 02:57:54.542382 19625 tablet_bootstrap.cc:654] T a46702bef0e6442d85e4ca49a5f7cfbb P d870164263bb44b6b780a557daea3656: Neither blocks nor log segments found. Creating new log.
I20260130 02:57:54.546842 19626 tablet_bootstrap.cc:492] T 38a7a10e36874183bf9fe55a38d829f6 P 78c2c46c7520492db35d51c244fb4f74: No bootstrap required, opened a new log
I20260130 02:57:54.547334 19626 ts_tablet_manager.cc:1403] T 38a7a10e36874183bf9fe55a38d829f6 P 78c2c46c7520492db35d51c244fb4f74: Time spent bootstrapping tablet: real 0.257s user 0.018s sys 0.046s
I20260130 02:57:54.549860 19626 raft_consensus.cc:359] T 38a7a10e36874183bf9fe55a38d829f6 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:57:54.550624 19626 raft_consensus.cc:385] T 38a7a10e36874183bf9fe55a38d829f6 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:57:54.550907 19626 raft_consensus.cc:740] T 38a7a10e36874183bf9fe55a38d829f6 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78c2c46c7520492db35d51c244fb4f74, State: Initialized, Role: FOLLOWER
I20260130 02:57:54.551771 19626 consensus_queue.cc:260] T 38a7a10e36874183bf9fe55a38d829f6 P 78c2c46c7520492db35d51c244fb4f74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:57:54.553637 19626 ts_tablet_manager.cc:1434] T 38a7a10e36874183bf9fe55a38d829f6 P 78c2c46c7520492db35d51c244fb4f74: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260130 02:57:54.555053 19626 tablet_bootstrap.cc:492] T c2a9675427754d0db1578637a37440c6 P 78c2c46c7520492db35d51c244fb4f74: Bootstrap starting.
I20260130 02:57:54.560245 19626 tablet_bootstrap.cc:654] T c2a9675427754d0db1578637a37440c6 P 78c2c46c7520492db35d51c244fb4f74: Neither blocks nor log segments found. Creating new log.
I20260130 02:57:54.570125 19626 tablet_bootstrap.cc:492] T c2a9675427754d0db1578637a37440c6 P 78c2c46c7520492db35d51c244fb4f74: No bootstrap required, opened a new log
I20260130 02:57:54.570544 19626 ts_tablet_manager.cc:1403] T c2a9675427754d0db1578637a37440c6 P 78c2c46c7520492db35d51c244fb4f74: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.005s
I20260130 02:57:54.572260 19624 tablet_bootstrap.cc:492] T 79e28c3a566c4521ac2453bbaa3567ed P 371a36d219164974abdf848841528152: No bootstrap required, opened a new log
I20260130 02:57:54.572716 19624 ts_tablet_manager.cc:1403] T 79e28c3a566c4521ac2453bbaa3567ed P 371a36d219164974abdf848841528152: Time spent bootstrapping tablet: real 0.124s user 0.021s sys 0.006s
I20260130 02:57:54.572626 19626 raft_consensus.cc:359] T c2a9675427754d0db1578637a37440c6 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:57:54.573170 19626 raft_consensus.cc:385] T c2a9675427754d0db1578637a37440c6 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:57:54.575102 19624 raft_consensus.cc:359] T 79e28c3a566c4521ac2453bbaa3567ed P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } }
I20260130 02:57:54.575757 19624 raft_consensus.cc:385] T 79e28c3a566c4521ac2453bbaa3567ed P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:57:54.576058 19624 raft_consensus.cc:740] T 79e28c3a566c4521ac2453bbaa3567ed P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 371a36d219164974abdf848841528152, State: Initialized, Role: FOLLOWER
I20260130 02:57:54.576814 19624 consensus_queue.cc:260] T 79e28c3a566c4521ac2453bbaa3567ed P 371a36d219164974abdf848841528152 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } }
I20260130 02:57:54.577565 19626 raft_consensus.cc:740] T c2a9675427754d0db1578637a37440c6 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78c2c46c7520492db35d51c244fb4f74, State: Initialized, Role: FOLLOWER
I20260130 02:57:54.578241 19626 consensus_queue.cc:260] T c2a9675427754d0db1578637a37440c6 P 78c2c46c7520492db35d51c244fb4f74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:57:54.579613 19379 raft_consensus.cc:2468] T f73338d4ce584880a9bec751d45ae753 P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d870164263bb44b6b780a557daea3656 in term 1.
I20260130 02:57:54.581831 19625 tablet_bootstrap.cc:492] T a46702bef0e6442d85e4ca49a5f7cfbb P d870164263bb44b6b780a557daea3656: No bootstrap required, opened a new log
I20260130 02:57:54.582201 19625 ts_tablet_manager.cc:1403] T a46702bef0e6442d85e4ca49a5f7cfbb P d870164263bb44b6b780a557daea3656: Time spent bootstrapping tablet: real 0.054s user 0.006s sys 0.007s
I20260130 02:57:54.580137 19626 ts_tablet_manager.cc:1434] T c2a9675427754d0db1578637a37440c6 P 78c2c46c7520492db35d51c244fb4f74: Time spent starting tablet: real 0.009s user 0.004s sys 0.000s
I20260130 02:57:54.584412 19625 raft_consensus.cc:359] T a46702bef0e6442d85e4ca49a5f7cfbb P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } }
I20260130 02:57:54.585001 19625 raft_consensus.cc:385] T a46702bef0e6442d85e4ca49a5f7cfbb P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:57:54.585237 19625 raft_consensus.cc:740] T a46702bef0e6442d85e4ca49a5f7cfbb P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d870164263bb44b6b780a557daea3656, State: Initialized, Role: FOLLOWER
I20260130 02:57:54.585906 19625 consensus_queue.cc:260] T a46702bef0e6442d85e4ca49a5f7cfbb P d870164263bb44b6b780a557daea3656 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } }
I20260130 02:57:54.590938 19625 ts_tablet_manager.cc:1434] T a46702bef0e6442d85e4ca49a5f7cfbb P d870164263bb44b6b780a557daea3656: Time spent starting tablet: real 0.008s user 0.005s sys 0.000s
I20260130 02:57:54.591447 19626 tablet_bootstrap.cc:492] T b078934be4bb4fd1915c691eeea9d00f P 78c2c46c7520492db35d51c244fb4f74: Bootstrap starting.
I20260130 02:57:54.598356 19625 tablet_bootstrap.cc:492] T c2a9675427754d0db1578637a37440c6 P d870164263bb44b6b780a557daea3656: Bootstrap starting.
I20260130 02:57:54.603794 19624 ts_tablet_manager.cc:1434] T 79e28c3a566c4521ac2453bbaa3567ed P 371a36d219164974abdf848841528152: Time spent starting tablet: real 0.031s user 0.004s sys 0.000s
I20260130 02:57:54.608278 19626 tablet_bootstrap.cc:654] T b078934be4bb4fd1915c691eeea9d00f P 78c2c46c7520492db35d51c244fb4f74: Neither blocks nor log segments found. Creating new log.
I20260130 02:57:54.622875 19624 tablet_bootstrap.cc:492] T c2a9675427754d0db1578637a37440c6 P 371a36d219164974abdf848841528152: Bootstrap starting.
I20260130 02:57:54.623082 19626 tablet_bootstrap.cc:492] T b078934be4bb4fd1915c691eeea9d00f P 78c2c46c7520492db35d51c244fb4f74: No bootstrap required, opened a new log
I20260130 02:57:54.623468 19626 ts_tablet_manager.cc:1403] T b078934be4bb4fd1915c691eeea9d00f P 78c2c46c7520492db35d51c244fb4f74: Time spent bootstrapping tablet: real 0.040s user 0.003s sys 0.008s
I20260130 02:57:54.626055 19626 raft_consensus.cc:359] T b078934be4bb4fd1915c691eeea9d00f P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } }
I20260130 02:57:54.626682 19626 raft_consensus.cc:385] T b078934be4bb4fd1915c691eeea9d00f P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:57:54.626943 19626 raft_consensus.cc:740] T b078934be4bb4fd1915c691eeea9d00f P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78c2c46c7520492db35d51c244fb4f74, State: Initialized, Role: FOLLOWER
I20260130 02:57:54.627055 19625 tablet_bootstrap.cc:654] T c2a9675427754d0db1578637a37440c6 P d870164263bb44b6b780a557daea3656: Neither blocks nor log segments found. Creating new log.
I20260130 02:57:54.627532 19626 consensus_queue.cc:260] T b078934be4bb4fd1915c691eeea9d00f P 78c2c46c7520492db35d51c244fb4f74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } }
I20260130 02:57:54.629763 19626 ts_tablet_manager.cc:1434] T b078934be4bb4fd1915c691eeea9d00f P 78c2c46c7520492db35d51c244fb4f74: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20260130 02:57:54.632638 18868 catalog_manager.cc:5746] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571) does not have the latest schema for tablet 96584ca174b140f7ad1a2ca698baca71 (table default.Table_39345d64a2bc4f46b0136c6c4155ea8b [id=1515db389f284e2f9ed1a3cd0061d0f1]). Expected version 4 got 3
I20260130 02:57:54.633267 18868 catalog_manager.cc:5746] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571) does not have the latest schema for tablet 06b31d020b864c418476e6dcc1e4eafb (table default.Table_e9d466c5c67e4df1a06a8eb665da9570 [id=b78b791f363d4c0f906e2a4b21dfacc4]). Expected version 2 got 0
I20260130 02:57:54.633932 18868 catalog_manager.cc:5654] T df2c8989d6c1480c97baa23db01141eb P d870164263bb44b6b780a557daea3656 reported cstate change: term changed from 0 to 1, leader changed from <none> to d870164263bb44b6b780a557daea3656 (127.18.100.67). New cstate: current_term: 1 leader_uuid: "d870164263bb44b6b780a557daea3656" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } health_report { overall_health: HEALTHY } } }
I20260130 02:57:54.635646 18868 catalog_manager.cc:5746] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571) does not have the latest schema for tablet df2c8989d6c1480c97baa23db01141eb (table default.Table_f03fffd6ca4c48cdaf9b143f1c753710 [id=747af828147149efb7b7b8d33c49286d]). Expected version 1 got 0
I20260130 02:57:54.643814 19624 tablet_bootstrap.cc:654] T c2a9675427754d0db1578637a37440c6 P 371a36d219164974abdf848841528152: Neither blocks nor log segments found. Creating new log.
I20260130 02:57:54.644013 19203 tablet_service.cc:1505] Processing CreateTablet for tablet f2d01bfdc9c4494386bd02bed655fe63 (DEFAULT_TABLE table=default.Table_23120efe2754472c9f41c54904af4081 [id=d128767e303145a885eec3c5f714a27f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:57:54.645279 19203 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f2d01bfdc9c4494386bd02bed655fe63. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:57:54.652359 19624 tablet_bootstrap.cc:492] T c2a9675427754d0db1578637a37440c6 P 371a36d219164974abdf848841528152: No bootstrap required, opened a new log
I20260130 02:57:54.652818 19624 ts_tablet_manager.cc:1403] T c2a9675427754d0db1578637a37440c6 P 371a36d219164974abdf848841528152: Time spent bootstrapping tablet: real 0.030s user 0.007s sys 0.002s
I20260130 02:57:54.655256 19624 raft_consensus.cc:359] T c2a9675427754d0db1578637a37440c6 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:57:54.655920 19624 raft_consensus.cc:385] T c2a9675427754d0db1578637a37440c6 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:57:54.656157 19624 raft_consensus.cc:740] T c2a9675427754d0db1578637a37440c6 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 371a36d219164974abdf848841528152, State: Initialized, Role: FOLLOWER
I20260130 02:57:54.657531 19624 consensus_queue.cc:260] T c2a9675427754d0db1578637a37440c6 P 371a36d219164974abdf848841528152 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:57:54.659451 19624 ts_tablet_manager.cc:1434] T c2a9675427754d0db1578637a37440c6 P 371a36d219164974abdf848841528152: Time spent starting tablet: real 0.006s user 0.006s sys 0.001s
I20260130 02:57:54.662384 19625 tablet_bootstrap.cc:492] T c2a9675427754d0db1578637a37440c6 P d870164263bb44b6b780a557daea3656: No bootstrap required, opened a new log
I20260130 02:57:54.663448 19624 tablet_bootstrap.cc:492] T a46702bef0e6442d85e4ca49a5f7cfbb P 371a36d219164974abdf848841528152: Bootstrap starting.
I20260130 02:57:54.662889 19625 ts_tablet_manager.cc:1403] T c2a9675427754d0db1578637a37440c6 P d870164263bb44b6b780a557daea3656: Time spent bootstrapping tablet: real 0.065s user 0.016s sys 0.015s
I20260130 02:57:54.664377 19677 raft_consensus.cc:493] T a46702bef0e6442d85e4ca49a5f7cfbb P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:57:54.664947 19677 raft_consensus.cc:515] T a46702bef0e6442d85e4ca49a5f7cfbb P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } }
I20260130 02:57:54.666752 19625 raft_consensus.cc:359] T c2a9675427754d0db1578637a37440c6 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:57:54.667377 19625 raft_consensus.cc:385] T c2a9675427754d0db1578637a37440c6 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:57:54.667619 19625 raft_consensus.cc:740] T c2a9675427754d0db1578637a37440c6 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d870164263bb44b6b780a557daea3656, State: Initialized, Role: FOLLOWER
I20260130 02:57:54.669313 19625 consensus_queue.cc:260] T c2a9675427754d0db1578637a37440c6 P d870164263bb44b6b780a557daea3656 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:57:54.668591 19677 leader_election.cc:290] T a46702bef0e6442d85e4ca49a5f7cfbb P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767), 371a36d219164974abdf848841528152 (127.18.100.65:42021)
I20260130 02:57:54.671591 19225 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a46702bef0e6442d85e4ca49a5f7cfbb" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152" is_pre_election: true
I20260130 02:57:54.673559 19625 ts_tablet_manager.cc:1434] T c2a9675427754d0db1578637a37440c6 P d870164263bb44b6b780a557daea3656: Time spent starting tablet: real 0.009s user 0.003s sys 0.002s
I20260130 02:57:54.674566 19625 tablet_bootstrap.cc:492] T 38a7a10e36874183bf9fe55a38d829f6 P d870164263bb44b6b780a557daea3656: Bootstrap starting.
I20260130 02:57:54.676083 19624 tablet_bootstrap.cc:654] T a46702bef0e6442d85e4ca49a5f7cfbb P 371a36d219164974abdf848841528152: Neither blocks nor log segments found. Creating new log.
W20260130 02:57:54.679563 19463 leader_election.cc:343] T a46702bef0e6442d85e4ca49a5f7cfbb P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:57:54.679904 19625 tablet_bootstrap.cc:654] T 38a7a10e36874183bf9fe55a38d829f6 P d870164263bb44b6b780a557daea3656: Neither blocks nor log segments found. Creating new log.
I20260130 02:57:54.686864 19624 tablet_bootstrap.cc:492] T a46702bef0e6442d85e4ca49a5f7cfbb P 371a36d219164974abdf848841528152: No bootstrap required, opened a new log
I20260130 02:57:54.687278 19624 ts_tablet_manager.cc:1403] T a46702bef0e6442d85e4ca49a5f7cfbb P 371a36d219164974abdf848841528152: Time spent bootstrapping tablet: real 0.024s user 0.006s sys 0.004s
I20260130 02:57:54.689541 19624 raft_consensus.cc:359] T a46702bef0e6442d85e4ca49a5f7cfbb P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } }
I20260130 02:57:54.690155 19624 raft_consensus.cc:385] T a46702bef0e6442d85e4ca49a5f7cfbb P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:57:54.691403 19625 tablet_bootstrap.cc:492] T 38a7a10e36874183bf9fe55a38d829f6 P d870164263bb44b6b780a557daea3656: No bootstrap required, opened a new log
W20260130 02:57:54.693032 18860 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.100.125:39711, user_credentials={real_user=slave}} blocked reactor thread for 76014.4us
I20260130 02:57:54.694057 19625 ts_tablet_manager.cc:1403] T 38a7a10e36874183bf9fe55a38d829f6 P d870164263bb44b6b780a557daea3656: Time spent bootstrapping tablet: real 0.020s user 0.010s sys 0.004s
I20260130 02:57:54.690405 19624 raft_consensus.cc:740] T a46702bef0e6442d85e4ca49a5f7cfbb P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 371a36d219164974abdf848841528152, State: Initialized, Role: FOLLOWER
I20260130 02:57:54.695906 19506 tablet_service.cc:1505] Processing CreateTablet for tablet f2d01bfdc9c4494386bd02bed655fe63 (DEFAULT_TABLE table=default.Table_23120efe2754472c9f41c54904af4081 [id=d128767e303145a885eec3c5f714a27f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:57:54.696368 19625 raft_consensus.cc:359] T 38a7a10e36874183bf9fe55a38d829f6 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:57:54.696942 19506 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f2d01bfdc9c4494386bd02bed655fe63. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:57:54.697067 19624 consensus_queue.cc:260] T a46702bef0e6442d85e4ca49a5f7cfbb P 371a36d219164974abdf848841528152 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } }
I20260130 02:57:54.704674 19624 ts_tablet_manager.cc:1434] T a46702bef0e6442d85e4ca49a5f7cfbb P 371a36d219164974abdf848841528152: Time spent starting tablet: real 0.017s user 0.006s sys 0.000s
I20260130 02:57:54.705169 19379 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a46702bef0e6442d85e4ca49a5f7cfbb" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74" is_pre_election: true
I20260130 02:57:54.705756 19626 tablet_bootstrap.cc:492] T a46702bef0e6442d85e4ca49a5f7cfbb P 78c2c46c7520492db35d51c244fb4f74: Bootstrap starting.
W20260130 02:57:54.706785 19462 leader_election.cc:343] T a46702bef0e6442d85e4ca49a5f7cfbb P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:57:54.707294 19462 leader_election.cc:304] T a46702bef0e6442d85e4ca49a5f7cfbb P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d870164263bb44b6b780a557daea3656; no voters: 371a36d219164974abdf848841528152, 78c2c46c7520492db35d51c244fb4f74
I20260130 02:57:54.710785 19677 raft_consensus.cc:2749] T a46702bef0e6442d85e4ca49a5f7cfbb P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260130 02:57:54.711710 19624 tablet_bootstrap.cc:492] T b078934be4bb4fd1915c691eeea9d00f P 371a36d219164974abdf848841528152: Bootstrap starting.
I20260130 02:57:54.696933 19625 raft_consensus.cc:385] T 38a7a10e36874183bf9fe55a38d829f6 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:57:54.716168 19625 raft_consensus.cc:740] T 38a7a10e36874183bf9fe55a38d829f6 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d870164263bb44b6b780a557daea3656, State: Initialized, Role: FOLLOWER
I20260130 02:57:54.716756 19625 consensus_queue.cc:260] T 38a7a10e36874183bf9fe55a38d829f6 P d870164263bb44b6b780a557daea3656 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:57:54.717403 18867 catalog_manager.cc:5941] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4: Sending DeleteTablet for 3 replicas of tablet 3b8cdac4bf4b46efa26236fc930eb9a6
I20260130 02:57:54.721287 19205 tablet_service.cc:1552] Processing DeleteTablet for tablet 3b8cdac4bf4b46efa26236fc930eb9a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:57:54 UTC) from {username='slave'} at 127.0.0.1:42870
W20260130 02:57:54.723346 18861 catalog_manager.cc:4952] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021): delete failed for tablet 3b8cdac4bf4b46efa26236fc930eb9a6 because the tablet was not found. No further retry: Not found: Tablet not found: 3b8cdac4bf4b46efa26236fc930eb9a6
I20260130 02:57:54.724225 19625 ts_tablet_manager.cc:1434] T 38a7a10e36874183bf9fe55a38d829f6 P d870164263bb44b6b780a557daea3656: Time spent starting tablet: real 0.030s user 0.005s sys 0.000s
I20260130 02:57:54.724630 19624 tablet_bootstrap.cc:654] T b078934be4bb4fd1915c691eeea9d00f P 371a36d219164974abdf848841528152: Neither blocks nor log segments found. Creating new log.
I20260130 02:57:54.725136 19625 tablet_bootstrap.cc:492] T b078934be4bb4fd1915c691eeea9d00f P d870164263bb44b6b780a557daea3656: Bootstrap starting.
I20260130 02:57:54.730162 19625 tablet_bootstrap.cc:654] T b078934be4bb4fd1915c691eeea9d00f P d870164263bb44b6b780a557daea3656: Neither blocks nor log segments found. Creating new log.
I20260130 02:57:54.738370 18874 catalog_manager.cc:5746] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767) does not have the latest schema for tablet 96584ca174b140f7ad1a2ca698baca71 (table default.Table_39345d64a2bc4f46b0136c6c4155ea8b [id=1515db389f284e2f9ed1a3cd0061d0f1]). Expected version 4 got 3
I20260130 02:57:54.739297 18874 catalog_manager.cc:5746] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767) does not have the latest schema for tablet 06b31d020b864c418476e6dcc1e4eafb (table default.Table_e9d466c5c67e4df1a06a8eb665da9570 [id=b78b791f363d4c0f906e2a4b21dfacc4]). Expected version 2 got 0
I20260130 02:57:54.742091 18870 catalog_manager.cc:5941] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4: Sending DeleteTablet for 3 replicas of tablet f73338d4ce584880a9bec751d45ae753
I20260130 02:57:54.745635 19624 tablet_bootstrap.cc:492] T b078934be4bb4fd1915c691eeea9d00f P 371a36d219164974abdf848841528152: No bootstrap required, opened a new log
I20260130 02:57:54.746048 19624 ts_tablet_manager.cc:1403] T b078934be4bb4fd1915c691eeea9d00f P 371a36d219164974abdf848841528152: Time spent bootstrapping tablet: real 0.035s user 0.010s sys 0.000s
I20260130 02:57:54.746129 18871 catalog_manager.cc:5654] T 06b31d020b864c418476e6dcc1e4eafb P 371a36d219164974abdf848841528152 reported cstate change: term changed from 0 to 1, leader changed from <none> to 371a36d219164974abdf848841528152 (127.18.100.65). New cstate: current_term: 1 leader_uuid: "371a36d219164974abdf848841528152" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } health_report { overall_health: HEALTHY } } }
I20260130 02:57:54.747750 18871 catalog_manager.cc:5746] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021) does not have the latest schema for tablet 06b31d020b864c418476e6dcc1e4eafb (table default.Table_e9d466c5c67e4df1a06a8eb665da9570 [id=b78b791f363d4c0f906e2a4b21dfacc4]). Expected version 2 got 0
I20260130 02:57:54.748395 19624 raft_consensus.cc:359] T b078934be4bb4fd1915c691eeea9d00f P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } }
I20260130 02:57:54.749035 19624 raft_consensus.cc:385] T b078934be4bb4fd1915c691eeea9d00f P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:57:54.749275 19624 raft_consensus.cc:740] T b078934be4bb4fd1915c691eeea9d00f P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 371a36d219164974abdf848841528152, State: Initialized, Role: FOLLOWER
I20260130 02:57:54.750108 19624 consensus_queue.cc:260] T b078934be4bb4fd1915c691eeea9d00f P 371a36d219164974abdf848841528152 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } }
I20260130 02:57:54.752584 19624 ts_tablet_manager.cc:1434] T b078934be4bb4fd1915c691eeea9d00f P 371a36d219164974abdf848841528152: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260130 02:57:54.752629 19204 tablet_service.cc:1552] Processing DeleteTablet for tablet f73338d4ce584880a9bec751d45ae753 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:57:53 UTC) from {username='slave'} at 127.0.0.1:42870
I20260130 02:57:54.753448 19646 tablet_replica.cc:333] T f73338d4ce584880a9bec751d45ae753 P 371a36d219164974abdf848841528152: stopping tablet replica
I20260130 02:57:54.754283 19646 raft_consensus.cc:2243] T f73338d4ce584880a9bec751d45ae753 P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260130 02:57:54.754781 19646 raft_consensus.cc:2272] T f73338d4ce584880a9bec751d45ae753 P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260130 02:57:54.753433 19624 tablet_bootstrap.cc:492] T 38a7a10e36874183bf9fe55a38d829f6 P 371a36d219164974abdf848841528152: Bootstrap starting.
I20260130 02:57:54.757325 19646 ts_tablet_manager.cc:1916] T f73338d4ce584880a9bec751d45ae753 P 371a36d219164974abdf848841528152: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260130 02:57:54.761027 19624 tablet_bootstrap.cc:654] T 38a7a10e36874183bf9fe55a38d829f6 P 371a36d219164974abdf848841528152: Neither blocks nor log segments found. Creating new log.
I20260130 02:57:54.766026 19625 tablet_bootstrap.cc:492] T b078934be4bb4fd1915c691eeea9d00f P d870164263bb44b6b780a557daea3656: No bootstrap required, opened a new log
I20260130 02:57:54.766420 19625 ts_tablet_manager.cc:1403] T b078934be4bb4fd1915c691eeea9d00f P d870164263bb44b6b780a557daea3656: Time spent bootstrapping tablet: real 0.042s user 0.015s sys 0.010s
I20260130 02:57:54.769044 19625 raft_consensus.cc:359] T b078934be4bb4fd1915c691eeea9d00f P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } }
I20260130 02:57:54.769651 19625 raft_consensus.cc:385] T b078934be4bb4fd1915c691eeea9d00f P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:57:54.769904 19625 raft_consensus.cc:740] T b078934be4bb4fd1915c691eeea9d00f P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d870164263bb44b6b780a557daea3656, State: Initialized, Role: FOLLOWER
I20260130 02:57:54.770599 19625 consensus_queue.cc:260] T b078934be4bb4fd1915c691eeea9d00f P d870164263bb44b6b780a557daea3656 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } }
I20260130 02:57:54.776353 19625 ts_tablet_manager.cc:1434] T b078934be4bb4fd1915c691eeea9d00f P d870164263bb44b6b780a557daea3656: Time spent starting tablet: real 0.010s user 0.004s sys 0.000s
I20260130 02:57:54.777228 19625 tablet_bootstrap.cc:492] T f2d01bfdc9c4494386bd02bed655fe63 P d870164263bb44b6b780a557daea3656: Bootstrap starting.
I20260130 02:57:54.780092 19663 raft_consensus.cc:493] T b078934be4bb4fd1915c691eeea9d00f P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:57:54.780593 19663 raft_consensus.cc:515] T b078934be4bb4fd1915c691eeea9d00f P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } }
I20260130 02:57:54.782444 19663 leader_election.cc:290] T b078934be4bb4fd1915c691eeea9d00f P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d870164263bb44b6b780a557daea3656 (127.18.100.67:39571), 371a36d219164974abdf848841528152 (127.18.100.65:42021)
I20260130 02:57:54.784322 19527 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b078934be4bb4fd1915c691eeea9d00f" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
I20260130 02:57:54.784965 19527 raft_consensus.cc:2468] T b078934be4bb4fd1915c691eeea9d00f P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78c2c46c7520492db35d51c244fb4f74 in term 0.
I20260130 02:57:54.786113 19625 tablet_bootstrap.cc:654] T f2d01bfdc9c4494386bd02bed655fe63 P d870164263bb44b6b780a557daea3656: Neither blocks nor log segments found. Creating new log.
I20260130 02:57:54.786478 19313 leader_election.cc:304] T b078934be4bb4fd1915c691eeea9d00f P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78c2c46c7520492db35d51c244fb4f74, d870164263bb44b6b780a557daea3656; no voters:
I20260130 02:57:54.787447 19663 raft_consensus.cc:2804] T b078934be4bb4fd1915c691eeea9d00f P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260130 02:57:54.787808 19663 raft_consensus.cc:493] T b078934be4bb4fd1915c691eeea9d00f P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260130 02:57:54.788121 19663 raft_consensus.cc:3060] T b078934be4bb4fd1915c691eeea9d00f P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:57:54.791148 19225 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b078934be4bb4fd1915c691eeea9d00f" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152" is_pre_election: true
I20260130 02:57:54.791843 19225 raft_consensus.cc:2468] T b078934be4bb4fd1915c691eeea9d00f P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78c2c46c7520492db35d51c244fb4f74 in term 0.
W20260130 02:57:54.820559 18860 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.100.125:39711, user_credentials={real_user=slave}} blocked reactor thread for 112611us
I20260130 02:57:54.826107 19506 tablet_service.cc:1552] Processing DeleteTablet for tablet 3b8cdac4bf4b46efa26236fc930eb9a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:57:54 UTC) from {username='slave'} at 127.0.0.1:59068
I20260130 02:57:54.826900 19625 tablet_bootstrap.cc:492] T f2d01bfdc9c4494386bd02bed655fe63 P d870164263bb44b6b780a557daea3656: No bootstrap required, opened a new log
I20260130 02:57:54.827311 19625 ts_tablet_manager.cc:1403] T f2d01bfdc9c4494386bd02bed655fe63 P d870164263bb44b6b780a557daea3656: Time spent bootstrapping tablet: real 0.050s user 0.012s sys 0.017s
I20260130 02:57:54.829734 19625 raft_consensus.cc:359] T f2d01bfdc9c4494386bd02bed655fe63 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:57:54.832690 19357 tablet_service.cc:1552] Processing DeleteTablet for tablet 3b8cdac4bf4b46efa26236fc930eb9a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:57:54 UTC) from {username='slave'} at 127.0.0.1:60110
I20260130 02:57:54.836290 19204 tablet_service.cc:1505] Processing CreateTablet for tablet 93809949599a449ca09b13e9b8bbdef6 (DEFAULT_TABLE table=default.Table_1c77605822574b10a05074a093ad30a3 [id=aeba5f5a21564264a8cd975244774ab8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:57:54.838689 19625 raft_consensus.cc:385] T f2d01bfdc9c4494386bd02bed655fe63 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:57:54.838975 19625 raft_consensus.cc:740] T f2d01bfdc9c4494386bd02bed655fe63 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d870164263bb44b6b780a557daea3656, State: Initialized, Role: FOLLOWER
I20260130 02:57:54.839568 19625 consensus_queue.cc:260] T f2d01bfdc9c4494386bd02bed655fe63 P d870164263bb44b6b780a557daea3656 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:57:54.841779 19625 ts_tablet_manager.cc:1434] T f2d01bfdc9c4494386bd02bed655fe63 P d870164263bb44b6b780a557daea3656: Time spent starting tablet: real 0.014s user 0.002s sys 0.004s
I20260130 02:57:54.863000 19204 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 93809949599a449ca09b13e9b8bbdef6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:57:54.863039 19663 raft_consensus.cc:515] T b078934be4bb4fd1915c691eeea9d00f P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } }
I20260130 02:57:54.866086 19527 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b078934be4bb4fd1915c691eeea9d00f" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656"
I20260130 02:57:54.866639 19527 raft_consensus.cc:3060] T b078934be4bb4fd1915c691eeea9d00f P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:57:54.873260 19663 leader_election.cc:290] T b078934be4bb4fd1915c691eeea9d00f P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 election: Requested vote from peers d870164263bb44b6b780a557daea3656 (127.18.100.67:39571), 371a36d219164974abdf848841528152 (127.18.100.65:42021)
I20260130 02:57:54.874814 19225 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b078934be4bb4fd1915c691eeea9d00f" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152"
I20260130 02:57:54.875525 19225 raft_consensus.cc:3060] T b078934be4bb4fd1915c691eeea9d00f P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:57:54.878854 19506 tablet_service.cc:1552] Processing DeleteTablet for tablet f73338d4ce584880a9bec751d45ae753 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:57:53 UTC) from {username='slave'} at 127.0.0.1:59068
I20260130 02:57:54.879567 19710 tablet_replica.cc:333] T f73338d4ce584880a9bec751d45ae753 P d870164263bb44b6b780a557daea3656: stopping tablet replica
I20260130 02:57:54.880339 19710 raft_consensus.cc:2243] T f73338d4ce584880a9bec751d45ae753 P d870164263bb44b6b780a557daea3656 [term 1 LEADER]: Raft consensus shutting down.
I20260130 02:57:54.881027 19710 pending_rounds.cc:70] T f73338d4ce584880a9bec751d45ae753 P d870164263bb44b6b780a557daea3656: Trying to abort 1 pending ops.
I20260130 02:57:54.881256 19710 pending_rounds.cc:77] T f73338d4ce584880a9bec751d45ae753 P d870164263bb44b6b780a557daea3656: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7248862717896998912 op_type: NO_OP noop_request { }
I20260130 02:57:54.881623 19710 raft_consensus.cc:2889] T f73338d4ce584880a9bec751d45ae753 P d870164263bb44b6b780a557daea3656 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20260130 02:57:54.881944 19710 raft_consensus.cc:2272] T f73338d4ce584880a9bec751d45ae753 P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260130 02:57:54.887058 19710 ts_tablet_manager.cc:1916] T f73338d4ce584880a9bec751d45ae753 P d870164263bb44b6b780a557daea3656: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260130 02:57:54.898826 19357 tablet_service.cc:1552] Processing DeleteTablet for tablet f73338d4ce584880a9bec751d45ae753 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:57:53 UTC) from {username='slave'} at 127.0.0.1:60110
I20260130 02:57:54.899605 19711 tablet_replica.cc:333] T f73338d4ce584880a9bec751d45ae753 P 78c2c46c7520492db35d51c244fb4f74: stopping tablet replica
I20260130 02:57:54.900369 19711 raft_consensus.cc:2243] T f73338d4ce584880a9bec751d45ae753 P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260130 02:57:54.900826 19711 raft_consensus.cc:2272] T f73338d4ce584880a9bec751d45ae753 P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260130 02:57:54.902545 19711 ts_tablet_manager.cc:1916] T f73338d4ce584880a9bec751d45ae753 P 78c2c46c7520492db35d51c244fb4f74: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260130 02:57:54.912207 19689 tablet.cc:1722] T 96584ca174b140f7ad1a2ca698baca71 P d870164263bb44b6b780a557daea3656: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 5
I20260130 02:57:54.914736 19357 tablet_service.cc:1552] Processing DeleteTablet for tablet 79e28c3a566c4521ac2453bbaa3567ed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:57:53 UTC) from {username='slave'} at 127.0.0.1:60110
I20260130 02:57:54.916150 19712 tablet.cc:1722] T 96584ca174b140f7ad1a2ca698baca71 P 371a36d219164974abdf848841528152: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 5
I20260130 02:57:54.930665 19205 tablet_service.cc:1505] Processing CreateTablet for tablet 2806ae64d7324b918397c3c6b72ebcfe (DEFAULT_TABLE table=default.Table_d1b81d6a8e704777b0fa35cf23dc91e0 [id=78a1af9c84574331a9112fdea3f67fcf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:57:54.936378 18875 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50284:
name: "default.Table_50c3bb8b3ede494ba9b9f826a2adba72"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260130 02:57:54.936467 18866 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50284:
name: "default.Table_e7b6f8ca4ed143888d55e87c1b157efd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260130 02:57:54.936753 18860 catalog_manager.cc:4952] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): delete failed for tablet 3b8cdac4bf4b46efa26236fc930eb9a6 because the tablet was not found. No further retry: Not found: Tablet not found: 3b8cdac4bf4b46efa26236fc930eb9a6
W20260130 02:57:54.938124 18866 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e7b6f8ca4ed143888d55e87c1b157efd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260130 02:57:54.939155 18867 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50284:
name: "default.Table_235e240c644b42baaf12b08d1b983e8a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260130 02:57:54.939739 19357 tablet_service.cc:1505] Processing CreateTablet for tablet f2d01bfdc9c4494386bd02bed655fe63 (DEFAULT_TABLE table=default.Table_23120efe2754472c9f41c54904af4081 [id=d128767e303145a885eec3c5f714a27f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:57:54.939719 19506 tablet_service.cc:1505] Processing CreateTablet for tablet 93809949599a449ca09b13e9b8bbdef6 (DEFAULT_TABLE table=default.Table_1c77605822574b10a05074a093ad30a3 [id=aeba5f5a21564264a8cd975244774ab8]), partition=RANGE (key) PARTITION UNBOUNDED
W20260130 02:57:54.940752 18867 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_235e240c644b42baaf12b08d1b983e8a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260130 02:57:54.940974 18872 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50284:
name: "default.Table_ab500271b22e4a789171a24c55b79fce"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260130 02:57:54.942512 18872 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ab500271b22e4a789171a24c55b79fce in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260130 02:57:54.944274 19507 tablet_service.cc:1505] Processing CreateTablet for tablet 2806ae64d7324b918397c3c6b72ebcfe (DEFAULT_TABLE table=default.Table_d1b81d6a8e704777b0fa35cf23dc91e0 [id=78a1af9c84574331a9112fdea3f67fcf]), partition=RANGE (key) PARTITION UNBOUNDED
W20260130 02:57:54.945220 18875 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_50c3bb8b3ede494ba9b9f826a2adba72 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260130 02:57:54.945806 19360 tablet_service.cc:1505] Processing CreateTablet for tablet 93809949599a449ca09b13e9b8bbdef6 (DEFAULT_TABLE table=default.Table_1c77605822574b10a05074a093ad30a3 [id=aeba5f5a21564264a8cd975244774ab8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:57:54.947978 19626 tablet_bootstrap.cc:654] T a46702bef0e6442d85e4ca49a5f7cfbb P 78c2c46c7520492db35d51c244fb4f74: Neither blocks nor log segments found. Creating new log.
I20260130 02:57:54.950027 18868 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50284:
table { table_name: "default.Table_49d7b6a1e24c433eb9103003ed2dfc8b" } new_table_name: "default.Table_007e969f70d345a7bcb3dfbaabbc025d" modify_external_catalogs: true
I20260130 02:57:54.958777 19202 tablet_service.cc:1505] Processing CreateTablet for tablet 3b8cdac4bf4b46efa26236fc930eb9a6 (DEFAULT_TABLE table=default.Table_9428a7751159470793188f91cdcba4f3 [id=eaed0f6841e94faa858449fbb74bd5c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:57:54.962958 19624 tablet_bootstrap.cc:492] T 38a7a10e36874183bf9fe55a38d829f6 P 371a36d219164974abdf848841528152: No bootstrap required, opened a new log
I20260130 02:57:54.963511 19624 ts_tablet_manager.cc:1403] T 38a7a10e36874183bf9fe55a38d829f6 P 371a36d219164974abdf848841528152: Time spent bootstrapping tablet: real 0.210s user 0.043s sys 0.072s
I20260130 02:57:54.963878 19527 raft_consensus.cc:2468] T b078934be4bb4fd1915c691eeea9d00f P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78c2c46c7520492db35d51c244fb4f74 in term 1.
I20260130 02:57:54.965052 19646 ts_tablet_manager.cc:1929] T f73338d4ce584880a9bec751d45ae753 P 371a36d219164974abdf848841528152: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260130 02:57:54.965422 19646 log.cc:1199] T f73338d4ce584880a9bec751d45ae753 P 371a36d219164974abdf848841528152: Deleting WAL directory at /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-0/wal/wals/f73338d4ce584880a9bec751d45ae753
I20260130 02:57:54.966054 19646 ts_tablet_manager.cc:1950] T f73338d4ce584880a9bec751d45ae753 P 371a36d219164974abdf848841528152: Deleting consensus metadata
I20260130 02:57:54.967823 19357 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f2d01bfdc9c4494386bd02bed655fe63. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:57:54.969115 19202 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3b8cdac4bf4b46efa26236fc930eb9a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:57:54.969103 19313 leader_election.cc:304] T b078934be4bb4fd1915c691eeea9d00f P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78c2c46c7520492db35d51c244fb4f74, d870164263bb44b6b780a557daea3656; no voters:
I20260130 02:57:54.969794 19205 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2806ae64d7324b918397c3c6b72ebcfe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:57:54.970331 19360 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 93809949599a449ca09b13e9b8bbdef6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:57:54.970898 19663 raft_consensus.cc:2804] T b078934be4bb4fd1915c691eeea9d00f P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Leader election won for term 1
I20260130 02:57:54.972204 19225 raft_consensus.cc:2468] T b078934be4bb4fd1915c691eeea9d00f P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78c2c46c7520492db35d51c244fb4f74 in term 1.
I20260130 02:57:54.975762 19663 raft_consensus.cc:697] T b078934be4bb4fd1915c691eeea9d00f P 78c2c46c7520492db35d51c244fb4f74 [term 1 LEADER]: Becoming Leader. State: Replica: 78c2c46c7520492db35d51c244fb4f74, State: Running, Role: LEADER
I20260130 02:57:54.976580 19663 consensus_queue.cc:237] T b078934be4bb4fd1915c691eeea9d00f P 78c2c46c7520492db35d51c244fb4f74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } }
I20260130 02:57:54.981417 19626 tablet_bootstrap.cc:492] T a46702bef0e6442d85e4ca49a5f7cfbb P 78c2c46c7520492db35d51c244fb4f74: No bootstrap required, opened a new log
I20260130 02:57:54.981827 19626 ts_tablet_manager.cc:1403] T a46702bef0e6442d85e4ca49a5f7cfbb P 78c2c46c7520492db35d51c244fb4f74: Time spent bootstrapping tablet: real 0.276s user 0.007s sys 0.002s
W20260130 02:57:54.984241 18860 catalog_manager.cc:4952] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767): delete failed for tablet 3b8cdac4bf4b46efa26236fc930eb9a6 because the tablet was not found. No further retry: Not found: Tablet not found: 3b8cdac4bf4b46efa26236fc930eb9a6
I20260130 02:57:54.986248 19507 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2806ae64d7324b918397c3c6b72ebcfe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:57:54.986266 19358 tablet_service.cc:1505] Processing CreateTablet for tablet 2806ae64d7324b918397c3c6b72ebcfe (DEFAULT_TABLE table=default.Table_d1b81d6a8e704777b0fa35cf23dc91e0 [id=78a1af9c84574331a9112fdea3f67fcf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:57:54.987334 19358 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2806ae64d7324b918397c3c6b72ebcfe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:57:54.987115 19508 tablet_service.cc:1505] Processing CreateTablet for tablet 3b8cdac4bf4b46efa26236fc930eb9a6 (DEFAULT_TABLE table=default.Table_9428a7751159470793188f91cdcba4f3 [id=eaed0f6841e94faa858449fbb74bd5c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:57:54.987592 19506 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 93809949599a449ca09b13e9b8bbdef6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:57:54.994124 19508 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3b8cdac4bf4b46efa26236fc930eb9a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:57:54.995894 19716 tablet.cc:1722] T 96584ca174b140f7ad1a2ca698baca71 P 78c2c46c7520492db35d51c244fb4f74: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 5
I20260130 02:57:54.997627 19626 raft_consensus.cc:359] T a46702bef0e6442d85e4ca49a5f7cfbb P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } }
I20260130 02:57:54.998248 19626 raft_consensus.cc:385] T a46702bef0e6442d85e4ca49a5f7cfbb P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:57:54.998488 19626 raft_consensus.cc:740] T a46702bef0e6442d85e4ca49a5f7cfbb P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78c2c46c7520492db35d51c244fb4f74, State: Initialized, Role: FOLLOWER
I20260130 02:57:54.999053 19626 consensus_queue.cc:260] T a46702bef0e6442d85e4ca49a5f7cfbb P 78c2c46c7520492db35d51c244fb4f74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } }
I20260130 02:57:55.002341 18861 catalog_manager.cc:4985] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021): tablet f73338d4ce584880a9bec751d45ae753 (table default.Table_8e5ff21c1d224aab962d02463ea05a49 [id=74a7608fe84149d9bf37bcea9fe47b00]) successfully deleted
I20260130 02:57:55.015987 19626 ts_tablet_manager.cc:1434] T a46702bef0e6442d85e4ca49a5f7cfbb P 78c2c46c7520492db35d51c244fb4f74: Time spent starting tablet: real 0.034s user 0.006s sys 0.000s
I20260130 02:57:55.019411 19624 raft_consensus.cc:359] T 38a7a10e36874183bf9fe55a38d829f6 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:57:55.020033 19624 raft_consensus.cc:385] T 38a7a10e36874183bf9fe55a38d829f6 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:57:55.020257 19624 raft_consensus.cc:740] T 38a7a10e36874183bf9fe55a38d829f6 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 371a36d219164974abdf848841528152, State: Initialized, Role: FOLLOWER
I20260130 02:57:55.020828 19624 consensus_queue.cc:260] T 38a7a10e36874183bf9fe55a38d829f6 P 371a36d219164974abdf848841528152 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:57:55.021538 18870 catalog_manager.cc:5746] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571) does not have the latest schema for tablet 96584ca174b140f7ad1a2ca698baca71 (table default.Table_4f2d6e4d9c8e465087d65efc9c2de442 [id=1515db389f284e2f9ed1a3cd0061d0f1]). Expected version 5 got 4
I20260130 02:57:55.029326 19624 ts_tablet_manager.cc:1434] T 38a7a10e36874183bf9fe55a38d829f6 P 371a36d219164974abdf848841528152: Time spent starting tablet: real 0.065s user 0.004s sys 0.000s
I20260130 02:57:55.031184 19624 tablet_bootstrap.cc:492] T 2806ae64d7324b918397c3c6b72ebcfe P 371a36d219164974abdf848841528152: Bootstrap starting.
I20260130 02:57:55.037726 19624 tablet_bootstrap.cc:654] T 2806ae64d7324b918397c3c6b72ebcfe P 371a36d219164974abdf848841528152: Neither blocks nor log segments found. Creating new log.
I20260130 02:57:55.044179 18870 catalog_manager.cc:5746] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571) does not have the latest schema for tablet 38a7a10e36874183bf9fe55a38d829f6 (table default.Table_1a735ba1c9994f81ab61385e277b300c [id=27adf356fd004597a54c7a9887dd3ca8]). Expected version 1 got 0
I20260130 02:57:55.044996 18870 catalog_manager.cc:5746] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571) does not have the latest schema for tablet b078934be4bb4fd1915c691eeea9d00f (table default.Table_fcce594e0ed443158293215c25e4e5c3 [id=720decfb55c24f34983ef636f06e3169]). Expected version 1 got 0
I20260130 02:57:55.058427 18874 catalog_manager.cc:5746] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767) does not have the latest schema for tablet 38a7a10e36874183bf9fe55a38d829f6 (table default.Table_1a735ba1c9994f81ab61385e277b300c [id=27adf356fd004597a54c7a9887dd3ca8]). Expected version 1 got 0
I20260130 02:57:55.059134 18874 catalog_manager.cc:5746] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767) does not have the latest schema for tablet 96584ca174b140f7ad1a2ca698baca71 (table default.Table_4f2d6e4d9c8e465087d65efc9c2de442 [id=1515db389f284e2f9ed1a3cd0061d0f1]). Expected version 5 got 4
I20260130 02:57:55.059667 18874 catalog_manager.cc:5746] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767) does not have the latest schema for tablet b078934be4bb4fd1915c691eeea9d00f (table default.Table_fcce594e0ed443158293215c25e4e5c3 [id=720decfb55c24f34983ef636f06e3169]). Expected version 1 got 0
I20260130 02:57:55.065536 19711 ts_tablet_manager.cc:1929] T f73338d4ce584880a9bec751d45ae753 P 78c2c46c7520492db35d51c244fb4f74: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260130 02:57:55.066013 19711 log.cc:1199] T f73338d4ce584880a9bec751d45ae753 P 78c2c46c7520492db35d51c244fb4f74: Deleting WAL directory at /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-1/wal/wals/f73338d4ce584880a9bec751d45ae753
I20260130 02:57:55.077481 19711 ts_tablet_manager.cc:1950] T f73338d4ce584880a9bec751d45ae753 P 78c2c46c7520492db35d51c244fb4f74: Deleting consensus metadata
I20260130 02:57:55.083169 19626 tablet_bootstrap.cc:492] T f2d01bfdc9c4494386bd02bed655fe63 P 78c2c46c7520492db35d51c244fb4f74: Bootstrap starting.
I20260130 02:57:55.088621 19626 tablet_bootstrap.cc:654] T f2d01bfdc9c4494386bd02bed655fe63 P 78c2c46c7520492db35d51c244fb4f74: Neither blocks nor log segments found. Creating new log.
I20260130 02:57:55.090785 19625 tablet_bootstrap.cc:492] T 3b8cdac4bf4b46efa26236fc930eb9a6 P d870164263bb44b6b780a557daea3656: Bootstrap starting.
I20260130 02:57:55.095841 19625 tablet_bootstrap.cc:654] T 3b8cdac4bf4b46efa26236fc930eb9a6 P d870164263bb44b6b780a557daea3656: Neither blocks nor log segments found. Creating new log.
I20260130 02:57:55.104671 19360 tablet_service.cc:1505] Processing CreateTablet for tablet 3b8cdac4bf4b46efa26236fc930eb9a6 (DEFAULT_TABLE table=default.Table_9428a7751159470793188f91cdcba4f3 [id=eaed0f6841e94faa858449fbb74bd5c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:57:55.110296 19379 raft_consensus.cc:1275] T df2c8989d6c1480c97baa23db01141eb P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Refusing update from remote peer d870164263bb44b6b780a557daea3656: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260130 02:57:55.110447 19225 raft_consensus.cc:1275] T df2c8989d6c1480c97baa23db01141eb P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Refusing update from remote peer d870164263bb44b6b780a557daea3656: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260130 02:57:55.112046 19630 consensus_queue.cc:1048] T df2c8989d6c1480c97baa23db01141eb P d870164263bb44b6b780a557daea3656 [LEADER]: Connected to new peer: Peer: permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260130 02:57:55.114434 19677 consensus_queue.cc:1048] T df2c8989d6c1480c97baa23db01141eb P d870164263bb44b6b780a557daea3656 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260130 02:57:55.120285 19626 tablet_bootstrap.cc:492] T f2d01bfdc9c4494386bd02bed655fe63 P 78c2c46c7520492db35d51c244fb4f74: No bootstrap required, opened a new log
I20260130 02:57:55.120770 19626 ts_tablet_manager.cc:1403] T f2d01bfdc9c4494386bd02bed655fe63 P 78c2c46c7520492db35d51c244fb4f74: Time spent bootstrapping tablet: real 0.038s user 0.010s sys 0.001s
I20260130 02:57:55.124030 19677 raft_consensus.cc:493] T c2a9675427754d0db1578637a37440c6 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:57:55.124466 19677 raft_consensus.cc:515] T c2a9675427754d0db1578637a37440c6 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:57:55.124562 19626 raft_consensus.cc:359] T f2d01bfdc9c4494386bd02bed655fe63 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:57:55.125229 19626 raft_consensus.cc:385] T f2d01bfdc9c4494386bd02bed655fe63 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:57:55.125526 19626 raft_consensus.cc:740] T f2d01bfdc9c4494386bd02bed655fe63 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78c2c46c7520492db35d51c244fb4f74, State: Initialized, Role: FOLLOWER
I20260130 02:57:55.127141 19677 leader_election.cc:290] T c2a9675427754d0db1578637a37440c6 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767), 371a36d219164974abdf848841528152 (127.18.100.65:42021)
I20260130 02:57:55.127097 19380 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c2a9675427754d0db1578637a37440c6" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74" is_pre_election: true
I20260130 02:57:55.127660 19380 raft_consensus.cc:2468] T c2a9675427754d0db1578637a37440c6 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d870164263bb44b6b780a557daea3656 in term 0.
I20260130 02:57:55.127909 19225 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c2a9675427754d0db1578637a37440c6" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152" is_pre_election: true
I20260130 02:57:55.128595 19225 raft_consensus.cc:2468] T c2a9675427754d0db1578637a37440c6 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d870164263bb44b6b780a557daea3656 in term 0.
I20260130 02:57:55.128794 19462 leader_election.cc:304] T c2a9675427754d0db1578637a37440c6 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78c2c46c7520492db35d51c244fb4f74, d870164263bb44b6b780a557daea3656; no voters:
I20260130 02:57:55.129571 19677 raft_consensus.cc:2804] T c2a9675427754d0db1578637a37440c6 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260130 02:57:55.129936 19677 raft_consensus.cc:493] T c2a9675427754d0db1578637a37440c6 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260130 02:57:55.130373 19677 raft_consensus.cc:3060] T c2a9675427754d0db1578637a37440c6 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:57:55.140041 19679 raft_consensus.cc:493] T 79e28c3a566c4521ac2453bbaa3567ed P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:57:55.140695 19679 raft_consensus.cc:515] T 79e28c3a566c4521ac2453bbaa3567ed P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } }
I20260130 02:57:55.142721 19679 leader_election.cc:290] T 79e28c3a566c4521ac2453bbaa3567ed P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d870164263bb44b6b780a557daea3656 (127.18.100.67:39571), 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767)
I20260130 02:57:55.144244 19527 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "79e28c3a566c4521ac2453bbaa3567ed" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
I20260130 02:57:55.144913 19527 raft_consensus.cc:2468] T 79e28c3a566c4521ac2453bbaa3567ed P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 371a36d219164974abdf848841528152 in term 0.
W20260130 02:57:55.148187 19159 leader_election.cc:343] T 79e28c3a566c4521ac2453bbaa3567ed P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767): Not found: Tablet not found: 79e28c3a566c4521ac2453bbaa3567ed
W20260130 02:57:55.148895 18861 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet b078934be4bb4fd1915c691eeea9d00f (table default.Table_fcce594e0ed443158293215c25e4e5c3 [id=720decfb55c24f34983ef636f06e3169], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet b078934be4bb4fd1915c691eeea9d00f consensus state has no leader
I20260130 02:57:55.151834 19625 tablet_bootstrap.cc:492] T 3b8cdac4bf4b46efa26236fc930eb9a6 P d870164263bb44b6b780a557daea3656: No bootstrap required, opened a new log
I20260130 02:57:55.152750 19625 ts_tablet_manager.cc:1403] T 3b8cdac4bf4b46efa26236fc930eb9a6 P d870164263bb44b6b780a557daea3656: Time spent bootstrapping tablet: real 0.062s user 0.010s sys 0.000s
I20260130 02:57:55.156374 19625 raft_consensus.cc:359] T 3b8cdac4bf4b46efa26236fc930eb9a6 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:57:55.156579 19159 leader_election.cc:304] T 79e28c3a566c4521ac2453bbaa3567ed P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 371a36d219164974abdf848841528152, d870164263bb44b6b780a557daea3656; no voters: 78c2c46c7520492db35d51c244fb4f74
I20260130 02:57:55.160782 19625 raft_consensus.cc:385] T 3b8cdac4bf4b46efa26236fc930eb9a6 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:57:55.126118 19626 consensus_queue.cc:260] T f2d01bfdc9c4494386bd02bed655fe63 P 78c2c46c7520492db35d51c244fb4f74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:57:55.163872 19679 raft_consensus.cc:2804] T 79e28c3a566c4521ac2453bbaa3567ed P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260130 02:57:55.164249 19679 raft_consensus.cc:493] T 79e28c3a566c4521ac2453bbaa3567ed P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260130 02:57:55.164595 19679 raft_consensus.cc:3060] T 79e28c3a566c4521ac2453bbaa3567ed P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:57:55.167721 19360 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3b8cdac4bf4b46efa26236fc930eb9a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:57:55.168992 19677 raft_consensus.cc:515] T c2a9675427754d0db1578637a37440c6 P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:57:55.170790 19677 leader_election.cc:290] T c2a9675427754d0db1578637a37440c6 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 election: Requested vote from peers 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767), 371a36d219164974abdf848841528152 (127.18.100.65:42021)
I20260130 02:57:55.164919 19625 raft_consensus.cc:740] T 3b8cdac4bf4b46efa26236fc930eb9a6 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d870164263bb44b6b780a557daea3656, State: Initialized, Role: FOLLOWER
I20260130 02:57:55.172477 19225 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c2a9675427754d0db1578637a37440c6" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152"
I20260130 02:57:55.172437 19625 consensus_queue.cc:260] T 3b8cdac4bf4b46efa26236fc930eb9a6 P d870164263bb44b6b780a557daea3656 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:57:55.173019 19225 raft_consensus.cc:3060] T c2a9675427754d0db1578637a37440c6 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:57:55.175035 19710 ts_tablet_manager.cc:1929] T f73338d4ce584880a9bec751d45ae753 P d870164263bb44b6b780a557daea3656: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260130 02:57:55.175396 19710 log.cc:1199] T f73338d4ce584880a9bec751d45ae753 P d870164263bb44b6b780a557daea3656: Deleting WAL directory at /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-2/wal/wals/f73338d4ce584880a9bec751d45ae753
I20260130 02:57:55.176141 19710 ts_tablet_manager.cc:1950] T f73338d4ce584880a9bec751d45ae753 P d870164263bb44b6b780a557daea3656: Deleting consensus metadata
I20260130 02:57:55.177742 19625 ts_tablet_manager.cc:1434] T 3b8cdac4bf4b46efa26236fc930eb9a6 P d870164263bb44b6b780a557daea3656: Time spent starting tablet: real 0.023s user 0.004s sys 0.000s
I20260130 02:57:55.178556 19625 tablet_bootstrap.cc:492] T 93809949599a449ca09b13e9b8bbdef6 P d870164263bb44b6b780a557daea3656: Bootstrap starting.
I20260130 02:57:55.180718 19689 tablet.cc:1722] T df2c8989d6c1480c97baa23db01141eb P d870164263bb44b6b780a557daea3656: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260130 02:57:55.182741 19635 consensus_queue.cc:1048] T 06b31d020b864c418476e6dcc1e4eafb P 371a36d219164974abdf848841528152 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260130 02:57:55.191917 19626 ts_tablet_manager.cc:1434] T f2d01bfdc9c4494386bd02bed655fe63 P 78c2c46c7520492db35d51c244fb4f74: Time spent starting tablet: real 0.070s user 0.006s sys 0.000s
I20260130 02:57:55.193831 18860 catalog_manager.cc:4985] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767): tablet f73338d4ce584880a9bec751d45ae753 (table default.Table_8e5ff21c1d224aab962d02463ea05a49 [id=74a7608fe84149d9bf37bcea9fe47b00]) successfully deleted
W20260130 02:57:55.194626 18860 catalog_manager.cc:4952] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767): delete failed for tablet 79e28c3a566c4521ac2453bbaa3567ed because the tablet was not found. No further retry: Not found: Tablet not found: 79e28c3a566c4521ac2453bbaa3567ed
I20260130 02:57:55.203426 19625 tablet_bootstrap.cc:654] T 93809949599a449ca09b13e9b8bbdef6 P d870164263bb44b6b780a557daea3656: Neither blocks nor log segments found. Creating new log.
I20260130 02:57:55.203929 19379 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c2a9675427754d0db1578637a37440c6" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74"
I20260130 02:57:55.204461 19379 raft_consensus.cc:3060] T c2a9675427754d0db1578637a37440c6 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:57:55.212318 19716 tablet.cc:1722] T df2c8989d6c1480c97baa23db01141eb P 78c2c46c7520492db35d51c244fb4f74: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260130 02:57:55.217744 19626 tablet_bootstrap.cc:492] T 93809949599a449ca09b13e9b8bbdef6 P 78c2c46c7520492db35d51c244fb4f74: Bootstrap starting.
I20260130 02:57:55.223174 18869 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50284:
table { table_name: "default.Table_23120efe2754472c9f41c54904af4081" } new_table_name: "default.Table_14c76bd65820453fb5cade67cf66e752" modify_external_catalogs: true
I20260130 02:57:55.224984 19626 tablet_bootstrap.cc:654] T 93809949599a449ca09b13e9b8bbdef6 P 78c2c46c7520492db35d51c244fb4f74: Neither blocks nor log segments found. Creating new log.
I20260130 02:57:55.243559 19625 tablet_bootstrap.cc:492] T 93809949599a449ca09b13e9b8bbdef6 P d870164263bb44b6b780a557daea3656: No bootstrap required, opened a new log
I20260130 02:57:55.244561 19625 ts_tablet_manager.cc:1403] T 93809949599a449ca09b13e9b8bbdef6 P d870164263bb44b6b780a557daea3656: Time spent bootstrapping tablet: real 0.066s user 0.013s sys 0.000s
I20260130 02:57:55.246785 19625 raft_consensus.cc:359] T 93809949599a449ca09b13e9b8bbdef6 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:57:55.247332 18873 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50284:
table { table_name: "default.Table_d1b81d6a8e704777b0fa35cf23dc91e0" } modify_external_catalogs: true
I20260130 02:57:55.247661 19625 raft_consensus.cc:385] T 93809949599a449ca09b13e9b8bbdef6 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:57:55.247900 19625 raft_consensus.cc:740] T 93809949599a449ca09b13e9b8bbdef6 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d870164263bb44b6b780a557daea3656, State: Initialized, Role: FOLLOWER
I20260130 02:57:55.247874 18873 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50284:
table { table_name: "default.Table_d1b81d6a8e704777b0fa35cf23dc91e0" } modify_external_catalogs: true
I20260130 02:57:55.251035 19625 consensus_queue.cc:260] T 93809949599a449ca09b13e9b8bbdef6 P d870164263bb44b6b780a557daea3656 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:57:55.253019 19625 ts_tablet_manager.cc:1434] T 93809949599a449ca09b13e9b8bbdef6 P d870164263bb44b6b780a557daea3656: Time spent starting tablet: real 0.008s user 0.004s sys 0.000s
I20260130 02:57:55.253870 19625 tablet_bootstrap.cc:492] T 2806ae64d7324b918397c3c6b72ebcfe P d870164263bb44b6b780a557daea3656: Bootstrap starting.
I20260130 02:57:55.259393 19625 tablet_bootstrap.cc:654] T 2806ae64d7324b918397c3c6b72ebcfe P d870164263bb44b6b780a557daea3656: Neither blocks nor log segments found. Creating new log.
I20260130 02:57:55.260284 19626 tablet_bootstrap.cc:492] T 93809949599a449ca09b13e9b8bbdef6 P 78c2c46c7520492db35d51c244fb4f74: No bootstrap required, opened a new log
I20260130 02:57:55.262346 18867 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50284:
table { table_name: "default.Table_e9d466c5c67e4df1a06a8eb665da9570" } modify_external_catalogs: true
I20260130 02:57:55.263067 18867 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50284:
table { table_name: "default.Table_e9d466c5c67e4df1a06a8eb665da9570" } modify_external_catalogs: true
I20260130 02:57:55.266407 19379 raft_consensus.cc:2468] T c2a9675427754d0db1578637a37440c6 P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d870164263bb44b6b780a557daea3656 in term 1.
I20260130 02:57:55.273126 19630 raft_consensus.cc:493] T 38a7a10e36874183bf9fe55a38d829f6 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:57:55.273599 19630 raft_consensus.cc:515] T 38a7a10e36874183bf9fe55a38d829f6 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:57:55.275430 19630 leader_election.cc:290] T 38a7a10e36874183bf9fe55a38d829f6 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767)
I20260130 02:57:55.276336 19462 leader_election.cc:304] T c2a9675427754d0db1578637a37440c6 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78c2c46c7520492db35d51c244fb4f74, d870164263bb44b6b780a557daea3656; no voters:
I20260130 02:57:55.279310 19626 ts_tablet_manager.cc:1403] T 93809949599a449ca09b13e9b8bbdef6 P 78c2c46c7520492db35d51c244fb4f74: Time spent bootstrapping tablet: real 0.062s user 0.008s sys 0.004s
I20260130 02:57:55.279583 19630 raft_consensus.cc:2804] T c2a9675427754d0db1578637a37440c6 P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Leader election won for term 1
I20260130 02:57:55.280030 19630 raft_consensus.cc:697] T c2a9675427754d0db1578637a37440c6 P d870164263bb44b6b780a557daea3656 [term 1 LEADER]: Becoming Leader. State: Replica: d870164263bb44b6b780a557daea3656, State: Running, Role: LEADER
I20260130 02:57:55.280766 19630 consensus_queue.cc:237] T c2a9675427754d0db1578637a37440c6 P d870164263bb44b6b780a557daea3656 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:57:55.281656 19626 raft_consensus.cc:359] T 93809949599a449ca09b13e9b8bbdef6 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:57:55.282199 19507 tablet_service.cc:1552] Processing DeleteTablet for tablet 79e28c3a566c4521ac2453bbaa3567ed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:57:53 UTC) from {username='slave'} at 127.0.0.1:59068
I20260130 02:57:55.282264 19626 raft_consensus.cc:385] T 93809949599a449ca09b13e9b8bbdef6 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:57:55.282917 19710 tablet_replica.cc:333] T 79e28c3a566c4521ac2453bbaa3567ed P d870164263bb44b6b780a557daea3656: stopping tablet replica
I20260130 02:57:55.283725 19710 raft_consensus.cc:2243] T 79e28c3a566c4521ac2453bbaa3567ed P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260130 02:57:55.284149 19710 raft_consensus.cc:2272] T 79e28c3a566c4521ac2453bbaa3567ed P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260130 02:57:55.282507 19626 raft_consensus.cc:740] T 93809949599a449ca09b13e9b8bbdef6 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78c2c46c7520492db35d51c244fb4f74, State: Initialized, Role: FOLLOWER
I20260130 02:57:55.284950 19626 consensus_queue.cc:260] T 93809949599a449ca09b13e9b8bbdef6 P 78c2c46c7520492db35d51c244fb4f74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:57:55.286846 19710 ts_tablet_manager.cc:1916] T 79e28c3a566c4521ac2453bbaa3567ed P d870164263bb44b6b780a557daea3656: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260130 02:57:55.286957 19626 ts_tablet_manager.cc:1434] T 93809949599a449ca09b13e9b8bbdef6 P 78c2c46c7520492db35d51c244fb4f74: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260130 02:57:55.292114 19379 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "38a7a10e36874183bf9fe55a38d829f6" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74" is_pre_election: true
I20260130 02:57:55.292805 19379 raft_consensus.cc:2468] T 38a7a10e36874183bf9fe55a38d829f6 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d870164263bb44b6b780a557daea3656 in term 0.
I20260130 02:57:55.293779 19462 leader_election.cc:304] T 38a7a10e36874183bf9fe55a38d829f6 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78c2c46c7520492db35d51c244fb4f74, d870164263bb44b6b780a557daea3656; no voters:
I20260130 02:57:55.294529 19677 raft_consensus.cc:2804] T 38a7a10e36874183bf9fe55a38d829f6 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260130 02:57:55.294869 19677 raft_consensus.cc:493] T 38a7a10e36874183bf9fe55a38d829f6 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260130 02:57:55.295166 19677 raft_consensus.cc:3060] T 38a7a10e36874183bf9fe55a38d829f6 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:57:55.295444 19712 tablet.cc:1722] T df2c8989d6c1480c97baa23db01141eb P 371a36d219164974abdf848841528152: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260130 02:57:55.298878 19225 raft_consensus.cc:2468] T c2a9675427754d0db1578637a37440c6 P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d870164263bb44b6b780a557daea3656 in term 1.
I20260130 02:57:55.305510 19225 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "38a7a10e36874183bf9fe55a38d829f6" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152" is_pre_election: true
I20260130 02:57:55.306118 19225 raft_consensus.cc:2468] T 38a7a10e36874183bf9fe55a38d829f6 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d870164263bb44b6b780a557daea3656 in term 0.
I20260130 02:57:55.303117 19679 raft_consensus.cc:515] T 79e28c3a566c4521ac2453bbaa3567ed P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } }
I20260130 02:57:55.311367 19626 tablet_bootstrap.cc:492] T 2806ae64d7324b918397c3c6b72ebcfe P 78c2c46c7520492db35d51c244fb4f74: Bootstrap starting.
I20260130 02:57:55.311553 19679 leader_election.cc:290] T 79e28c3a566c4521ac2453bbaa3567ed P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 election: Requested vote from peers d870164263bb44b6b780a557daea3656 (127.18.100.67:39571), 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767)
I20260130 02:57:55.316730 19527 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "79e28c3a566c4521ac2453bbaa3567ed" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656"
W20260130 02:57:55.318193 19159 leader_election.cc:343] T 79e28c3a566c4521ac2453bbaa3567ed P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): Illegal state: Tablet not RUNNING: STOPPED
I20260130 02:57:55.325623 19679 raft_consensus.cc:493] T a46702bef0e6442d85e4ca49a5f7cfbb P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260130 02:57:55.326968 19159 leader_election.cc:343] T 79e28c3a566c4521ac2453bbaa3567ed P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767): Not found: Tablet not found: 79e28c3a566c4521ac2453bbaa3567ed
I20260130 02:57:55.327370 19159 leader_election.cc:304] T 79e28c3a566c4521ac2453bbaa3567ed P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 371a36d219164974abdf848841528152; no voters: 78c2c46c7520492db35d51c244fb4f74, d870164263bb44b6b780a557daea3656
I20260130 02:57:55.326108 19679 raft_consensus.cc:515] T a46702bef0e6442d85e4ca49a5f7cfbb P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } }
I20260130 02:57:55.328426 19635 raft_consensus.cc:2749] T 79e28c3a566c4521ac2453bbaa3567ed P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260130 02:57:55.329082 19679 leader_election.cc:290] T a46702bef0e6442d85e4ca49a5f7cfbb P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767), d870164263bb44b6b780a557daea3656 (127.18.100.67:39571)
I20260130 02:57:55.331106 19527 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a46702bef0e6442d85e4ca49a5f7cfbb" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
I20260130 02:57:55.331665 19527 raft_consensus.cc:2468] T a46702bef0e6442d85e4ca49a5f7cfbb P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 371a36d219164974abdf848841528152 in term 0.
I20260130 02:57:55.332228 19625 tablet_bootstrap.cc:492] T 2806ae64d7324b918397c3c6b72ebcfe P d870164263bb44b6b780a557daea3656: No bootstrap required, opened a new log
I20260130 02:57:55.332674 19625 ts_tablet_manager.cc:1403] T 2806ae64d7324b918397c3c6b72ebcfe P d870164263bb44b6b780a557daea3656: Time spent bootstrapping tablet: real 0.079s user 0.007s sys 0.002s
I20260130 02:57:55.332697 19159 leader_election.cc:304] T a46702bef0e6442d85e4ca49a5f7cfbb P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 371a36d219164974abdf848841528152, d870164263bb44b6b780a557daea3656; no voters:
I20260130 02:57:55.333999 19379 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a46702bef0e6442d85e4ca49a5f7cfbb" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74" is_pre_election: true
I20260130 02:57:55.334601 19379 raft_consensus.cc:2468] T a46702bef0e6442d85e4ca49a5f7cfbb P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 371a36d219164974abdf848841528152 in term 0.
I20260130 02:57:55.335253 19625 raft_consensus.cc:359] T 2806ae64d7324b918397c3c6b72ebcfe P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:57:55.336686 19625 raft_consensus.cc:385] T 2806ae64d7324b918397c3c6b72ebcfe P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:57:55.336689 19679 raft_consensus.cc:2804] T a46702bef0e6442d85e4ca49a5f7cfbb P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260130 02:57:55.337007 19679 raft_consensus.cc:493] T a46702bef0e6442d85e4ca49a5f7cfbb P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260130 02:57:55.337024 19625 raft_consensus.cc:740] T 2806ae64d7324b918397c3c6b72ebcfe P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d870164263bb44b6b780a557daea3656, State: Initialized, Role: FOLLOWER
I20260130 02:57:55.337313 19679 raft_consensus.cc:3060] T a46702bef0e6442d85e4ca49a5f7cfbb P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:57:55.337378 19626 tablet_bootstrap.cc:654] T 2806ae64d7324b918397c3c6b72ebcfe P 78c2c46c7520492db35d51c244fb4f74: Neither blocks nor log segments found. Creating new log.
I20260130 02:57:55.337970 19625 consensus_queue.cc:260] T 2806ae64d7324b918397c3c6b72ebcfe P d870164263bb44b6b780a557daea3656 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:57:55.349591 19625 ts_tablet_manager.cc:1434] T 2806ae64d7324b918397c3c6b72ebcfe P d870164263bb44b6b780a557daea3656: Time spent starting tablet: real 0.017s user 0.004s sys 0.000s
I20260130 02:57:55.369537 19677 raft_consensus.cc:515] T 38a7a10e36874183bf9fe55a38d829f6 P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:57:55.371483 19677 leader_election.cc:290] T 38a7a10e36874183bf9fe55a38d829f6 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 election: Requested vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767)
I20260130 02:57:55.374902 19380 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "38a7a10e36874183bf9fe55a38d829f6" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74"
I20260130 02:57:55.375576 19380 raft_consensus.cc:3060] T 38a7a10e36874183bf9fe55a38d829f6 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:57:55.375908 19225 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "38a7a10e36874183bf9fe55a38d829f6" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152"
I20260130 02:57:55.376439 19225 raft_consensus.cc:3060] T 38a7a10e36874183bf9fe55a38d829f6 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:57:55.379652 19635 consensus_queue.cc:1048] T 06b31d020b864c418476e6dcc1e4eafb P 371a36d219164974abdf848841528152 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260130 02:57:55.383266 18860 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.100.124:34629, user_credentials={real_user=slave}} blocked reactor thread for 87062.4us
I20260130 02:57:55.386518 19624 tablet_bootstrap.cc:492] T 2806ae64d7324b918397c3c6b72ebcfe P 371a36d219164974abdf848841528152: No bootstrap required, opened a new log
I20260130 02:57:55.386940 19624 ts_tablet_manager.cc:1403] T 2806ae64d7324b918397c3c6b72ebcfe P 371a36d219164974abdf848841528152: Time spent bootstrapping tablet: real 0.356s user 0.097s sys 0.097s
I20260130 02:57:55.389074 19624 raft_consensus.cc:359] T 2806ae64d7324b918397c3c6b72ebcfe P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:57:55.389668 19624 raft_consensus.cc:385] T 2806ae64d7324b918397c3c6b72ebcfe P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:57:55.389909 19624 raft_consensus.cc:740] T 2806ae64d7324b918397c3c6b72ebcfe P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 371a36d219164974abdf848841528152, State: Initialized, Role: FOLLOWER
I20260130 02:57:55.391003 18860 catalog_manager.cc:4985] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): tablet f73338d4ce584880a9bec751d45ae753 (table default.Table_8e5ff21c1d224aab962d02463ea05a49 [id=74a7608fe84149d9bf37bcea9fe47b00]) successfully deleted
I20260130 02:57:55.392015 19626 tablet_bootstrap.cc:492] T 2806ae64d7324b918397c3c6b72ebcfe P 78c2c46c7520492db35d51c244fb4f74: No bootstrap required, opened a new log
I20260130 02:57:55.392601 19626 ts_tablet_manager.cc:1403] T 2806ae64d7324b918397c3c6b72ebcfe P 78c2c46c7520492db35d51c244fb4f74: Time spent bootstrapping tablet: real 0.081s user 0.008s sys 0.004s
I20260130 02:57:55.393874 19624 consensus_queue.cc:260] T 2806ae64d7324b918397c3c6b72ebcfe P 371a36d219164974abdf848841528152 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:57:55.403048 19626 raft_consensus.cc:359] T 2806ae64d7324b918397c3c6b72ebcfe P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:57:55.406749 19624 ts_tablet_manager.cc:1434] T 2806ae64d7324b918397c3c6b72ebcfe P 371a36d219164974abdf848841528152: Time spent starting tablet: real 0.020s user 0.006s sys 0.000s
I20260130 02:57:55.406642 19679 raft_consensus.cc:515] T a46702bef0e6442d85e4ca49a5f7cfbb P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } }
I20260130 02:57:55.407604 19624 tablet_bootstrap.cc:492] T f2d01bfdc9c4494386bd02bed655fe63 P 371a36d219164974abdf848841528152: Bootstrap starting.
I20260130 02:57:55.407920 19626 raft_consensus.cc:385] T 2806ae64d7324b918397c3c6b72ebcfe P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:57:55.408262 19626 raft_consensus.cc:740] T 2806ae64d7324b918397c3c6b72ebcfe P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78c2c46c7520492db35d51c244fb4f74, State: Initialized, Role: FOLLOWER
I20260130 02:57:55.412391 19626 consensus_queue.cc:260] T 2806ae64d7324b918397c3c6b72ebcfe P 78c2c46c7520492db35d51c244fb4f74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:57:55.414218 19677 raft_consensus.cc:493] T f2d01bfdc9c4494386bd02bed655fe63 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:57:55.414698 19677 raft_consensus.cc:515] T f2d01bfdc9c4494386bd02bed655fe63 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:57:55.416513 19677 leader_election.cc:290] T f2d01bfdc9c4494386bd02bed655fe63 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767)
I20260130 02:57:55.418277 19377 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f2d01bfdc9c4494386bd02bed655fe63" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74" is_pre_election: true
I20260130 02:57:55.418953 19377 raft_consensus.cc:2468] T f2d01bfdc9c4494386bd02bed655fe63 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d870164263bb44b6b780a557daea3656 in term 0.
I20260130 02:57:55.420452 19462 leader_election.cc:304] T f2d01bfdc9c4494386bd02bed655fe63 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78c2c46c7520492db35d51c244fb4f74, d870164263bb44b6b780a557daea3656; no voters:
I20260130 02:57:55.421484 19677 raft_consensus.cc:2804] T f2d01bfdc9c4494386bd02bed655fe63 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260130 02:57:55.421866 19677 raft_consensus.cc:493] T f2d01bfdc9c4494386bd02bed655fe63 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260130 02:57:55.422390 19677 raft_consensus.cc:3060] T f2d01bfdc9c4494386bd02bed655fe63 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:57:55.434912 19626 ts_tablet_manager.cc:1434] T 2806ae64d7324b918397c3c6b72ebcfe P 78c2c46c7520492db35d51c244fb4f74: Time spent starting tablet: real 0.042s user 0.004s sys 0.002s
W20260130 02:57:55.436882 18868 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 37470ce4356f4542b621e21698551209 has no consensus state
I20260130 02:57:55.443513 19626 tablet_bootstrap.cc:492] T 3b8cdac4bf4b46efa26236fc930eb9a6 P 78c2c46c7520492db35d51c244fb4f74: Bootstrap starting.
I20260130 02:57:55.453037 19624 tablet_bootstrap.cc:654] T f2d01bfdc9c4494386bd02bed655fe63 P 371a36d219164974abdf848841528152: Neither blocks nor log segments found. Creating new log.
I20260130 02:57:55.452323 18871 catalog_manager.cc:5654] T 927c90b84d7449efa36bc5e2035bf0da P 371a36d219164974abdf848841528152 reported cstate change: term changed from 0 to 1, leader changed from <none> to 371a36d219164974abdf848841528152 (127.18.100.65). New cstate: current_term: 1 leader_uuid: "371a36d219164974abdf848841528152" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } health_report { overall_health: HEALTHY } } }
I20260130 02:57:55.458997 19679 leader_election.cc:290] T a46702bef0e6442d85e4ca49a5f7cfbb P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 election: Requested vote from peers 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767), d870164263bb44b6b780a557daea3656 (127.18.100.67:39571)
I20260130 02:57:55.460407 19679 consensus_queue.cc:1048] T 927c90b84d7449efa36bc5e2035bf0da P 371a36d219164974abdf848841528152 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260130 02:57:55.461086 19626 tablet_bootstrap.cc:654] T 3b8cdac4bf4b46efa26236fc930eb9a6 P 78c2c46c7520492db35d51c244fb4f74: Neither blocks nor log segments found. Creating new log.
I20260130 02:57:55.461495 19635 consensus_queue.cc:1048] T 927c90b84d7449efa36bc5e2035bf0da P 371a36d219164974abdf848841528152 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260130 02:57:55.461920 19377 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a46702bef0e6442d85e4ca49a5f7cfbb" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74"
I20260130 02:57:55.462481 19377 raft_consensus.cc:3060] T a46702bef0e6442d85e4ca49a5f7cfbb P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:57:55.466152 19527 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a46702bef0e6442d85e4ca49a5f7cfbb" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656"
I20260130 02:57:55.466717 19527 raft_consensus.cc:3060] T a46702bef0e6442d85e4ca49a5f7cfbb P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:57:55.466853 19380 raft_consensus.cc:2468] T 38a7a10e36874183bf9fe55a38d829f6 P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d870164263bb44b6b780a557daea3656 in term 1.
I20260130 02:57:55.466950 19677 raft_consensus.cc:515] T f2d01bfdc9c4494386bd02bed655fe63 P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:57:55.469462 19462 leader_election.cc:304] T 38a7a10e36874183bf9fe55a38d829f6 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78c2c46c7520492db35d51c244fb4f74, d870164263bb44b6b780a557daea3656; no voters:
I20260130 02:57:55.471280 19380 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f2d01bfdc9c4494386bd02bed655fe63" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74"
I20260130 02:57:55.471812 19380 raft_consensus.cc:3060] T f2d01bfdc9c4494386bd02bed655fe63 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:57:55.473068 19358 tablet_service.cc:1505] Processing CreateTablet for tablet 37470ce4356f4542b621e21698551209 (DEFAULT_TABLE table=default.Table_007e969f70d345a7bcb3dfbaabbc025d [id=77d4fa8705a3419cb0a84d3ca1bb4ce3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:57:55.473443 19710 ts_tablet_manager.cc:1929] T 79e28c3a566c4521ac2453bbaa3567ed P d870164263bb44b6b780a557daea3656: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260130 02:57:55.473788 19710 log.cc:1199] T 79e28c3a566c4521ac2453bbaa3567ed P d870164263bb44b6b780a557daea3656: Deleting WAL directory at /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-2/wal/wals/79e28c3a566c4521ac2453bbaa3567ed
I20260130 02:57:55.474093 19358 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 37470ce4356f4542b621e21698551209. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:57:55.474435 19710 ts_tablet_manager.cc:1950] T 79e28c3a566c4521ac2453bbaa3567ed P d870164263bb44b6b780a557daea3656: Deleting consensus metadata
I20260130 02:57:55.475245 19724 raft_consensus.cc:2804] T 38a7a10e36874183bf9fe55a38d829f6 P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Leader election won for term 1
I20260130 02:57:55.475956 19630 raft_consensus.cc:493] T 93809949599a449ca09b13e9b8bbdef6 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:57:55.476323 19630 raft_consensus.cc:515] T 93809949599a449ca09b13e9b8bbdef6 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:57:55.478727 19630 leader_election.cc:290] T 93809949599a449ca09b13e9b8bbdef6 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767)
I20260130 02:57:55.454275 18871 catalog_manager.cc:5746] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021) does not have the latest schema for tablet b078934be4bb4fd1915c691eeea9d00f (table default.Table_fcce594e0ed443158293215c25e4e5c3 [id=720decfb55c24f34983ef636f06e3169]). Expected version 1 got 0
I20260130 02:57:55.485941 19376 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "93809949599a449ca09b13e9b8bbdef6" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74" is_pre_election: true
I20260130 02:57:55.486642 19376 raft_consensus.cc:2468] T 93809949599a449ca09b13e9b8bbdef6 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d870164263bb44b6b780a557daea3656 in term 0.
I20260130 02:57:55.487051 19507 tablet_service.cc:1505] Processing CreateTablet for tablet 37470ce4356f4542b621e21698551209 (DEFAULT_TABLE table=default.Table_007e969f70d345a7bcb3dfbaabbc025d [id=77d4fa8705a3419cb0a84d3ca1bb4ce3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:57:55.488070 19507 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 37470ce4356f4542b621e21698551209. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:57:55.494995 18871 catalog_manager.cc:5746] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021) does not have the latest schema for tablet 38a7a10e36874183bf9fe55a38d829f6 (table default.Table_1a735ba1c9994f81ab61385e277b300c [id=27adf356fd004597a54c7a9887dd3ca8]). Expected version 1 got 0
I20260130 02:57:55.495682 19677 leader_election.cc:290] T f2d01bfdc9c4494386bd02bed655fe63 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 election: Requested vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767)
I20260130 02:57:55.500144 19224 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f2d01bfdc9c4494386bd02bed655fe63" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152" is_pre_election: true
I20260130 02:57:55.501665 19202 tablet_service.cc:1505] Processing CreateTablet for tablet 37470ce4356f4542b621e21698551209 (DEFAULT_TABLE table=default.Table_007e969f70d345a7bcb3dfbaabbc025d [id=77d4fa8705a3419cb0a84d3ca1bb4ce3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:57:55.502717 19202 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 37470ce4356f4542b621e21698551209. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:57:55.502686 19204 tablet_service.cc:1505] Processing CreateTablet for tablet 878f9346d3044d8d8b15c61f86869212 (DEFAULT_TABLE table=default.Table_a8b2bef1c4564242b8d5fc95a3a35f85 [id=2009496525c94d7f8ee6ce59694b652d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:57:55.503736 19204 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 878f9346d3044d8d8b15c61f86869212. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:57:55.506735 19224 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f2d01bfdc9c4494386bd02bed655fe63" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152"
I20260130 02:57:55.507498 19223 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "93809949599a449ca09b13e9b8bbdef6" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152" is_pre_election: true
I20260130 02:57:55.513093 19624 tablet_bootstrap.cc:492] T f2d01bfdc9c4494386bd02bed655fe63 P 371a36d219164974abdf848841528152: No bootstrap required, opened a new log
I20260130 02:57:55.513484 19624 ts_tablet_manager.cc:1403] T f2d01bfdc9c4494386bd02bed655fe63 P 371a36d219164974abdf848841528152: Time spent bootstrapping tablet: real 0.106s user 0.008s sys 0.008s
I20260130 02:57:55.515707 19624 raft_consensus.cc:359] T f2d01bfdc9c4494386bd02bed655fe63 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:57:55.516284 19624 raft_consensus.cc:385] T f2d01bfdc9c4494386bd02bed655fe63 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:57:55.516528 19624 raft_consensus.cc:740] T f2d01bfdc9c4494386bd02bed655fe63 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 371a36d219164974abdf848841528152, State: Initialized, Role: FOLLOWER
I20260130 02:57:55.517041 19624 consensus_queue.cc:260] T f2d01bfdc9c4494386bd02bed655fe63 P 371a36d219164974abdf848841528152 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:57:55.518939 19624 ts_tablet_manager.cc:1434] T f2d01bfdc9c4494386bd02bed655fe63 P 371a36d219164974abdf848841528152: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260130 02:57:55.519675 19624 tablet_bootstrap.cc:492] T 93809949599a449ca09b13e9b8bbdef6 P 371a36d219164974abdf848841528152: Bootstrap starting.
I20260130 02:57:55.532322 19624 tablet_bootstrap.cc:654] T 93809949599a449ca09b13e9b8bbdef6 P 371a36d219164974abdf848841528152: Neither blocks nor log segments found. Creating new log.
I20260130 02:57:55.533578 19203 tablet_service.cc:1505] Processing CreateTablet for tablet 01db06fcfffe4b259ac4240c0df38047 (DEFAULT_TABLE table=default.Table_e67bd1859b5846f386097cf33c5f1bcb [id=952ccb90477f47e4a352623247a8a6f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:57:55.534701 19203 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 01db06fcfffe4b259ac4240c0df38047. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:57:55.536840 19360 tablet_service.cc:1505] Processing CreateTablet for tablet 878f9346d3044d8d8b15c61f86869212 (DEFAULT_TABLE table=default.Table_a8b2bef1c4564242b8d5fc95a3a35f85 [id=2009496525c94d7f8ee6ce59694b652d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:57:55.537806 19360 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 878f9346d3044d8d8b15c61f86869212. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:57:55.497450 19724 raft_consensus.cc:697] T 38a7a10e36874183bf9fe55a38d829f6 P d870164263bb44b6b780a557daea3656 [term 1 LEADER]: Becoming Leader. State: Replica: d870164263bb44b6b780a557daea3656, State: Running, Role: LEADER
I20260130 02:57:55.538929 19462 leader_election.cc:304] T 93809949599a449ca09b13e9b8bbdef6 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78c2c46c7520492db35d51c244fb4f74, d870164263bb44b6b780a557daea3656; no voters:
I20260130 02:57:55.538875 19724 consensus_queue.cc:237] T 38a7a10e36874183bf9fe55a38d829f6 P d870164263bb44b6b780a557daea3656 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
W20260130 02:57:55.508111 19463 leader_election.cc:343] T f2d01bfdc9c4494386bd02bed655fe63 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Illegal state: must be running to vote when last-logged opid is not known
W20260130 02:57:55.543071 19463 leader_election.cc:343] T f2d01bfdc9c4494386bd02bed655fe63 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:57:55.543282 19742 raft_consensus.cc:2804] T 93809949599a449ca09b13e9b8bbdef6 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Leader pre-election won for term 1
W20260130 02:57:55.543686 19463 leader_election.cc:343] T 93809949599a449ca09b13e9b8bbdef6 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:57:55.543673 19742 raft_consensus.cc:493] T 93809949599a449ca09b13e9b8bbdef6 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260130 02:57:55.544729 19742 raft_consensus.cc:3060] T 93809949599a449ca09b13e9b8bbdef6 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:57:55.546474 19677 raft_consensus.cc:493] T 2806ae64d7324b918397c3c6b72ebcfe P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:57:55.546916 19677 raft_consensus.cc:515] T 2806ae64d7324b918397c3c6b72ebcfe P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:57:55.537607 19508 tablet_service.cc:1505] Processing CreateTablet for tablet 878f9346d3044d8d8b15c61f86869212 (DEFAULT_TABLE table=default.Table_a8b2bef1c4564242b8d5fc95a3a35f85 [id=2009496525c94d7f8ee6ce59694b652d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:57:55.548959 19508 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 878f9346d3044d8d8b15c61f86869212. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:57:55.549702 19223 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2806ae64d7324b918397c3c6b72ebcfe" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152" is_pre_election: true
I20260130 02:57:55.550412 19223 raft_consensus.cc:2468] T 2806ae64d7324b918397c3c6b72ebcfe P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d870164263bb44b6b780a557daea3656 in term 0.
I20260130 02:57:55.551384 19463 leader_election.cc:304] T 2806ae64d7324b918397c3c6b72ebcfe P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 371a36d219164974abdf848841528152, d870164263bb44b6b780a557daea3656; no voters:
I20260130 02:57:55.552048 19630 raft_consensus.cc:2804] T 2806ae64d7324b918397c3c6b72ebcfe P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260130 02:57:55.552333 19630 raft_consensus.cc:493] T 2806ae64d7324b918397c3c6b72ebcfe P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260130 02:57:55.552630 19630 raft_consensus.cc:3060] T 2806ae64d7324b918397c3c6b72ebcfe P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:57:55.554917 19379 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2806ae64d7324b918397c3c6b72ebcfe" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74" is_pre_election: true
I20260130 02:57:55.555056 19677 leader_election.cc:290] T 2806ae64d7324b918397c3c6b72ebcfe P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767)
I20260130 02:57:55.555488 19379 raft_consensus.cc:2468] T 2806ae64d7324b918397c3c6b72ebcfe P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d870164263bb44b6b780a557daea3656 in term 0.
I20260130 02:57:55.556324 19626 tablet_bootstrap.cc:492] T 3b8cdac4bf4b46efa26236fc930eb9a6 P 78c2c46c7520492db35d51c244fb4f74: No bootstrap required, opened a new log
I20260130 02:57:55.556720 19626 ts_tablet_manager.cc:1403] T 3b8cdac4bf4b46efa26236fc930eb9a6 P 78c2c46c7520492db35d51c244fb4f74: Time spent bootstrapping tablet: real 0.113s user 0.008s sys 0.000s
I20260130 02:57:55.563091 19626 raft_consensus.cc:359] T 3b8cdac4bf4b46efa26236fc930eb9a6 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:57:55.563891 19626 raft_consensus.cc:385] T 3b8cdac4bf4b46efa26236fc930eb9a6 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:57:55.564150 19626 raft_consensus.cc:740] T 3b8cdac4bf4b46efa26236fc930eb9a6 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78c2c46c7520492db35d51c244fb4f74, State: Initialized, Role: FOLLOWER
I20260130 02:57:55.565402 19527 raft_consensus.cc:2468] T a46702bef0e6442d85e4ca49a5f7cfbb P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 371a36d219164974abdf848841528152 in term 1.
I20260130 02:57:55.565397 19380 raft_consensus.cc:2468] T f2d01bfdc9c4494386bd02bed655fe63 P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d870164263bb44b6b780a557daea3656 in term 1.
I20260130 02:57:55.566331 19377 raft_consensus.cc:2468] T a46702bef0e6442d85e4ca49a5f7cfbb P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 371a36d219164974abdf848841528152 in term 1.
I20260130 02:57:55.567873 19159 leader_election.cc:304] T a46702bef0e6442d85e4ca49a5f7cfbb P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 371a36d219164974abdf848841528152, 78c2c46c7520492db35d51c244fb4f74; no voters:
I20260130 02:57:55.569511 19225 raft_consensus.cc:2468] T 38a7a10e36874183bf9fe55a38d829f6 P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d870164263bb44b6b780a557daea3656 in term 1.
I20260130 02:57:55.571677 19205 tablet_service.cc:1505] Processing CreateTablet for tablet 30d8136b5ea742bca20ee46b5634d04e (DEFAULT_TABLE table=default.Table_4a74769a488b4760b12165bbffa7cac8 [id=d4c5313ba6e443e0af44b805d21617a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:57:55.571962 19689 tablet.cc:1722] T 06b31d020b864c418476e6dcc1e4eafb P d870164263bb44b6b780a557daea3656: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20260130 02:57:55.572584 19635 raft_consensus.cc:2804] T a46702bef0e6442d85e4ca49a5f7cfbb P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Leader election won for term 1
I20260130 02:57:55.572759 19205 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 30d8136b5ea742bca20ee46b5634d04e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:57:55.573035 19635 raft_consensus.cc:697] T a46702bef0e6442d85e4ca49a5f7cfbb P 371a36d219164974abdf848841528152 [term 1 LEADER]: Becoming Leader. State: Replica: 371a36d219164974abdf848841528152, State: Running, Role: LEADER
I20260130 02:57:55.573767 19635 consensus_queue.cc:237] T a46702bef0e6442d85e4ca49a5f7cfbb P 371a36d219164974abdf848841528152 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } }
I20260130 02:57:55.564771 19626 consensus_queue.cc:260] T 3b8cdac4bf4b46efa26236fc930eb9a6 P 78c2c46c7520492db35d51c244fb4f74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:57:55.577411 19626 ts_tablet_manager.cc:1434] T 3b8cdac4bf4b46efa26236fc930eb9a6 P 78c2c46c7520492db35d51c244fb4f74: Time spent starting tablet: real 0.020s user 0.005s sys 0.001s
I20260130 02:57:55.585074 19712 tablet.cc:1722] T 06b31d020b864c418476e6dcc1e4eafb P 371a36d219164974abdf848841528152: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20260130 02:57:55.588804 19360 tablet_service.cc:1505] Processing CreateTablet for tablet 01db06fcfffe4b259ac4240c0df38047 (DEFAULT_TABLE table=default.Table_e67bd1859b5846f386097cf33c5f1bcb [id=952ccb90477f47e4a352623247a8a6f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:57:55.589135 19506 tablet_service.cc:1505] Processing CreateTablet for tablet 01db06fcfffe4b259ac4240c0df38047 (DEFAULT_TABLE table=default.Table_e67bd1859b5846f386097cf33c5f1bcb [id=952ccb90477f47e4a352623247a8a6f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:57:55.590499 18860 catalog_manager.cc:4985] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): tablet 79e28c3a566c4521ac2453bbaa3567ed (table default.Table_b8f4e9cf27f44a479e1adad0b7ce4e33 [id=26422816af0c430880cb44e1b0cce534]) successfully deleted
I20260130 02:57:55.591526 19360 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 01db06fcfffe4b259ac4240c0df38047. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:57:55.592803 19462 leader_election.cc:304] T f2d01bfdc9c4494386bd02bed655fe63 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 78c2c46c7520492db35d51c244fb4f74, d870164263bb44b6b780a557daea3656; no voters: 371a36d219164974abdf848841528152
I20260130 02:57:55.595459 19677 raft_consensus.cc:2804] T f2d01bfdc9c4494386bd02bed655fe63 P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Leader election won for term 1
I20260130 02:57:55.590728 19626 tablet_bootstrap.cc:492] T 878f9346d3044d8d8b15c61f86869212 P 78c2c46c7520492db35d51c244fb4f74: Bootstrap starting.
I20260130 02:57:55.607415 19677 raft_consensus.cc:697] T f2d01bfdc9c4494386bd02bed655fe63 P d870164263bb44b6b780a557daea3656 [term 1 LEADER]: Becoming Leader. State: Replica: d870164263bb44b6b780a557daea3656, State: Running, Role: LEADER
I20260130 02:57:55.608182 19677 consensus_queue.cc:237] T f2d01bfdc9c4494386bd02bed655fe63 P d870164263bb44b6b780a557daea3656 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:57:55.623023 19626 tablet_bootstrap.cc:654] T 878f9346d3044d8d8b15c61f86869212 P 78c2c46c7520492db35d51c244fb4f74: Neither blocks nor log segments found. Creating new log.
I20260130 02:57:55.632758 19505 tablet_service.cc:1505] Processing CreateTablet for tablet 30d8136b5ea742bca20ee46b5634d04e (DEFAULT_TABLE table=default.Table_4a74769a488b4760b12165bbffa7cac8 [id=d4c5313ba6e443e0af44b805d21617a4]), partition=RANGE (key) PARTITION UNBOUNDED
W20260130 02:57:55.635416 18860 consensus_peers.cc:597] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4 -> Peer 3749baa287ed4f249329a74071781edd (127.18.100.124:34629): Couldn't send request to peer 3749baa287ed4f249329a74071781edd. Status: Network error: Client connection negotiation failed: client connection to 127.18.100.124:34629: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260130 02:57:55.639698 18875 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50284:
name: "default.Table_6eb0a92b1926487cbf0b978d26d6a7b1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260130 02:57:55.639894 18868 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50284:
name: "default.Table_b1ea64451d1a4b48913ae455ba9382b2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260130 02:57:55.641464 18875 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6eb0a92b1926487cbf0b978d26d6a7b1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260130 02:57:55.641571 18868 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b1ea64451d1a4b48913ae455ba9382b2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260130 02:57:55.642894 18866 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50284:
name: "default.Table_ad6198b7bfdb4ce3835e07f2c8499b72"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260130 02:57:55.644539 18866 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ad6198b7bfdb4ce3835e07f2c8499b72 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260130 02:57:55.647517 19357 tablet_service.cc:1505] Processing CreateTablet for tablet 30d8136b5ea742bca20ee46b5634d04e (DEFAULT_TABLE table=default.Table_4a74769a488b4760b12165bbffa7cac8 [id=d4c5313ba6e443e0af44b805d21617a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:57:55.648623 19357 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 30d8136b5ea742bca20ee46b5634d04e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:57:55.654506 19624 tablet_bootstrap.cc:492] T 93809949599a449ca09b13e9b8bbdef6 P 371a36d219164974abdf848841528152: No bootstrap required, opened a new log
I20260130 02:57:55.662951 19624 ts_tablet_manager.cc:1403] T 93809949599a449ca09b13e9b8bbdef6 P 371a36d219164974abdf848841528152: Time spent bootstrapping tablet: real 0.143s user 0.018s sys 0.002s
I20260130 02:57:55.667065 19506 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 01db06fcfffe4b259ac4240c0df38047. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:57:55.665313 19624 raft_consensus.cc:359] T 93809949599a449ca09b13e9b8bbdef6 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:57:55.667714 19624 raft_consensus.cc:385] T 93809949599a449ca09b13e9b8bbdef6 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:57:55.667969 19624 raft_consensus.cc:740] T 93809949599a449ca09b13e9b8bbdef6 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 371a36d219164974abdf848841528152, State: Initialized, Role: FOLLOWER
I20260130 02:57:55.668533 19624 consensus_queue.cc:260] T 93809949599a449ca09b13e9b8bbdef6 P 371a36d219164974abdf848841528152 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:57:55.668524 19630 raft_consensus.cc:515] T 2806ae64d7324b918397c3c6b72ebcfe P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:57:55.676585 19716 tablet.cc:1722] T 06b31d020b864c418476e6dcc1e4eafb P 78c2c46c7520492db35d51c244fb4f74: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20260130 02:57:55.678803 19505 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 30d8136b5ea742bca20ee46b5634d04e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:57:55.670432 19630 leader_election.cc:290] T 2806ae64d7324b918397c3c6b72ebcfe P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 election: Requested vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767)
I20260130 02:57:55.684530 19626 tablet_bootstrap.cc:492] T 878f9346d3044d8d8b15c61f86869212 P 78c2c46c7520492db35d51c244fb4f74: No bootstrap required, opened a new log
I20260130 02:57:55.684967 19626 ts_tablet_manager.cc:1403] T 878f9346d3044d8d8b15c61f86869212 P 78c2c46c7520492db35d51c244fb4f74: Time spent bootstrapping tablet: real 0.095s user 0.008s sys 0.000s
I20260130 02:57:55.692288 19626 raft_consensus.cc:359] T 878f9346d3044d8d8b15c61f86869212 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:57:55.693017 19626 raft_consensus.cc:385] T 878f9346d3044d8d8b15c61f86869212 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:57:55.693282 19626 raft_consensus.cc:740] T 878f9346d3044d8d8b15c61f86869212 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78c2c46c7520492db35d51c244fb4f74, State: Initialized, Role: FOLLOWER
I20260130 02:57:55.693890 19626 consensus_queue.cc:260] T 878f9346d3044d8d8b15c61f86869212 P 78c2c46c7520492db35d51c244fb4f74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:57:55.696137 19378 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2806ae64d7324b918397c3c6b72ebcfe" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74"
I20260130 02:57:55.696779 19378 raft_consensus.cc:3060] T 2806ae64d7324b918397c3c6b72ebcfe P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Advancing to term 1
W20260130 02:57:55.701998 18869 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f2d01bfdc9c4494386bd02bed655fe63 consensus state has no leader
I20260130 02:57:55.702946 18869 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50284:
table { table_name: "default.Table_1a735ba1c9994f81ab61385e277b300c" } new_table_name: "default.Table_a601373812c04ecf9924ecb3ea056a8c" modify_external_catalogs: true
I20260130 02:57:55.708626 19624 ts_tablet_manager.cc:1434] T 93809949599a449ca09b13e9b8bbdef6 P 371a36d219164974abdf848841528152: Time spent starting tablet: real 0.045s user 0.004s sys 0.001s
I20260130 02:57:55.711037 19626 ts_tablet_manager.cc:1434] T 878f9346d3044d8d8b15c61f86869212 P 78c2c46c7520492db35d51c244fb4f74: Time spent starting tablet: real 0.026s user 0.008s sys 0.000s
I20260130 02:57:55.711875 19626 tablet_bootstrap.cc:492] T 37470ce4356f4542b621e21698551209 P 78c2c46c7520492db35d51c244fb4f74: Bootstrap starting.
I20260130 02:57:55.716881 19626 tablet_bootstrap.cc:654] T 37470ce4356f4542b621e21698551209 P 78c2c46c7520492db35d51c244fb4f74: Neither blocks nor log segments found. Creating new log.
I20260130 02:57:55.722272 18872 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50284:
name: "default.Table_19677b801d1f4a63b22139b9fc80c4ef"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260130 02:57:55.724048 18872 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_19677b801d1f4a63b22139b9fc80c4ef in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260130 02:57:55.725145 19624 tablet_bootstrap.cc:492] T 3b8cdac4bf4b46efa26236fc930eb9a6 P 371a36d219164974abdf848841528152: Bootstrap starting.
I20260130 02:57:55.730053 19624 tablet_bootstrap.cc:654] T 3b8cdac4bf4b46efa26236fc930eb9a6 P 371a36d219164974abdf848841528152: Neither blocks nor log segments found. Creating new log.
I20260130 02:57:55.730520 19225 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2806ae64d7324b918397c3c6b72ebcfe" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152"
I20260130 02:57:55.731081 19225 raft_consensus.cc:3060] T 2806ae64d7324b918397c3c6b72ebcfe P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:57:55.734165 18870 catalog_manager.cc:6199] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4: default.Table_4f2d6e4d9c8e465087d65efc9c2de442 [id=1515db389f284e2f9ed1a3cd0061d0f1] alter complete (version 5)
I20260130 02:57:55.735951 18867 catalog_manager.cc:5941] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4: Sending DeleteTablet for 3 replicas of tablet 06b31d020b864c418476e6dcc1e4eafb
I20260130 02:57:55.736507 19626 tablet_bootstrap.cc:492] T 37470ce4356f4542b621e21698551209 P 78c2c46c7520492db35d51c244fb4f74: No bootstrap required, opened a new log
I20260130 02:57:55.736891 19626 ts_tablet_manager.cc:1403] T 37470ce4356f4542b621e21698551209 P 78c2c46c7520492db35d51c244fb4f74: Time spent bootstrapping tablet: real 0.025s user 0.010s sys 0.001s
I20260130 02:57:55.739065 19626 raft_consensus.cc:359] T 37470ce4356f4542b621e21698551209 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:57:55.739650 19626 raft_consensus.cc:385] T 37470ce4356f4542b621e21698551209 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:57:55.739877 19626 raft_consensus.cc:740] T 37470ce4356f4542b621e21698551209 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78c2c46c7520492db35d51c244fb4f74, State: Initialized, Role: FOLLOWER
I20260130 02:57:55.740407 19626 consensus_queue.cc:260] T 37470ce4356f4542b621e21698551209 P 78c2c46c7520492db35d51c244fb4f74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:57:55.742303 19626 ts_tablet_manager.cc:1434] T 37470ce4356f4542b621e21698551209 P 78c2c46c7520492db35d51c244fb4f74: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20260130 02:57:55.743069 19626 tablet_bootstrap.cc:492] T 01db06fcfffe4b259ac4240c0df38047 P 78c2c46c7520492db35d51c244fb4f74: Bootstrap starting.
I20260130 02:57:55.743244 18873 catalog_manager.cc:5941] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4: Sending DeleteTablet for 3 replicas of tablet 2806ae64d7324b918397c3c6b72ebcfe
I20260130 02:57:55.745038 19203 tablet_service.cc:1552] Processing DeleteTablet for tablet 06b31d020b864c418476e6dcc1e4eafb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:57:55 UTC) from {username='slave'} at 127.0.0.1:42870
I20260130 02:57:55.746948 19205 tablet_service.cc:1552] Processing DeleteTablet for tablet 2806ae64d7324b918397c3c6b72ebcfe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:57:55 UTC) from {username='slave'} at 127.0.0.1:42870
I20260130 02:57:55.750599 18874 catalog_manager.cc:5654] T b078934be4bb4fd1915c691eeea9d00f P 78c2c46c7520492db35d51c244fb4f74 reported cstate change: term changed from 0 to 1, leader changed from <none> to 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66). New cstate: current_term: 1 leader_uuid: "78c2c46c7520492db35d51c244fb4f74" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } health_report { overall_health: UNKNOWN } } }
I20260130 02:57:55.751989 18874 catalog_manager.cc:5746] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767) does not have the latest schema for tablet b078934be4bb4fd1915c691eeea9d00f (table default.Table_fcce594e0ed443158293215c25e4e5c3 [id=720decfb55c24f34983ef636f06e3169]). Expected version 1 got 0
I20260130 02:57:55.760251 19626 tablet_bootstrap.cc:654] T 01db06fcfffe4b259ac4240c0df38047 P 78c2c46c7520492db35d51c244fb4f74: Neither blocks nor log segments found. Creating new log.
I20260130 02:57:55.764577 19378 raft_consensus.cc:2468] T 2806ae64d7324b918397c3c6b72ebcfe P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d870164263bb44b6b780a557daea3656 in term 1.
I20260130 02:57:55.764655 19742 raft_consensus.cc:515] T 93809949599a449ca09b13e9b8bbdef6 P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:57:55.766999 19225 raft_consensus.cc:2468] T 2806ae64d7324b918397c3c6b72ebcfe P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d870164263bb44b6b780a557daea3656 in term 1.
I20260130 02:57:55.768049 19463 leader_election.cc:304] T 2806ae64d7324b918397c3c6b72ebcfe P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 371a36d219164974abdf848841528152, d870164263bb44b6b780a557daea3656; no voters:
I20260130 02:57:55.768414 19223 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "93809949599a449ca09b13e9b8bbdef6" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152"
I20260130 02:57:55.768891 19223 raft_consensus.cc:3060] T 93809949599a449ca09b13e9b8bbdef6 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:57:55.770066 19625 tablet_bootstrap.cc:492] T 878f9346d3044d8d8b15c61f86869212 P d870164263bb44b6b780a557daea3656: Bootstrap starting.
I20260130 02:57:55.774739 19630 raft_consensus.cc:2804] T 2806ae64d7324b918397c3c6b72ebcfe P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Leader election won for term 1
I20260130 02:57:55.775166 19630 raft_consensus.cc:697] T 2806ae64d7324b918397c3c6b72ebcfe P d870164263bb44b6b780a557daea3656 [term 1 LEADER]: Becoming Leader. State: Replica: d870164263bb44b6b780a557daea3656, State: Running, Role: LEADER
I20260130 02:57:55.775882 19630 consensus_queue.cc:237] T 2806ae64d7324b918397c3c6b72ebcfe P d870164263bb44b6b780a557daea3656 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:57:55.777477 19378 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "93809949599a449ca09b13e9b8bbdef6" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74"
I20260130 02:57:55.777993 19378 raft_consensus.cc:3060] T 93809949599a449ca09b13e9b8bbdef6 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:57:55.780646 19624 tablet_bootstrap.cc:492] T 3b8cdac4bf4b46efa26236fc930eb9a6 P 371a36d219164974abdf848841528152: No bootstrap required, opened a new log
I20260130 02:57:55.780609 19677 raft_consensus.cc:493] T 3b8cdac4bf4b46efa26236fc930eb9a6 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:57:55.781039 19624 ts_tablet_manager.cc:1403] T 3b8cdac4bf4b46efa26236fc930eb9a6 P 371a36d219164974abdf848841528152: Time spent bootstrapping tablet: real 0.056s user 0.008s sys 0.002s
I20260130 02:57:55.781033 19677 raft_consensus.cc:515] T 3b8cdac4bf4b46efa26236fc930eb9a6 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:57:55.782801 19677 leader_election.cc:290] T 3b8cdac4bf4b46efa26236fc930eb9a6 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767)
I20260130 02:57:55.783368 19624 raft_consensus.cc:359] T 3b8cdac4bf4b46efa26236fc930eb9a6 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:57:55.783954 19624 raft_consensus.cc:385] T 3b8cdac4bf4b46efa26236fc930eb9a6 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:57:55.784201 19624 raft_consensus.cc:740] T 3b8cdac4bf4b46efa26236fc930eb9a6 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 371a36d219164974abdf848841528152, State: Initialized, Role: FOLLOWER
I20260130 02:57:55.785709 19626 tablet_bootstrap.cc:492] T 01db06fcfffe4b259ac4240c0df38047 P 78c2c46c7520492db35d51c244fb4f74: No bootstrap required, opened a new log
I20260130 02:57:55.785797 19377 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3b8cdac4bf4b46efa26236fc930eb9a6" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74" is_pre_election: true
I20260130 02:57:55.786125 19626 ts_tablet_manager.cc:1403] T 01db06fcfffe4b259ac4240c0df38047 P 78c2c46c7520492db35d51c244fb4f74: Time spent bootstrapping tablet: real 0.043s user 0.006s sys 0.005s
I20260130 02:57:55.786437 19377 raft_consensus.cc:2468] T 3b8cdac4bf4b46efa26236fc930eb9a6 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d870164263bb44b6b780a557daea3656 in term 0.
I20260130 02:57:55.786334 19624 consensus_queue.cc:260] T 3b8cdac4bf4b46efa26236fc930eb9a6 P 371a36d219164974abdf848841528152 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:57:55.787444 19462 leader_election.cc:304] T 3b8cdac4bf4b46efa26236fc930eb9a6 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78c2c46c7520492db35d51c244fb4f74, d870164263bb44b6b780a557daea3656; no voters:
I20260130 02:57:55.788545 19626 raft_consensus.cc:359] T 01db06fcfffe4b259ac4240c0df38047 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:57:55.789161 19626 raft_consensus.cc:385] T 01db06fcfffe4b259ac4240c0df38047 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:57:55.789408 19626 raft_consensus.cc:740] T 01db06fcfffe4b259ac4240c0df38047 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78c2c46c7520492db35d51c244fb4f74, State: Initialized, Role: FOLLOWER
I20260130 02:57:55.789974 19626 consensus_queue.cc:260] T 01db06fcfffe4b259ac4240c0df38047 P 78c2c46c7520492db35d51c244fb4f74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:57:55.790539 19225 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3b8cdac4bf4b46efa26236fc930eb9a6" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152" is_pre_election: true
I20260130 02:57:55.791167 19225 raft_consensus.cc:2468] T 3b8cdac4bf4b46efa26236fc930eb9a6 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d870164263bb44b6b780a557daea3656 in term 0.
I20260130 02:57:55.791440 19626 ts_tablet_manager.cc:1434] T 01db06fcfffe4b259ac4240c0df38047 P 78c2c46c7520492db35d51c244fb4f74: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260130 02:57:55.792369 19360 tablet_service.cc:1552] Processing DeleteTablet for tablet 06b31d020b864c418476e6dcc1e4eafb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:57:55 UTC) from {username='slave'} at 127.0.0.1:60110
I20260130 02:57:55.793510 19507 tablet_service.cc:1552] Processing DeleteTablet for tablet 06b31d020b864c418476e6dcc1e4eafb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:57:55 UTC) from {username='slave'} at 127.0.0.1:59068
I20260130 02:57:55.793622 19357 tablet_service.cc:1552] Processing DeleteTablet for tablet 2806ae64d7324b918397c3c6b72ebcfe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:57:55 UTC) from {username='slave'} at 127.0.0.1:60110
I20260130 02:57:55.794164 19710 tablet_replica.cc:333] T 06b31d020b864c418476e6dcc1e4eafb P d870164263bb44b6b780a557daea3656: stopping tablet replica
I20260130 02:57:55.794996 19710 raft_consensus.cc:2243] T 06b31d020b864c418476e6dcc1e4eafb P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260130 02:57:55.795547 19710 raft_consensus.cc:2272] T 06b31d020b864c418476e6dcc1e4eafb P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260130 02:57:55.801945 19507 tablet_service.cc:1552] Processing DeleteTablet for tablet 2806ae64d7324b918397c3c6b72ebcfe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:57:55 UTC) from {username='slave'} at 127.0.0.1:59068
I20260130 02:57:55.802790 19624 ts_tablet_manager.cc:1434] T 3b8cdac4bf4b46efa26236fc930eb9a6 P 371a36d219164974abdf848841528152: Time spent starting tablet: real 0.021s user 0.007s sys 0.000s
I20260130 02:57:55.804500 19710 ts_tablet_manager.cc:1916] T 06b31d020b864c418476e6dcc1e4eafb P d870164263bb44b6b780a557daea3656: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260130 02:57:55.805394 19679 raft_consensus.cc:493] T f2d01bfdc9c4494386bd02bed655fe63 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:57:55.805830 19679 raft_consensus.cc:515] T f2d01bfdc9c4494386bd02bed655fe63 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:57:55.807595 19679 leader_election.cc:290] T f2d01bfdc9c4494386bd02bed655fe63 P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d870164263bb44b6b780a557daea3656 (127.18.100.67:39571), 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767)
I20260130 02:57:55.808993 19742 leader_election.cc:290] T 93809949599a449ca09b13e9b8bbdef6 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 election: Requested vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767)
I20260130 02:57:55.809512 19527 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f2d01bfdc9c4494386bd02bed655fe63" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
I20260130 02:57:55.809520 19742 raft_consensus.cc:2804] T 3b8cdac4bf4b46efa26236fc930eb9a6 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260130 02:57:55.811424 19742 raft_consensus.cc:493] T 3b8cdac4bf4b46efa26236fc930eb9a6 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260130 02:57:55.811712 19742 raft_consensus.cc:3060] T 3b8cdac4bf4b46efa26236fc930eb9a6 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:57:55.815403 19377 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f2d01bfdc9c4494386bd02bed655fe63" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74" is_pre_election: true
I20260130 02:57:55.815714 19624 tablet_bootstrap.cc:492] T 878f9346d3044d8d8b15c61f86869212 P 371a36d219164974abdf848841528152: Bootstrap starting.
I20260130 02:57:55.816183 19377 raft_consensus.cc:2393] T f2d01bfdc9c4494386bd02bed655fe63 P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 371a36d219164974abdf848841528152 in current term 1: Already voted for candidate d870164263bb44b6b780a557daea3656 in this term.
I20260130 02:57:55.821293 19624 tablet_bootstrap.cc:654] T 878f9346d3044d8d8b15c61f86869212 P 371a36d219164974abdf848841528152: Neither blocks nor log segments found. Creating new log.
I20260130 02:57:55.822283 19159 leader_election.cc:304] T f2d01bfdc9c4494386bd02bed655fe63 P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 371a36d219164974abdf848841528152; no voters: 78c2c46c7520492db35d51c244fb4f74, d870164263bb44b6b780a557daea3656
I20260130 02:57:55.822981 19625 tablet_bootstrap.cc:654] T 878f9346d3044d8d8b15c61f86869212 P d870164263bb44b6b780a557daea3656: Neither blocks nor log segments found. Creating new log.
I20260130 02:57:55.823410 19679 raft_consensus.cc:3060] T f2d01bfdc9c4494386bd02bed655fe63 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:57:55.822353 19754 tablet_replica.cc:333] T 06b31d020b864c418476e6dcc1e4eafb P 371a36d219164974abdf848841528152: stopping tablet replica
I20260130 02:57:55.831046 19754 raft_consensus.cc:2243] T 06b31d020b864c418476e6dcc1e4eafb P 371a36d219164974abdf848841528152 [term 1 LEADER]: Raft consensus shutting down.
I20260130 02:57:55.834105 19754 raft_consensus.cc:2272] T 06b31d020b864c418476e6dcc1e4eafb P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260130 02:57:55.835079 19663 raft_consensus.cc:493] T 37470ce4356f4542b621e21698551209 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:57:55.835541 19663 raft_consensus.cc:515] T 37470ce4356f4542b621e21698551209 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:57:55.837471 19663 leader_election.cc:290] T 37470ce4356f4542b621e21698551209 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), d870164263bb44b6b780a557daea3656 (127.18.100.67:39571)
I20260130 02:57:55.839354 19527 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "37470ce4356f4542b621e21698551209" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
W20260130 02:57:55.840783 19313 leader_election.cc:343] T 37470ce4356f4542b621e21698551209 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:57:55.843307 19626 tablet_bootstrap.cc:492] T 30d8136b5ea742bca20ee46b5634d04e P 78c2c46c7520492db35d51c244fb4f74: Bootstrap starting.
I20260130 02:57:55.808040 19758 tablet_replica.cc:333] T 06b31d020b864c418476e6dcc1e4eafb P 78c2c46c7520492db35d51c244fb4f74: stopping tablet replica
I20260130 02:57:55.846951 19225 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "37470ce4356f4542b621e21698551209" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152" is_pre_election: true
I20260130 02:57:55.847295 19758 raft_consensus.cc:2243] T 06b31d020b864c418476e6dcc1e4eafb P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260130 02:57:55.847803 19758 raft_consensus.cc:2272] T 06b31d020b864c418476e6dcc1e4eafb P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260130 02:57:55.850692 19314 leader_election.cc:343] T 37470ce4356f4542b621e21698551209 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:57:55.850805 19754 ts_tablet_manager.cc:1916] T 06b31d020b864c418476e6dcc1e4eafb P 371a36d219164974abdf848841528152: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260130 02:57:55.851112 19314 leader_election.cc:304] T 37470ce4356f4542b621e21698551209 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78c2c46c7520492db35d51c244fb4f74; no voters: 371a36d219164974abdf848841528152, d870164263bb44b6b780a557daea3656
I20260130 02:57:55.854141 19663 raft_consensus.cc:2749] T 37470ce4356f4542b621e21698551209 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260130 02:57:55.856380 19624 tablet_bootstrap.cc:492] T 878f9346d3044d8d8b15c61f86869212 P 371a36d219164974abdf848841528152: No bootstrap required, opened a new log
I20260130 02:57:55.856786 19624 ts_tablet_manager.cc:1403] T 878f9346d3044d8d8b15c61f86869212 P 371a36d219164974abdf848841528152: Time spent bootstrapping tablet: real 0.053s user 0.005s sys 0.011s
I20260130 02:57:55.860139 19758 ts_tablet_manager.cc:1916] T 06b31d020b864c418476e6dcc1e4eafb P 78c2c46c7520492db35d51c244fb4f74: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260130 02:57:55.863672 19624 raft_consensus.cc:359] T 878f9346d3044d8d8b15c61f86869212 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:57:55.865885 19624 raft_consensus.cc:385] T 878f9346d3044d8d8b15c61f86869212 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:57:55.866143 19624 raft_consensus.cc:740] T 878f9346d3044d8d8b15c61f86869212 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 371a36d219164974abdf848841528152, State: Initialized, Role: FOLLOWER
I20260130 02:57:55.866760 19624 consensus_queue.cc:260] T 878f9346d3044d8d8b15c61f86869212 P 371a36d219164974abdf848841528152 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:57:55.870200 19625 tablet_bootstrap.cc:492] T 878f9346d3044d8d8b15c61f86869212 P d870164263bb44b6b780a557daea3656: No bootstrap required, opened a new log
I20260130 02:57:55.870067 19223 raft_consensus.cc:2468] T 93809949599a449ca09b13e9b8bbdef6 P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d870164263bb44b6b780a557daea3656 in term 1.
I20260130 02:57:55.870607 19625 ts_tablet_manager.cc:1403] T 878f9346d3044d8d8b15c61f86869212 P d870164263bb44b6b780a557daea3656: Time spent bootstrapping tablet: real 0.101s user 0.005s sys 0.006s
I20260130 02:57:55.870831 19378 raft_consensus.cc:2468] T 93809949599a449ca09b13e9b8bbdef6 P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d870164263bb44b6b780a557daea3656 in term 1.
I20260130 02:57:55.865178 19626 tablet_bootstrap.cc:654] T 30d8136b5ea742bca20ee46b5634d04e P 78c2c46c7520492db35d51c244fb4f74: Neither blocks nor log segments found. Creating new log.
I20260130 02:57:55.873447 19462 leader_election.cc:304] T 93809949599a449ca09b13e9b8bbdef6 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78c2c46c7520492db35d51c244fb4f74, d870164263bb44b6b780a557daea3656; no voters:
I20260130 02:57:55.878084 19630 raft_consensus.cc:2804] T 93809949599a449ca09b13e9b8bbdef6 P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Leader election won for term 1
I20260130 02:57:55.880420 19624 ts_tablet_manager.cc:1434] T 878f9346d3044d8d8b15c61f86869212 P 371a36d219164974abdf848841528152: Time spent starting tablet: real 0.023s user 0.006s sys 0.001s
I20260130 02:57:55.881260 19624 tablet_bootstrap.cc:492] T 01db06fcfffe4b259ac4240c0df38047 P 371a36d219164974abdf848841528152: Bootstrap starting.
I20260130 02:57:55.881882 19630 raft_consensus.cc:697] T 93809949599a449ca09b13e9b8bbdef6 P d870164263bb44b6b780a557daea3656 [term 1 LEADER]: Becoming Leader. State: Replica: d870164263bb44b6b780a557daea3656, State: Running, Role: LEADER
I20260130 02:57:55.885017 19742 raft_consensus.cc:515] T 3b8cdac4bf4b46efa26236fc930eb9a6 P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:57:55.888346 19742 leader_election.cc:290] T 3b8cdac4bf4b46efa26236fc930eb9a6 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 election: Requested vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767)
I20260130 02:57:55.889197 19223 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3b8cdac4bf4b46efa26236fc930eb9a6" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152"
I20260130 02:57:55.889325 19378 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3b8cdac4bf4b46efa26236fc930eb9a6" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74"
I20260130 02:57:55.889734 19223 raft_consensus.cc:3060] T 3b8cdac4bf4b46efa26236fc930eb9a6 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:57:55.889851 19378 raft_consensus.cc:3060] T 3b8cdac4bf4b46efa26236fc930eb9a6 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:57:55.882763 19630 consensus_queue.cc:237] T 93809949599a449ca09b13e9b8bbdef6 P d870164263bb44b6b780a557daea3656 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:57:55.899933 19626 tablet_bootstrap.cc:492] T 30d8136b5ea742bca20ee46b5634d04e P 78c2c46c7520492db35d51c244fb4f74: No bootstrap required, opened a new log
I20260130 02:57:55.900444 19626 ts_tablet_manager.cc:1403] T 30d8136b5ea742bca20ee46b5634d04e P 78c2c46c7520492db35d51c244fb4f74: Time spent bootstrapping tablet: real 0.057s user 0.007s sys 0.001s
I20260130 02:57:55.903041 19626 raft_consensus.cc:359] T 30d8136b5ea742bca20ee46b5634d04e P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:57:55.903725 19626 raft_consensus.cc:385] T 30d8136b5ea742bca20ee46b5634d04e P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:57:55.904026 19626 raft_consensus.cc:740] T 30d8136b5ea742bca20ee46b5634d04e P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78c2c46c7520492db35d51c244fb4f74, State: Initialized, Role: FOLLOWER
I20260130 02:57:55.904664 19626 consensus_queue.cc:260] T 30d8136b5ea742bca20ee46b5634d04e P 78c2c46c7520492db35d51c244fb4f74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:57:55.906607 19624 tablet_bootstrap.cc:654] T 01db06fcfffe4b259ac4240c0df38047 P 371a36d219164974abdf848841528152: Neither blocks nor log segments found. Creating new log.
I20260130 02:57:55.921556 18867 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50284:
table { table_name: "default.Table_0f26dbdebb8c49f6997d31582bbab3d1" } new_table_name: "default.Table_ebc8b8379c274c18b0a06a919dbe0ee0" modify_external_catalogs: true
I20260130 02:57:55.925644 19624 tablet_bootstrap.cc:492] T 01db06fcfffe4b259ac4240c0df38047 P 371a36d219164974abdf848841528152: No bootstrap required, opened a new log
I20260130 02:57:55.926038 19624 ts_tablet_manager.cc:1403] T 01db06fcfffe4b259ac4240c0df38047 P 371a36d219164974abdf848841528152: Time spent bootstrapping tablet: real 0.045s user 0.004s sys 0.008s
I20260130 02:57:55.927232 19626 ts_tablet_manager.cc:1434] T 30d8136b5ea742bca20ee46b5634d04e P 78c2c46c7520492db35d51c244fb4f74: Time spent starting tablet: real 0.026s user 0.008s sys 0.000s
I20260130 02:57:55.928260 19624 raft_consensus.cc:359] T 01db06fcfffe4b259ac4240c0df38047 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:57:55.928855 19624 raft_consensus.cc:385] T 01db06fcfffe4b259ac4240c0df38047 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:57:55.929077 19624 raft_consensus.cc:740] T 01db06fcfffe4b259ac4240c0df38047 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 371a36d219164974abdf848841528152, State: Initialized, Role: FOLLOWER
I20260130 02:57:55.929610 19624 consensus_queue.cc:260] T 01db06fcfffe4b259ac4240c0df38047 P 371a36d219164974abdf848841528152 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:57:55.937417 19624 ts_tablet_manager.cc:1434] T 01db06fcfffe4b259ac4240c0df38047 P 371a36d219164974abdf848841528152: Time spent starting tablet: real 0.011s user 0.002s sys 0.005s
I20260130 02:57:55.938253 19624 tablet_bootstrap.cc:492] T 30d8136b5ea742bca20ee46b5634d04e P 371a36d219164974abdf848841528152: Bootstrap starting.
I20260130 02:57:55.949517 19624 tablet_bootstrap.cc:654] T 30d8136b5ea742bca20ee46b5634d04e P 371a36d219164974abdf848841528152: Neither blocks nor log segments found. Creating new log.
I20260130 02:57:55.960561 19754 ts_tablet_manager.cc:1929] T 06b31d020b864c418476e6dcc1e4eafb P 371a36d219164974abdf848841528152: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260130 02:57:55.960975 19754 log.cc:1199] T 06b31d020b864c418476e6dcc1e4eafb P 371a36d219164974abdf848841528152: Deleting WAL directory at /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-0/wal/wals/06b31d020b864c418476e6dcc1e4eafb
I20260130 02:57:55.961834 19754 ts_tablet_manager.cc:1950] T 06b31d020b864c418476e6dcc1e4eafb P 371a36d219164974abdf848841528152: Deleting consensus metadata
I20260130 02:57:55.964708 19679 raft_consensus.cc:2749] T f2d01bfdc9c4494386bd02bed655fe63 P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260130 02:57:55.967994 19624 tablet_bootstrap.cc:492] T 30d8136b5ea742bca20ee46b5634d04e P 371a36d219164974abdf848841528152: No bootstrap required, opened a new log
I20260130 02:57:55.968351 19624 ts_tablet_manager.cc:1403] T 30d8136b5ea742bca20ee46b5634d04e P 371a36d219164974abdf848841528152: Time spent bootstrapping tablet: real 0.030s user 0.008s sys 0.000s
I20260130 02:57:55.970767 19624 raft_consensus.cc:359] T 30d8136b5ea742bca20ee46b5634d04e P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:57:55.971349 19624 raft_consensus.cc:385] T 30d8136b5ea742bca20ee46b5634d04e P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:57:55.971585 19624 raft_consensus.cc:740] T 30d8136b5ea742bca20ee46b5634d04e P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 371a36d219164974abdf848841528152, State: Initialized, Role: FOLLOWER
I20260130 02:57:55.972265 19624 consensus_queue.cc:260] T 30d8136b5ea742bca20ee46b5634d04e P 371a36d219164974abdf848841528152 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:57:55.974118 19624 ts_tablet_manager.cc:1434] T 30d8136b5ea742bca20ee46b5634d04e P 371a36d219164974abdf848841528152: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260130 02:57:55.974974 19624 tablet_bootstrap.cc:492] T 37470ce4356f4542b621e21698551209 P 371a36d219164974abdf848841528152: Bootstrap starting.
I20260130 02:57:55.976814 19625 raft_consensus.cc:359] T 878f9346d3044d8d8b15c61f86869212 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:57:55.977419 19625 raft_consensus.cc:385] T 878f9346d3044d8d8b15c61f86869212 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:57:55.977666 19625 raft_consensus.cc:740] T 878f9346d3044d8d8b15c61f86869212 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d870164263bb44b6b780a557daea3656, State: Initialized, Role: FOLLOWER
I20260130 02:57:55.979915 19624 tablet_bootstrap.cc:654] T 37470ce4356f4542b621e21698551209 P 371a36d219164974abdf848841528152: Neither blocks nor log segments found. Creating new log.
I20260130 02:57:55.981226 18861 catalog_manager.cc:4985] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021): tablet 06b31d020b864c418476e6dcc1e4eafb (table default.Table_e9d466c5c67e4df1a06a8eb665da9570 [id=b78b791f363d4c0f906e2a4b21dfacc4]) successfully deleted
I20260130 02:57:55.977030 19754 tablet_replica.cc:333] T 2806ae64d7324b918397c3c6b72ebcfe P 371a36d219164974abdf848841528152: stopping tablet replica
I20260130 02:57:55.983933 19754 raft_consensus.cc:2243] T 2806ae64d7324b918397c3c6b72ebcfe P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260130 02:57:55.984390 19754 raft_consensus.cc:2272] T 2806ae64d7324b918397c3c6b72ebcfe P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260130 02:57:55.987118 19754 ts_tablet_manager.cc:1916] T 2806ae64d7324b918397c3c6b72ebcfe P 371a36d219164974abdf848841528152: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260130 02:57:55.990916 19625 consensus_queue.cc:260] T 878f9346d3044d8d8b15c61f86869212 P d870164263bb44b6b780a557daea3656 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
W20260130 02:57:55.993149 18874 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.18.100.66:59691 (request call id 15) took 846 ms (client timeout 1000 ms). Trace:
I20260130 02:57:55.995327 18868 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50284:
table { table_name: "default.Table_69ffd00165c6422292d9976b685d5d0d" } modify_external_catalogs: true
I20260130 02:57:55.995422 18872 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50284:
table { table_name: "default.Table_ad6198b7bfdb4ce3835e07f2c8499b72" } modify_external_catalogs: true
I20260130 02:57:55.995858 18868 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50284:
table { table_name: "default.Table_69ffd00165c6422292d9976b685d5d0d" } modify_external_catalogs: true
I20260130 02:57:55.995966 18872 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50284:
table { table_name: "default.Table_ad6198b7bfdb4ce3835e07f2c8499b72" } modify_external_catalogs: true
I20260130 02:57:55.997066 19679 raft_consensus.cc:493] T 878f9346d3044d8d8b15c61f86869212 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:57:55.997506 19679 raft_consensus.cc:515] T 878f9346d3044d8d8b15c61f86869212 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:57:56.000126 19377 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "878f9346d3044d8d8b15c61f86869212" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74" is_pre_election: true
I20260130 02:57:56.000735 19377 raft_consensus.cc:2468] T 878f9346d3044d8d8b15c61f86869212 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 371a36d219164974abdf848841528152 in term 0.
I20260130 02:57:56.002854 19527 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "878f9346d3044d8d8b15c61f86869212" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
I20260130 02:57:56.003436 19527 raft_consensus.cc:2468] T 878f9346d3044d8d8b15c61f86869212 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 371a36d219164974abdf848841528152 in term 0.
I20260130 02:57:56.004756 19679 leader_election.cc:290] T 878f9346d3044d8d8b15c61f86869212 P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767), d870164263bb44b6b780a557daea3656 (127.18.100.67:39571)
I20260130 02:57:56.005473 19159 leader_election.cc:304] T 878f9346d3044d8d8b15c61f86869212 P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 371a36d219164974abdf848841528152, d870164263bb44b6b780a557daea3656; no voters:
I20260130 02:57:56.012805 19624 tablet_bootstrap.cc:492] T 37470ce4356f4542b621e21698551209 P 371a36d219164974abdf848841528152: No bootstrap required, opened a new log
I20260130 02:57:56.013199 19624 ts_tablet_manager.cc:1403] T 37470ce4356f4542b621e21698551209 P 371a36d219164974abdf848841528152: Time spent bootstrapping tablet: real 0.038s user 0.011s sys 0.000s
I20260130 02:57:56.015403 19624 raft_consensus.cc:359] T 37470ce4356f4542b621e21698551209 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:57:56.012570 19625 ts_tablet_manager.cc:1434] T 878f9346d3044d8d8b15c61f86869212 P d870164263bb44b6b780a557daea3656: Time spent starting tablet: real 0.142s user 0.004s sys 0.001s
I20260130 02:57:56.016003 19624 raft_consensus.cc:385] T 37470ce4356f4542b621e21698551209 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:57:56.016252 19624 raft_consensus.cc:740] T 37470ce4356f4542b621e21698551209 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 371a36d219164974abdf848841528152, State: Initialized, Role: FOLLOWER
I20260130 02:57:56.016630 19625 tablet_bootstrap.cc:492] T 37470ce4356f4542b621e21698551209 P d870164263bb44b6b780a557daea3656: Bootstrap starting.
W20260130 02:57:55.993532 18874 rpcz_store.cc:269] 0130 02:57:55.146441 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:57:55.147674 (+ 1233us) service_pool.cc:225] Handling call
0130 02:57:55.993085 (+845411us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0130 02:57:55.970269 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0130 02:57:55.970674 (+ 405us) write_op.cc:432] Acquiring schema lock in shared mode
0130 02:57:55.970696 (+ 22us) write_op.cc:435] Acquired schema lock
0130 02:57:55.970707 (+ 11us) tablet.cc:660] Decoding operations
0130 02:57:55.970942 (+ 235us) write_op.cc:620] Acquiring the partition lock for write op
0130 02:57:55.970976 (+ 34us) write_op.cc:641] Partition lock acquired for write op
0130 02:57:55.970996 (+ 20us) tablet.cc:683] Acquiring locks for 1 operations
0130 02:57:55.971125 (+ 129us) tablet.cc:699] Row locks acquired
0130 02:57:55.971136 (+ 11us) write_op.cc:260] PREPARE: finished
0130 02:57:55.971228 (+ 92us) write_op.cc:270] Start()
0130 02:57:55.971301 (+ 73us) write_op.cc:276] Timestamp: P: 1769741875971160 usec, L: 0
0130 02:57:55.971314 (+ 13us) op_driver.cc:348] REPLICATION: starting
0130 02:57:55.971745 (+ 431us) log.cc:844] Serialized 532 byte log entry
0130 02:57:55.989825 (+ 18080us) op_driver.cc:464] REPLICATION: finished
0130 02:57:55.990152 (+ 327us) write_op.cc:301] APPLY: starting
0130 02:57:55.990209 (+ 57us) tablet.cc:1365] starting BulkCheckPresence
0130 02:57:55.990310 (+ 101us) tablet.cc:1368] finished BulkCheckPresence
0130 02:57:55.990437 (+ 127us) tablet.cc:1370] starting ApplyRowOperation cycle
0130 02:57:55.990630 (+ 193us) tablet.cc:1381] finished ApplyRowOperation cycle
0130 02:57:55.990801 (+ 171us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0130 02:57:55.990848 (+ 47us) write_op.cc:312] APPLY: finished
0130 02:57:55.991051 (+ 203us) log.cc:844] Serialized 22 byte log entry
0130 02:57:55.991202 (+ 151us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:57:55.991273 (+ 71us) write_op.cc:454] Released schema lock
0130 02:57:55.991307 (+ 34us) write_op.cc:341] FINISH: Updating metrics
0130 02:57:55.991576 (+ 269us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:57:55.991601 (+ 25us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":4702,"reported_tablets":2,"child_traces":[["op",{"apply.queue_time_us":178,"apply.run_cpu_time_us":1381,"apply.run_wall_time_us":1381,"num_ops":1,"prepare.queue_time_us":209184,"prepare.run_cpu_time_us":1715,"prepare.run_wall_time_us":18358,"replication_time_us":18407}]]}
I20260130 02:57:56.016819 19624 consensus_queue.cc:260] T 37470ce4356f4542b621e21698551209 P 371a36d219164974abdf848841528152 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:57:56.019297 19726 raft_consensus.cc:2804] T 878f9346d3044d8d8b15c61f86869212 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260130 02:57:56.019614 19726 raft_consensus.cc:493] T 878f9346d3044d8d8b15c61f86869212 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260130 02:57:56.019918 19726 raft_consensus.cc:3060] T 878f9346d3044d8d8b15c61f86869212 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:57:56.021721 19624 ts_tablet_manager.cc:1434] T 37470ce4356f4542b621e21698551209 P 371a36d219164974abdf848841528152: Time spent starting tablet: real 0.008s user 0.005s sys 0.000s
I20260130 02:57:56.024753 18872 catalog_manager.cc:5936] Not sending DeleteTablet requests; no consensus state for tablet 2db32ad85b9042118dba956ec273873c
I20260130 02:57:56.035828 19625 tablet_bootstrap.cc:654] T 37470ce4356f4542b621e21698551209 P d870164263bb44b6b780a557daea3656: Neither blocks nor log segments found. Creating new log.
I20260130 02:57:56.035673 19378 raft_consensus.cc:2468] T 3b8cdac4bf4b46efa26236fc930eb9a6 P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d870164263bb44b6b780a557daea3656 in term 1.
I20260130 02:57:56.036885 19462 leader_election.cc:304] T 3b8cdac4bf4b46efa26236fc930eb9a6 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78c2c46c7520492db35d51c244fb4f74, d870164263bb44b6b780a557daea3656; no voters:
I20260130 02:57:56.040318 19742 raft_consensus.cc:2804] T 3b8cdac4bf4b46efa26236fc930eb9a6 P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Leader election won for term 1
I20260130 02:57:56.040752 19742 raft_consensus.cc:697] T 3b8cdac4bf4b46efa26236fc930eb9a6 P d870164263bb44b6b780a557daea3656 [term 1 LEADER]: Becoming Leader. State: Replica: d870164263bb44b6b780a557daea3656, State: Running, Role: LEADER
I20260130 02:57:56.041631 19710 ts_tablet_manager.cc:1929] T 06b31d020b864c418476e6dcc1e4eafb P d870164263bb44b6b780a557daea3656: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260130 02:57:56.041463 19742 consensus_queue.cc:237] T 3b8cdac4bf4b46efa26236fc930eb9a6 P d870164263bb44b6b780a557daea3656 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:57:56.065083 19223 raft_consensus.cc:2468] T 3b8cdac4bf4b46efa26236fc930eb9a6 P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d870164263bb44b6b780a557daea3656 in term 1.
I20260130 02:57:56.070029 19758 ts_tablet_manager.cc:1929] T 06b31d020b864c418476e6dcc1e4eafb P 78c2c46c7520492db35d51c244fb4f74: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260130 02:57:56.070429 19758 log.cc:1199] T 06b31d020b864c418476e6dcc1e4eafb P 78c2c46c7520492db35d51c244fb4f74: Deleting WAL directory at /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-1/wal/wals/06b31d020b864c418476e6dcc1e4eafb
I20260130 02:57:56.071918 19758 ts_tablet_manager.cc:1950] T 06b31d020b864c418476e6dcc1e4eafb P 78c2c46c7520492db35d51c244fb4f74: Deleting consensus metadata
I20260130 02:57:56.080154 19679 raft_consensus.cc:493] T 37470ce4356f4542b621e21698551209 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:57:56.080729 19679 raft_consensus.cc:515] T 37470ce4356f4542b621e21698551209 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:57:56.082736 19679 leader_election.cc:290] T 37470ce4356f4542b621e21698551209 P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767), d870164263bb44b6b780a557daea3656 (127.18.100.67:39571)
I20260130 02:57:56.041962 19710 log.cc:1199] T 06b31d020b864c418476e6dcc1e4eafb P d870164263bb44b6b780a557daea3656: Deleting WAL directory at /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-2/wal/wals/06b31d020b864c418476e6dcc1e4eafb
I20260130 02:57:56.087327 19710 ts_tablet_manager.cc:1950] T 06b31d020b864c418476e6dcc1e4eafb P d870164263bb44b6b780a557daea3656: Deleting consensus metadata
I20260130 02:57:56.087556 19378 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "37470ce4356f4542b621e21698551209" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74" is_pre_election: true
I20260130 02:57:56.088265 19378 raft_consensus.cc:2468] T 37470ce4356f4542b621e21698551209 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 371a36d219164974abdf848841528152 in term 0.
I20260130 02:57:56.089700 19159 leader_election.cc:304] T 37470ce4356f4542b621e21698551209 P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 371a36d219164974abdf848841528152, 78c2c46c7520492db35d51c244fb4f74; no voters:
I20260130 02:57:56.090063 19203 tablet_service.cc:1505] Processing CreateTablet for tablet 4035b8a330844dd28af9f3244bf8eb9a (DEFAULT_TABLE table=default.Table_e7b6f8ca4ed143888d55e87c1b157efd [id=53806f69131d48fdb3328e9fbf8209a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:57:56.091141 19203 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4035b8a330844dd28af9f3244bf8eb9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:57:56.091526 19679 raft_consensus.cc:2804] T 37470ce4356f4542b621e21698551209 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260130 02:57:56.091907 19679 raft_consensus.cc:493] T 37470ce4356f4542b621e21698551209 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260130 02:57:56.092216 19679 raft_consensus.cc:3060] T 37470ce4356f4542b621e21698551209 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:57:56.095937 19360 tablet_service.cc:1505] Processing CreateTablet for tablet 4035b8a330844dd28af9f3244bf8eb9a (DEFAULT_TABLE table=default.Table_e7b6f8ca4ed143888d55e87c1b157efd [id=53806f69131d48fdb3328e9fbf8209a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:57:56.096870 19360 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4035b8a330844dd28af9f3244bf8eb9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:57:56.100307 18873 catalog_manager.cc:5654] T f2d01bfdc9c4494386bd02bed655fe63 P d870164263bb44b6b780a557daea3656 reported cstate change: term changed from 0 to 1, leader changed from <none> to d870164263bb44b6b780a557daea3656 (127.18.100.67). New cstate: current_term: 1 leader_uuid: "d870164263bb44b6b780a557daea3656" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } health_report { overall_health: UNKNOWN } } }
I20260130 02:57:56.102059 18873 catalog_manager.cc:5746] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571) does not have the latest schema for tablet f2d01bfdc9c4494386bd02bed655fe63 (table default.Table_14c76bd65820453fb5cade67cf66e752 [id=d128767e303145a885eec3c5f714a27f]). Expected version 1 got 0
I20260130 02:57:56.103945 18873 catalog_manager.cc:5654] T 38a7a10e36874183bf9fe55a38d829f6 P d870164263bb44b6b780a557daea3656 reported cstate change: term changed from 0 to 1, leader changed from <none> to d870164263bb44b6b780a557daea3656 (127.18.100.67). New cstate: current_term: 1 leader_uuid: "d870164263bb44b6b780a557daea3656" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } health_report { overall_health: HEALTHY } } }
I20260130 02:57:56.106681 19663 consensus_queue.cc:1048] T b078934be4bb4fd1915c691eeea9d00f P 78c2c46c7520492db35d51c244fb4f74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260130 02:57:56.116515 18873 catalog_manager.cc:5746] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571) does not have the latest schema for tablet 38a7a10e36874183bf9fe55a38d829f6 (table default.Table_1a735ba1c9994f81ab61385e277b300c [id=27adf356fd004597a54c7a9887dd3ca8]). Expected version 1 got 0
W20260130 02:57:56.117667 18869 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 38a7a10e36874183bf9fe55a38d829f6 consensus state has no leader
I20260130 02:57:56.119010 18868 catalog_manager.cc:5941] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4: Sending DeleteTablet for 3 replicas of tablet c2a9675427754d0db1578637a37440c6
W20260130 02:57:56.119132 18871 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.18.100.65:51021 (request call id 19) took 1088 ms (client timeout 1000 ms). Trace:
I20260130 02:57:56.122349 19505 tablet_service.cc:1552] Processing DeleteTablet for tablet c2a9675427754d0db1578637a37440c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:57:56 UTC) from {username='slave'} at 127.0.0.1:59068
I20260130 02:57:56.124562 19505 tablet_service.cc:1505] Processing CreateTablet for tablet 4035b8a330844dd28af9f3244bf8eb9a (DEFAULT_TABLE table=default.Table_e7b6f8ca4ed143888d55e87c1b157efd [id=53806f69131d48fdb3328e9fbf8209a2]), partition=RANGE (key) PARTITION UNBOUNDED
W20260130 02:57:56.119444 18871 rpcz_store.cc:269] 0130 02:57:55.030796 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:57:55.031162 (+ 366us) service_pool.cc:225] Handling call
0130 02:57:56.119053 (+1087891us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0130 02:57:55.652348 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0130 02:57:55.652725 (+ 377us) write_op.cc:432] Acquiring schema lock in shared mode
0130 02:57:55.652747 (+ 22us) write_op.cc:435] Acquired schema lock
0130 02:57:55.652759 (+ 12us) tablet.cc:660] Decoding operations
0130 02:57:55.652984 (+ 225us) write_op.cc:620] Acquiring the partition lock for write op
0130 02:57:55.653017 (+ 33us) write_op.cc:641] Partition lock acquired for write op
0130 02:57:55.653039 (+ 22us) tablet.cc:683] Acquiring locks for 1 operations
0130 02:57:55.653163 (+ 124us) tablet.cc:699] Row locks acquired
0130 02:57:55.653175 (+ 12us) write_op.cc:260] PREPARE: finished
0130 02:57:55.653269 (+ 94us) write_op.cc:270] Start()
0130 02:57:55.653341 (+ 72us) write_op.cc:276] Timestamp: P: 1769741875653187 usec, L: 0
0130 02:57:55.653355 (+ 14us) op_driver.cc:348] REPLICATION: starting
0130 02:57:55.653787 (+ 432us) log.cc:844] Serialized 532 byte log entry
0130 02:57:55.731792 (+ 78005us) op_driver.cc:464] REPLICATION: finished
0130 02:57:55.733660 (+ 1868us) write_op.cc:301] APPLY: starting
0130 02:57:55.733702 (+ 42us) tablet.cc:1365] starting BulkCheckPresence
0130 02:57:55.733779 (+ 77us) tablet.cc:1368] finished BulkCheckPresence
0130 02:57:55.733856 (+ 77us) tablet.cc:1370] starting ApplyRowOperation cycle
0130 02:57:55.747027 (+ 13171us) tablet.cc:1381] finished ApplyRowOperation cycle
0130 02:57:55.747175 (+ 148us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0130 02:57:55.747215 (+ 40us) write_op.cc:312] APPLY: finished
0130 02:57:55.747453 (+ 238us) log.cc:844] Serialized 22 byte log entry
0130 02:57:55.747606 (+ 153us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:57:55.747656 (+ 50us) write_op.cc:454] Released schema lock
0130 02:57:55.747686 (+ 30us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"reported_tablets":7,"child_traces":[["op",{"apply.queue_time_us":1760,"apply.run_cpu_time_us":1216,"apply.run_wall_time_us":14190,"mutex_wait_us":12989,"num_ops":1,"prepare.queue_time_us":53929,"prepare.run_cpu_time_us":1698,"prepare.run_wall_time_us":49390,"raft.queue_time_us":1896,"raft.run_cpu_time_us":11,"raft.run_wall_time_us":12,"replication_time_us":78368,"thread_start_us":91861,"threads_started":1}]]}
I20260130 02:57:56.128038 19357 tablet_service.cc:1552] Processing DeleteTablet for tablet c2a9675427754d0db1578637a37440c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:57:56 UTC) from {username='slave'} at 127.0.0.1:60110
I20260130 02:57:56.129146 19358 tablet_service.cc:1505] Processing CreateTablet for tablet bac739bf4cac4272975f5587ad9c71ff (DEFAULT_TABLE table=default.Table_235e240c644b42baaf12b08d1b983e8a [id=18ed268eb9ad4450abf9874310ace074]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:57:56.130138 19358 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bac739bf4cac4272975f5587ad9c71ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:57:56.141083 19527 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "37470ce4356f4542b621e21698551209" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
I20260130 02:57:56.142683 19505 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4035b8a330844dd28af9f3244bf8eb9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:57:56.156118 18874 catalog_manager.cc:5746] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767) does not have the latest schema for tablet b078934be4bb4fd1915c691eeea9d00f (table default.Table_fcce594e0ed443158293215c25e4e5c3 [id=720decfb55c24f34983ef636f06e3169]). Expected version 1 got 0
I20260130 02:57:56.165741 19758 tablet_replica.cc:333] T 2806ae64d7324b918397c3c6b72ebcfe P 78c2c46c7520492db35d51c244fb4f74: stopping tablet replica
I20260130 02:57:56.166288 19726 raft_consensus.cc:515] T 878f9346d3044d8d8b15c61f86869212 P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:57:56.168134 19726 leader_election.cc:290] T 878f9346d3044d8d8b15c61f86869212 P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 election: Requested vote from peers 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767), d870164263bb44b6b780a557daea3656 (127.18.100.67:39571)
I20260130 02:57:56.172968 19710 tablet_replica.cc:333] T 2806ae64d7324b918397c3c6b72ebcfe P d870164263bb44b6b780a557daea3656: stopping tablet replica
I20260130 02:57:56.173776 19710 raft_consensus.cc:2243] T 2806ae64d7324b918397c3c6b72ebcfe P d870164263bb44b6b780a557daea3656 [term 1 LEADER]: Raft consensus shutting down.
I20260130 02:57:56.174466 19710 pending_rounds.cc:70] T 2806ae64d7324b918397c3c6b72ebcfe P d870164263bb44b6b780a557daea3656: Trying to abort 1 pending ops.
I20260130 02:57:56.174715 19710 pending_rounds.cc:77] T 2806ae64d7324b918397c3c6b72ebcfe P d870164263bb44b6b780a557daea3656: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7248862723191869440 op_type: NO_OP noop_request { }
I20260130 02:57:56.175055 19710 raft_consensus.cc:2889] T 2806ae64d7324b918397c3c6b72ebcfe P d870164263bb44b6b780a557daea3656 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20260130 02:57:56.175367 19710 raft_consensus.cc:2272] T 2806ae64d7324b918397c3c6b72ebcfe P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260130 02:57:56.181643 19758 raft_consensus.cc:2243] T 2806ae64d7324b918397c3c6b72ebcfe P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260130 02:57:56.182225 19758 raft_consensus.cc:2272] T 2806ae64d7324b918397c3c6b72ebcfe P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260130 02:57:56.184613 19758 ts_tablet_manager.cc:1916] T 2806ae64d7324b918397c3c6b72ebcfe P 78c2c46c7520492db35d51c244fb4f74: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260130 02:57:56.194130 19663 consensus_queue.cc:1048] T b078934be4bb4fd1915c691eeea9d00f P 78c2c46c7520492db35d51c244fb4f74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260130 02:57:56.159519 18871 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50284:
name: "default.Table_495035e92963454c8690f28fb4557c00"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260130 02:57:56.162981 18860 catalog_manager.cc:4985] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): tablet 06b31d020b864c418476e6dcc1e4eafb (table default.Table_e9d466c5c67e4df1a06a8eb665da9570 [id=b78b791f363d4c0f906e2a4b21dfacc4]) successfully deleted
I20260130 02:57:56.170893 18869 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50284:
name: "default.Table_298b1af220674b6ab20d3652e8a637a1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260130 02:57:56.196661 18860 catalog_manager.cc:4985] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767): tablet 06b31d020b864c418476e6dcc1e4eafb (table default.Table_e9d466c5c67e4df1a06a8eb665da9570 [id=b78b791f363d4c0f906e2a4b21dfacc4]) successfully deleted
W20260130 02:57:56.197059 18869 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_298b1af220674b6ab20d3652e8a637a1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260130 02:57:56.197515 18872 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50284:
name: "default.Table_85611716686c4097bacb64df677d5793"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260130 02:57:56.199088 18872 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_85611716686c4097bacb64df677d5793 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260130 02:57:56.200047 19357 tablet_service.cc:1505] Processing CreateTablet for tablet d4f2b54930cc4f5c825562e3e0a79f4d (DEFAULT_TABLE table=default.Table_50c3bb8b3ede494ba9b9f826a2adba72 [id=17150cb73e374ff9a1e92c43e0c3f2c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:57:56.201120 19357 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d4f2b54930cc4f5c825562e3e0a79f4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:57:56.203840 19507 tablet_service.cc:1505] Processing CreateTablet for tablet bac739bf4cac4272975f5587ad9c71ff (DEFAULT_TABLE table=default.Table_235e240c644b42baaf12b08d1b983e8a [id=18ed268eb9ad4450abf9874310ace074]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:57:56.177016 18868 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50284:
name: "default.Table_b87dd1ee48f143538e487968bbea94bb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260130 02:57:56.204764 19507 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bac739bf4cac4272975f5587ad9c71ff. 1 dirs total, 0 dirs full, 0 dirs failed
W20260130 02:57:56.205277 18868 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b87dd1ee48f143538e487968bbea94bb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260130 02:57:56.207278 19508 tablet_service.cc:1505] Processing CreateTablet for tablet d4f2b54930cc4f5c825562e3e0a79f4d (DEFAULT_TABLE table=default.Table_50c3bb8b3ede494ba9b9f826a2adba72 [id=17150cb73e374ff9a1e92c43e0c3f2c1]), partition=RANGE (key) PARTITION UNBOUNDED
W20260130 02:57:56.207589 18871 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_495035e92963454c8690f28fb4557c00 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260130 02:57:56.208302 19508 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d4f2b54930cc4f5c825562e3e0a79f4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:57:56.228197 18875 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50284:
table { table_name: "default.Table_e67bd1859b5846f386097cf33c5f1bcb" } modify_external_catalogs: true
I20260130 02:57:56.228766 18875 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50284:
table { table_name: "default.Table_e67bd1859b5846f386097cf33c5f1bcb" } modify_external_catalogs: true
I20260130 02:57:56.233963 19626 tablet_bootstrap.cc:492] T 4035b8a330844dd28af9f3244bf8eb9a P 78c2c46c7520492db35d51c244fb4f74: Bootstrap starting.
I20260130 02:57:56.235062 18874 catalog_manager.cc:5746] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767) does not have the latest schema for tablet 37470ce4356f4542b621e21698551209 (table default.Table_007e969f70d345a7bcb3dfbaabbc025d [id=77d4fa8705a3419cb0a84d3ca1bb4ce3]). Expected version 1 got 0
I20260130 02:57:56.236183 18874 catalog_manager.cc:5746] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767) does not have the latest schema for tablet f2d01bfdc9c4494386bd02bed655fe63 (table default.Table_14c76bd65820453fb5cade67cf66e752 [id=d128767e303145a885eec3c5f714a27f]). Expected version 1 got 0
I20260130 02:57:56.236678 19710 ts_tablet_manager.cc:1916] T 2806ae64d7324b918397c3c6b72ebcfe P d870164263bb44b6b780a557daea3656: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260130 02:57:56.239943 19626 tablet_bootstrap.cc:654] T 4035b8a330844dd28af9f3244bf8eb9a P 78c2c46c7520492db35d51c244fb4f74: Neither blocks nor log segments found. Creating new log.
I20260130 02:57:56.252350 19626 tablet_bootstrap.cc:492] T 4035b8a330844dd28af9f3244bf8eb9a P 78c2c46c7520492db35d51c244fb4f74: No bootstrap required, opened a new log
I20260130 02:57:56.252758 19626 ts_tablet_manager.cc:1403] T 4035b8a330844dd28af9f3244bf8eb9a P 78c2c46c7520492db35d51c244fb4f74: Time spent bootstrapping tablet: real 0.019s user 0.006s sys 0.006s
I20260130 02:57:56.254458 19663 raft_consensus.cc:493] T 30d8136b5ea742bca20ee46b5634d04e P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:57:56.254909 19663 raft_consensus.cc:515] T 30d8136b5ea742bca20ee46b5634d04e P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:57:56.257365 19663 leader_election.cc:290] T 30d8136b5ea742bca20ee46b5634d04e P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), d870164263bb44b6b780a557daea3656 (127.18.100.67:39571)
I20260130 02:57:56.258477 19528 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "30d8136b5ea742bca20ee46b5634d04e" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
W20260130 02:57:56.259774 19313 leader_election.cc:343] T 30d8136b5ea742bca20ee46b5634d04e P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:57:56.264606 19378 raft_consensus.cc:1275] T 38a7a10e36874183bf9fe55a38d829f6 P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Refusing update from remote peer d870164263bb44b6b780a557daea3656: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260130 02:57:56.266047 19630 consensus_queue.cc:1048] T 38a7a10e36874183bf9fe55a38d829f6 P d870164263bb44b6b780a557daea3656 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260130 02:57:56.266467 19506 tablet_service.cc:1505] Processing CreateTablet for tablet ceda64471a92476f9ba183d2beab30d1 (DEFAULT_TABLE table=default.Table_ab500271b22e4a789171a24c55b79fce [id=4f59d133b0cd4773bb012145927a92a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:57:56.267553 19506 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ceda64471a92476f9ba183d2beab30d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:57:56.267848 19754 ts_tablet_manager.cc:1929] T 2806ae64d7324b918397c3c6b72ebcfe P 371a36d219164974abdf848841528152: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260130 02:57:56.268214 19754 log.cc:1199] T 2806ae64d7324b918397c3c6b72ebcfe P 371a36d219164974abdf848841528152: Deleting WAL directory at /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-0/wal/wals/2806ae64d7324b918397c3c6b72ebcfe
I20260130 02:57:56.277614 19625 tablet_bootstrap.cc:492] T 37470ce4356f4542b621e21698551209 P d870164263bb44b6b780a557daea3656: No bootstrap required, opened a new log
I20260130 02:57:56.278028 19625 ts_tablet_manager.cc:1403] T 37470ce4356f4542b621e21698551209 P d870164263bb44b6b780a557daea3656: Time spent bootstrapping tablet: real 0.262s user 0.004s sys 0.004s
I20260130 02:57:56.280447 19625 raft_consensus.cc:359] T 37470ce4356f4542b621e21698551209 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:57:56.281052 19625 raft_consensus.cc:385] T 37470ce4356f4542b621e21698551209 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:57:56.281291 19625 raft_consensus.cc:740] T 37470ce4356f4542b621e21698551209 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d870164263bb44b6b780a557daea3656, State: Initialized, Role: FOLLOWER
I20260130 02:57:56.282740 19625 consensus_queue.cc:260] T 37470ce4356f4542b621e21698551209 P d870164263bb44b6b780a557daea3656 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
W20260130 02:57:56.289863 19159 leader_election.cc:343] T 37470ce4356f4542b621e21698551209 P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:57:56.291699 19528 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "878f9346d3044d8d8b15c61f86869212" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656"
I20260130 02:57:56.292256 19528 raft_consensus.cc:3060] T 878f9346d3044d8d8b15c61f86869212 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:57:56.293788 19202 tablet_service.cc:1552] Processing DeleteTablet for tablet c2a9675427754d0db1578637a37440c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:57:56 UTC) from {username='slave'} at 127.0.0.1:42870
I20260130 02:57:56.294974 19377 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "878f9346d3044d8d8b15c61f86869212" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74"
I20260130 02:57:56.295516 19377 raft_consensus.cc:3060] T 878f9346d3044d8d8b15c61f86869212 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:57:56.295722 19204 tablet_service.cc:1505] Processing CreateTablet for tablet bac739bf4cac4272975f5587ad9c71ff (DEFAULT_TABLE table=default.Table_235e240c644b42baaf12b08d1b983e8a [id=18ed268eb9ad4450abf9874310ace074]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:57:56.297325 19204 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bac739bf4cac4272975f5587ad9c71ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:57:56.299774 19205 tablet_service.cc:1552] Processing DeleteTablet for tablet 79e28c3a566c4521ac2453bbaa3567ed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:57:53 UTC) from {username='slave'} at 127.0.0.1:42870
I20260130 02:57:56.300966 19679 raft_consensus.cc:515] T 37470ce4356f4542b621e21698551209 P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:57:56.303596 19679 leader_election.cc:290] T 37470ce4356f4542b621e21698551209 P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 election: Requested vote from peers 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767), d870164263bb44b6b780a557daea3656 (127.18.100.67:39571)
I20260130 02:57:56.305245 19380 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "37470ce4356f4542b621e21698551209" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74"
I20260130 02:57:56.305856 19380 raft_consensus.cc:3060] T 37470ce4356f4542b621e21698551209 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:57:56.312914 19526 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "37470ce4356f4542b621e21698551209" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656"
I20260130 02:57:56.313483 19526 raft_consensus.cc:3060] T 37470ce4356f4542b621e21698551209 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:57:56.313359 19205 tablet_service.cc:1505] Processing CreateTablet for tablet d4f2b54930cc4f5c825562e3e0a79f4d (DEFAULT_TABLE table=default.Table_50c3bb8b3ede494ba9b9f826a2adba72 [id=17150cb73e374ff9a1e92c43e0c3f2c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:57:56.314393 19205 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d4f2b54930cc4f5c825562e3e0a79f4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:57:56.321777 19201 tablet_service.cc:1505] Processing CreateTablet for tablet ceda64471a92476f9ba183d2beab30d1 (DEFAULT_TABLE table=default.Table_ab500271b22e4a789171a24c55b79fce [id=4f59d133b0cd4773bb012145927a92a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:57:56.323030 19201 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ceda64471a92476f9ba183d2beab30d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:57:56.325554 19223 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "30d8136b5ea742bca20ee46b5634d04e" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152" is_pre_election: true
I20260130 02:57:56.326261 19223 raft_consensus.cc:2468] T 30d8136b5ea742bca20ee46b5634d04e P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78c2c46c7520492db35d51c244fb4f74 in term 0.
I20260130 02:57:56.311395 19754 ts_tablet_manager.cc:1950] T 2806ae64d7324b918397c3c6b72ebcfe P 371a36d219164974abdf848841528152: Deleting consensus metadata
I20260130 02:57:56.333551 19225 raft_consensus.cc:1275] T 38a7a10e36874183bf9fe55a38d829f6 P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Refusing update from remote peer d870164263bb44b6b780a557daea3656: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260130 02:57:56.335176 19314 leader_election.cc:304] T 30d8136b5ea742bca20ee46b5634d04e P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 371a36d219164974abdf848841528152, 78c2c46c7520492db35d51c244fb4f74; no voters: d870164263bb44b6b780a557daea3656
I20260130 02:57:56.339030 19782 raft_consensus.cc:2804] T 30d8136b5ea742bca20ee46b5634d04e P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260130 02:57:56.339485 19782 raft_consensus.cc:493] T 30d8136b5ea742bca20ee46b5634d04e P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260130 02:57:56.339926 19782 raft_consensus.cc:3060] T 30d8136b5ea742bca20ee46b5634d04e P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:57:56.340471 18866 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50284:
table { table_name: "default.Table_4a74769a488b4760b12165bbffa7cac8" } modify_external_catalogs: true
I20260130 02:57:56.340719 18861 catalog_manager.cc:4985] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021): tablet 2806ae64d7324b918397c3c6b72ebcfe (table default.Table_d1b81d6a8e704777b0fa35cf23dc91e0 [id=78a1af9c84574331a9112fdea3f67fcf]) successfully deleted
I20260130 02:57:56.341009 18866 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50284:
table { table_name: "default.Table_4a74769a488b4760b12165bbffa7cac8" } modify_external_catalogs: true
I20260130 02:57:56.341966 19625 ts_tablet_manager.cc:1434] T 37470ce4356f4542b621e21698551209 P d870164263bb44b6b780a557daea3656: Time spent starting tablet: real 0.064s user 0.006s sys 0.000s
I20260130 02:57:56.342871 19625 tablet_bootstrap.cc:492] T 01db06fcfffe4b259ac4240c0df38047 P d870164263bb44b6b780a557daea3656: Bootstrap starting.
I20260130 02:57:56.343240 19742 consensus_queue.cc:1048] T 38a7a10e36874183bf9fe55a38d829f6 P d870164263bb44b6b780a557daea3656 [LEADER]: Connected to new peer: Peer: permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260130 02:57:56.344447 18867 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50284:
table { table_name: "default.Table_b1ea64451d1a4b48913ae455ba9382b2" } new_table_name: "default.Table_da3d581b15a94840898777f74611994e" modify_external_catalogs: true
I20260130 02:57:56.347294 19626 raft_consensus.cc:359] T 4035b8a330844dd28af9f3244bf8eb9a P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:57:56.347960 19626 raft_consensus.cc:385] T 4035b8a330844dd28af9f3244bf8eb9a P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:57:56.348217 19626 raft_consensus.cc:740] T 4035b8a330844dd28af9f3244bf8eb9a P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78c2c46c7520492db35d51c244fb4f74, State: Initialized, Role: FOLLOWER
I20260130 02:57:56.348780 19626 consensus_queue.cc:260] T 4035b8a330844dd28af9f3244bf8eb9a P 78c2c46c7520492db35d51c244fb4f74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:57:56.363405 19526 raft_consensus.cc:2468] T 37470ce4356f4542b621e21698551209 P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 371a36d219164974abdf848841528152 in term 1.
I20260130 02:57:56.363415 19380 raft_consensus.cc:2468] T 37470ce4356f4542b621e21698551209 P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 371a36d219164974abdf848841528152 in term 1.
I20260130 02:57:56.366003 19377 raft_consensus.cc:2468] T 878f9346d3044d8d8b15c61f86869212 P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 371a36d219164974abdf848841528152 in term 1.
I20260130 02:57:56.367416 19785 tablet.cc:1722] T 927c90b84d7449efa36bc5e2035bf0da P 78c2c46c7520492db35d51c244fb4f74: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260130 02:57:56.373504 19783 raft_consensus.cc:493] T 01db06fcfffe4b259ac4240c0df38047 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:57:56.373965 19783 raft_consensus.cc:515] T 01db06fcfffe4b259ac4240c0df38047 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:57:56.377480 19626 ts_tablet_manager.cc:1434] T 4035b8a330844dd28af9f3244bf8eb9a P 78c2c46c7520492db35d51c244fb4f74: Time spent starting tablet: real 0.124s user 0.006s sys 0.001s
I20260130 02:57:56.382391 19626 tablet_bootstrap.cc:492] T d4f2b54930cc4f5c825562e3e0a79f4d P 78c2c46c7520492db35d51c244fb4f74: Bootstrap starting.
I20260130 02:57:56.386682 19783 leader_election.cc:290] T 01db06fcfffe4b259ac4240c0df38047 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), d870164263bb44b6b780a557daea3656 (127.18.100.67:39571)
I20260130 02:57:56.390451 19625 tablet_bootstrap.cc:654] T 01db06fcfffe4b259ac4240c0df38047 P d870164263bb44b6b780a557daea3656: Neither blocks nor log segments found. Creating new log.
I20260130 02:57:56.393523 19788 tablet.cc:1722] T 38a7a10e36874183bf9fe55a38d829f6 P d870164263bb44b6b780a557daea3656: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20260130 02:57:56.330406 19754 tablet_replica.cc:333] T c2a9675427754d0db1578637a37440c6 P 371a36d219164974abdf848841528152: stopping tablet replica
I20260130 02:57:56.400928 19626 tablet_bootstrap.cc:654] T d4f2b54930cc4f5c825562e3e0a79f4d P 78c2c46c7520492db35d51c244fb4f74: Neither blocks nor log segments found. Creating new log.
I20260130 02:57:56.410519 19360 tablet_service.cc:1505] Processing CreateTablet for tablet ceda64471a92476f9ba183d2beab30d1 (DEFAULT_TABLE table=default.Table_ab500271b22e4a789171a24c55b79fce [id=4f59d133b0cd4773bb012145927a92a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:57:56.411469 19791 tablet.cc:1722] T 38a7a10e36874183bf9fe55a38d829f6 P 78c2c46c7520492db35d51c244fb4f74: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20260130 02:57:56.420107 19677 consensus_queue.cc:1048] T c2a9675427754d0db1578637a37440c6 P d870164263bb44b6b780a557daea3656 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260130 02:57:56.422084 19526 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "01db06fcfffe4b259ac4240c0df38047" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
I20260130 02:57:56.425328 19224 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "01db06fcfffe4b259ac4240c0df38047" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152" is_pre_election: true
I20260130 02:57:56.426134 19224 raft_consensus.cc:2468] T 01db06fcfffe4b259ac4240c0df38047 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78c2c46c7520492db35d51c244fb4f74 in term 0.
I20260130 02:57:56.427342 19314 leader_election.cc:304] T 01db06fcfffe4b259ac4240c0df38047 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 371a36d219164974abdf848841528152, 78c2c46c7520492db35d51c244fb4f74; no voters:
I20260130 02:57:56.428097 19783 raft_consensus.cc:2804] T 01db06fcfffe4b259ac4240c0df38047 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260130 02:57:56.428419 19783 raft_consensus.cc:493] T 01db06fcfffe4b259ac4240c0df38047 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260130 02:57:56.428725 19783 raft_consensus.cc:3060] T 01db06fcfffe4b259ac4240c0df38047 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:57:56.430961 19626 tablet_bootstrap.cc:492] T d4f2b54930cc4f5c825562e3e0a79f4d P 78c2c46c7520492db35d51c244fb4f74: No bootstrap required, opened a new log
I20260130 02:57:56.431382 19626 ts_tablet_manager.cc:1403] T d4f2b54930cc4f5c825562e3e0a79f4d P 78c2c46c7520492db35d51c244fb4f74: Time spent bootstrapping tablet: real 0.049s user 0.003s sys 0.007s
W20260130 02:57:56.435367 19313 leader_election.cc:343] T 01db06fcfffe4b259ac4240c0df38047 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:57:56.444808 19794 tablet.cc:1722] T 927c90b84d7449efa36bc5e2035bf0da P d870164263bb44b6b780a557daea3656: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260130 02:57:56.443634 19626 raft_consensus.cc:359] T d4f2b54930cc4f5c825562e3e0a79f4d P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:57:56.452486 19626 raft_consensus.cc:385] T d4f2b54930cc4f5c825562e3e0a79f4d P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:57:56.453598 19626 raft_consensus.cc:740] T d4f2b54930cc4f5c825562e3e0a79f4d P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78c2c46c7520492db35d51c244fb4f74, State: Initialized, Role: FOLLOWER
I20260130 02:57:56.455541 19625 tablet_bootstrap.cc:492] T 01db06fcfffe4b259ac4240c0df38047 P d870164263bb44b6b780a557daea3656: No bootstrap required, opened a new log
I20260130 02:57:56.455960 19625 ts_tablet_manager.cc:1403] T 01db06fcfffe4b259ac4240c0df38047 P d870164263bb44b6b780a557daea3656: Time spent bootstrapping tablet: real 0.113s user 0.008s sys 0.000s
I20260130 02:57:56.458238 19625 raft_consensus.cc:359] T 01db06fcfffe4b259ac4240c0df38047 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:57:56.459919 19625 raft_consensus.cc:385] T 01db06fcfffe4b259ac4240c0df38047 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:57:56.460175 19625 raft_consensus.cc:740] T 01db06fcfffe4b259ac4240c0df38047 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d870164263bb44b6b780a557daea3656, State: Initialized, Role: FOLLOWER
I20260130 02:57:56.460460 19159 leader_election.cc:304] T 37470ce4356f4542b621e21698551209 P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 371a36d219164974abdf848841528152, 78c2c46c7520492db35d51c244fb4f74; no voters:
I20260130 02:57:56.461022 19625 consensus_queue.cc:260] T 01db06fcfffe4b259ac4240c0df38047 P d870164263bb44b6b780a557daea3656 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
W20260130 02:57:56.462556 19463 consensus_peers.cc:597] T c2a9675427754d0db1578637a37440c6 P d870164263bb44b6b780a557daea3656 -> Peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Couldn't send request to peer 371a36d219164974abdf848841528152. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: STOPPING. This is attempt 1: this message will repeat every 5th retry.
I20260130 02:57:56.462945 19625 ts_tablet_manager.cc:1434] T 01db06fcfffe4b259ac4240c0df38047 P d870164263bb44b6b780a557daea3656: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260130 02:57:56.464174 19625 tablet_bootstrap.cc:492] T 30d8136b5ea742bca20ee46b5634d04e P d870164263bb44b6b780a557daea3656: Bootstrap starting.
I20260130 02:57:56.466833 19635 raft_consensus.cc:2804] T 37470ce4356f4542b621e21698551209 P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Leader election won for term 1
I20260130 02:57:56.467272 19635 raft_consensus.cc:697] T 37470ce4356f4542b621e21698551209 P 371a36d219164974abdf848841528152 [term 1 LEADER]: Becoming Leader. State: Replica: 371a36d219164974abdf848841528152, State: Running, Role: LEADER
I20260130 02:57:56.468079 19635 consensus_queue.cc:237] T 37470ce4356f4542b621e21698551209 P 371a36d219164974abdf848841528152 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:57:56.468079 19626 consensus_queue.cc:260] T d4f2b54930cc4f5c825562e3e0a79f4d P 78c2c46c7520492db35d51c244fb4f74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:57:56.478161 19742 raft_consensus.cc:493] T 01db06fcfffe4b259ac4240c0df38047 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:57:56.472730 18873 catalog_manager.cc:6199] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4: default.Table_f03fffd6ca4c48cdaf9b143f1c753710 [id=747af828147149efb7b7b8d33c49286d] alter complete (version 1)
I20260130 02:57:56.478641 19742 raft_consensus.cc:515] T 01db06fcfffe4b259ac4240c0df38047 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:57:56.480463 19742 leader_election.cc:290] T 01db06fcfffe4b259ac4240c0df38047 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767)
I20260130 02:57:56.483381 19626 ts_tablet_manager.cc:1434] T d4f2b54930cc4f5c825562e3e0a79f4d P 78c2c46c7520492db35d51c244fb4f74: Time spent starting tablet: real 0.052s user 0.008s sys 0.001s
I20260130 02:57:56.487145 19504 tablet_service.cc:1552] Processing DeleteTablet for tablet 06b31d020b864c418476e6dcc1e4eafb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:57:55 UTC) from {username='slave'} at 127.0.0.1:59068
I20260130 02:57:56.488420 19626 tablet_bootstrap.cc:492] T bac739bf4cac4272975f5587ad9c71ff P 78c2c46c7520492db35d51c244fb4f74: Bootstrap starting.
I20260130 02:57:56.489137 19625 tablet_bootstrap.cc:654] T 30d8136b5ea742bca20ee46b5634d04e P d870164263bb44b6b780a557daea3656: Neither blocks nor log segments found. Creating new log.
I20260130 02:57:56.493300 19505 tablet_service.cc:1552] Processing DeleteTablet for tablet 3b8cdac4bf4b46efa26236fc930eb9a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:57:54 UTC) from {username='slave'} at 127.0.0.1:59068
I20260130 02:57:56.494917 19782 raft_consensus.cc:515] T 30d8136b5ea742bca20ee46b5634d04e P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:57:56.495473 19528 raft_consensus.cc:2468] T 878f9346d3044d8d8b15c61f86869212 P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 371a36d219164974abdf848841528152 in term 1.
I20260130 02:57:56.496846 19782 leader_election.cc:290] T 30d8136b5ea742bca20ee46b5634d04e P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 election: Requested vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), d870164263bb44b6b780a557daea3656 (127.18.100.67:39571)
I20260130 02:57:56.499830 19360 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ceda64471a92476f9ba183d2beab30d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:57:56.508658 19224 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "01db06fcfffe4b259ac4240c0df38047" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152" is_pre_election: true
I20260130 02:57:56.509243 19224 raft_consensus.cc:2468] T 01db06fcfffe4b259ac4240c0df38047 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d870164263bb44b6b780a557daea3656 in term 0.
I20260130 02:57:56.490338 19758 ts_tablet_manager.cc:1929] T 2806ae64d7324b918397c3c6b72ebcfe P 78c2c46c7520492db35d51c244fb4f74: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260130 02:57:56.515969 19758 log.cc:1199] T 2806ae64d7324b918397c3c6b72ebcfe P 78c2c46c7520492db35d51c244fb4f74: Deleting WAL directory at /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-1/wal/wals/2806ae64d7324b918397c3c6b72ebcfe
I20260130 02:57:56.516670 19758 ts_tablet_manager.cc:1950] T 2806ae64d7324b918397c3c6b72ebcfe P 78c2c46c7520492db35d51c244fb4f74: Deleting consensus metadata
I20260130 02:57:56.516764 19463 leader_election.cc:304] T 01db06fcfffe4b259ac4240c0df38047 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 371a36d219164974abdf848841528152, d870164263bb44b6b780a557daea3656; no voters:
I20260130 02:57:56.517740 19222 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "30d8136b5ea742bca20ee46b5634d04e" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152"
I20260130 02:57:56.519670 19630 raft_consensus.cc:2804] T 01db06fcfffe4b259ac4240c0df38047 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260130 02:57:56.520002 19630 raft_consensus.cc:493] T 01db06fcfffe4b259ac4240c0df38047 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260130 02:57:56.520296 19630 raft_consensus.cc:3060] T 01db06fcfffe4b259ac4240c0df38047 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:57:56.532225 19626 tablet_bootstrap.cc:654] T bac739bf4cac4272975f5587ad9c71ff P 78c2c46c7520492db35d51c244fb4f74: Neither blocks nor log segments found. Creating new log.
I20260130 02:57:56.532313 18872 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50284:
table { table_name: "default.Table_ab500271b22e4a789171a24c55b79fce" } new_table_name: "default.Table_90ad2e09fa9b4952a1564cee70ab8c1c" modify_external_catalogs: true
I20260130 02:57:56.536880 19625 tablet_bootstrap.cc:492] T 30d8136b5ea742bca20ee46b5634d04e P d870164263bb44b6b780a557daea3656: No bootstrap required, opened a new log
I20260130 02:57:56.537278 19625 ts_tablet_manager.cc:1403] T 30d8136b5ea742bca20ee46b5634d04e P d870164263bb44b6b780a557daea3656: Time spent bootstrapping tablet: real 0.073s user 0.009s sys 0.001s
I20260130 02:57:56.539366 19625 raft_consensus.cc:359] T 30d8136b5ea742bca20ee46b5634d04e P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:57:56.539963 19625 raft_consensus.cc:385] T 30d8136b5ea742bca20ee46b5634d04e P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:57:56.540210 19625 raft_consensus.cc:740] T 30d8136b5ea742bca20ee46b5634d04e P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d870164263bb44b6b780a557daea3656, State: Initialized, Role: FOLLOWER
I20260130 02:57:56.541213 19625 consensus_queue.cc:260] T 30d8136b5ea742bca20ee46b5634d04e P d870164263bb44b6b780a557daea3656 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
W20260130 02:57:56.542253 18859 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet 37470ce4356f4542b621e21698551209 (table default.Table_007e969f70d345a7bcb3dfbaabbc025d [id=77d4fa8705a3419cb0a84d3ca1bb4ce3], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 37470ce4356f4542b621e21698551209 consensus state has no leader
I20260130 02:57:56.546157 19625 ts_tablet_manager.cc:1434] T 30d8136b5ea742bca20ee46b5634d04e P d870164263bb44b6b780a557daea3656: Time spent starting tablet: real 0.009s user 0.007s sys 0.000s
I20260130 02:57:56.546976 19625 tablet_bootstrap.cc:492] T 4035b8a330844dd28af9f3244bf8eb9a P d870164263bb44b6b780a557daea3656: Bootstrap starting.
I20260130 02:57:56.553141 19528 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "30d8136b5ea742bca20ee46b5634d04e" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656"
I20260130 02:57:56.553711 19528 raft_consensus.cc:3060] T 30d8136b5ea742bca20ee46b5634d04e P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:57:56.554430 19626 tablet_bootstrap.cc:492] T bac739bf4cac4272975f5587ad9c71ff P 78c2c46c7520492db35d51c244fb4f74: No bootstrap required, opened a new log
I20260130 02:57:56.554847 19626 ts_tablet_manager.cc:1403] T bac739bf4cac4272975f5587ad9c71ff P 78c2c46c7520492db35d51c244fb4f74: Time spent bootstrapping tablet: real 0.068s user 0.004s sys 0.005s
I20260130 02:57:56.557056 19626 raft_consensus.cc:359] T bac739bf4cac4272975f5587ad9c71ff P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:57:56.557674 19626 raft_consensus.cc:385] T bac739bf4cac4272975f5587ad9c71ff P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:57:56.557940 19626 raft_consensus.cc:740] T bac739bf4cac4272975f5587ad9c71ff P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78c2c46c7520492db35d51c244fb4f74, State: Initialized, Role: FOLLOWER
I20260130 02:57:56.558503 19626 consensus_queue.cc:260] T bac739bf4cac4272975f5587ad9c71ff P 78c2c46c7520492db35d51c244fb4f74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:57:56.560372 19626 ts_tablet_manager.cc:1434] T bac739bf4cac4272975f5587ad9c71ff P 78c2c46c7520492db35d51c244fb4f74: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260130 02:57:56.561825 19783 raft_consensus.cc:515] T 01db06fcfffe4b259ac4240c0df38047 P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:57:56.563570 19783 leader_election.cc:290] T 01db06fcfffe4b259ac4240c0df38047 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 election: Requested vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), d870164263bb44b6b780a557daea3656 (127.18.100.67:39571)
I20260130 02:57:56.565678 19783 raft_consensus.cc:493] T 4035b8a330844dd28af9f3244bf8eb9a P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:57:56.566040 19783 raft_consensus.cc:515] T 4035b8a330844dd28af9f3244bf8eb9a P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:57:56.566423 19377 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "01db06fcfffe4b259ac4240c0df38047" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74" is_pre_election: true
I20260130 02:57:56.567137 19377 raft_consensus.cc:2393] T 01db06fcfffe4b259ac4240c0df38047 P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d870164263bb44b6b780a557daea3656 in current term 1: Already voted for candidate 78c2c46c7520492db35d51c244fb4f74 in this term.
I20260130 02:57:56.567706 19783 leader_election.cc:290] T 4035b8a330844dd28af9f3244bf8eb9a P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), d870164263bb44b6b780a557daea3656 (127.18.100.67:39571)
I20260130 02:57:56.569465 19626 tablet_bootstrap.cc:492] T ceda64471a92476f9ba183d2beab30d1 P 78c2c46c7520492db35d51c244fb4f74: Bootstrap starting.
I20260130 02:57:56.575162 19626 tablet_bootstrap.cc:654] T ceda64471a92476f9ba183d2beab30d1 P 78c2c46c7520492db35d51c244fb4f74: Neither blocks nor log segments found. Creating new log.
I20260130 02:57:56.579960 19625 tablet_bootstrap.cc:654] T 4035b8a330844dd28af9f3244bf8eb9a P d870164263bb44b6b780a557daea3656: Neither blocks nor log segments found. Creating new log.
I20260130 02:57:56.585315 19377 raft_consensus.cc:1275] T f2d01bfdc9c4494386bd02bed655fe63 P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Refusing update from remote peer d870164263bb44b6b780a557daea3656: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260130 02:57:56.586333 19626 tablet_bootstrap.cc:492] T ceda64471a92476f9ba183d2beab30d1 P 78c2c46c7520492db35d51c244fb4f74: No bootstrap required, opened a new log
I20260130 02:57:56.586750 19626 ts_tablet_manager.cc:1403] T ceda64471a92476f9ba183d2beab30d1 P 78c2c46c7520492db35d51c244fb4f74: Time spent bootstrapping tablet: real 0.017s user 0.012s sys 0.003s
I20260130 02:57:56.588810 19626 raft_consensus.cc:359] T ceda64471a92476f9ba183d2beab30d1 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:57:56.589372 19626 raft_consensus.cc:385] T ceda64471a92476f9ba183d2beab30d1 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:57:56.589609 19626 raft_consensus.cc:740] T ceda64471a92476f9ba183d2beab30d1 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78c2c46c7520492db35d51c244fb4f74, State: Initialized, Role: FOLLOWER
I20260130 02:57:56.590428 19626 consensus_queue.cc:260] T ceda64471a92476f9ba183d2beab30d1 P 78c2c46c7520492db35d51c244fb4f74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:57:56.592579 19626 ts_tablet_manager.cc:1434] T ceda64471a92476f9ba183d2beab30d1 P 78c2c46c7520492db35d51c244fb4f74: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20260130 02:57:56.597968 19630 raft_consensus.cc:515] T 01db06fcfffe4b259ac4240c0df38047 P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:57:56.600248 19625 tablet_bootstrap.cc:492] T 4035b8a330844dd28af9f3244bf8eb9a P d870164263bb44b6b780a557daea3656: No bootstrap required, opened a new log
I20260130 02:57:56.600627 19625 ts_tablet_manager.cc:1403] T 4035b8a330844dd28af9f3244bf8eb9a P d870164263bb44b6b780a557daea3656: Time spent bootstrapping tablet: real 0.054s user 0.009s sys 0.000s
I20260130 02:57:56.604794 19625 raft_consensus.cc:359] T 4035b8a330844dd28af9f3244bf8eb9a P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:57:56.600605 19630 leader_election.cc:290] T 01db06fcfffe4b259ac4240c0df38047 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 election: Requested vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767)
I20260130 02:57:56.605410 19625 raft_consensus.cc:385] T 4035b8a330844dd28af9f3244bf8eb9a P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:57:56.606988 19625 raft_consensus.cc:740] T 4035b8a330844dd28af9f3244bf8eb9a P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d870164263bb44b6b780a557daea3656, State: Initialized, Role: FOLLOWER
I20260130 02:57:56.607636 19625 consensus_queue.cc:260] T 4035b8a330844dd28af9f3244bf8eb9a P d870164263bb44b6b780a557daea3656 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:57:56.609786 19625 ts_tablet_manager.cc:1434] T 4035b8a330844dd28af9f3244bf8eb9a P d870164263bb44b6b780a557daea3656: Time spent starting tablet: real 0.007s user 0.004s sys 0.000s
I20260130 02:57:56.518736 19222 raft_consensus.cc:3060] T 30d8136b5ea742bca20ee46b5634d04e P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:57:56.615085 19526 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "01db06fcfffe4b259ac4240c0df38047" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656"
I20260130 02:57:56.615824 19526 raft_consensus.cc:2393] T 01db06fcfffe4b259ac4240c0df38047 P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 78c2c46c7520492db35d51c244fb4f74 in current term 1: Already voted for candidate d870164263bb44b6b780a557daea3656 in this term.
I20260130 02:57:56.617331 18873 catalog_manager.cc:5654] T 93809949599a449ca09b13e9b8bbdef6 P d870164263bb44b6b780a557daea3656 reported cstate change: term changed from 0 to 1, leader changed from <none> to d870164263bb44b6b780a557daea3656 (127.18.100.67). New cstate: current_term: 1 leader_uuid: "d870164263bb44b6b780a557daea3656" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } health_report { overall_health: UNKNOWN } } }
I20260130 02:57:56.618016 19526 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4035b8a330844dd28af9f3244bf8eb9a" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
I20260130 02:57:56.618549 19526 raft_consensus.cc:2468] T 4035b8a330844dd28af9f3244bf8eb9a P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78c2c46c7520492db35d51c244fb4f74 in term 0.
I20260130 02:57:56.619504 18873 catalog_manager.cc:5746] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571) does not have the latest schema for tablet 37470ce4356f4542b621e21698551209 (table default.Table_007e969f70d345a7bcb3dfbaabbc025d [id=77d4fa8705a3419cb0a84d3ca1bb4ce3]). Expected version 1 got 0
I20260130 02:57:56.620559 18873 catalog_manager.cc:5746] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571) does not have the latest schema for tablet b078934be4bb4fd1915c691eeea9d00f (table default.Table_fcce594e0ed443158293215c25e4e5c3 [id=720decfb55c24f34983ef636f06e3169]). Expected version 1 got 0
I20260130 02:57:56.620893 19313 leader_election.cc:304] T 4035b8a330844dd28af9f3244bf8eb9a P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78c2c46c7520492db35d51c244fb4f74, d870164263bb44b6b780a557daea3656; no voters:
I20260130 02:57:56.622172 19677 consensus_queue.cc:1048] T f2d01bfdc9c4494386bd02bed655fe63 P d870164263bb44b6b780a557daea3656 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260130 02:57:56.630155 19788 tablet.cc:1717] T 38a7a10e36874183bf9fe55a38d829f6 P d870164263bb44b6b780a557daea3656: Skipping requested alter to schema version 2, tablet already version 2
I20260130 02:57:56.631549 19783 raft_consensus.cc:2804] T 4035b8a330844dd28af9f3244bf8eb9a P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260130 02:57:56.631909 19783 raft_consensus.cc:493] T 4035b8a330844dd28af9f3244bf8eb9a P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260130 02:57:56.632207 19783 raft_consensus.cc:3060] T 4035b8a330844dd28af9f3244bf8eb9a P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:57:56.635104 19221 raft_consensus.cc:1275] T f2d01bfdc9c4494386bd02bed655fe63 P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Refusing update from remote peer d870164263bb44b6b780a557daea3656: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260130 02:57:56.636952 19781 tablet.cc:1722] T 927c90b84d7449efa36bc5e2035bf0da P 371a36d219164974abdf848841528152: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260130 02:57:56.612866 19625 tablet_bootstrap.cc:492] T bac739bf4cac4272975f5587ad9c71ff P d870164263bb44b6b780a557daea3656: Bootstrap starting.
I20260130 02:57:56.644680 18871 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50284:
name: "default.Table_e3b63d7d5d874900aac850654d98d62e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260130 02:57:56.646557 18871 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e3b63d7d5d874900aac850654d98d62e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260130 02:57:56.648881 18870 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50284:
name: "default.Table_5d29239ce1984d2fad02607a2bdd4dd0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260130 02:57:56.648952 18868 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50284:
name: "default.Table_2f60fd0292ab40c8a3ae9b842cae8faf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260130 02:57:56.665026 19528 raft_consensus.cc:2468] T 30d8136b5ea742bca20ee46b5634d04e P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78c2c46c7520492db35d51c244fb4f74 in term 1.
I20260130 02:57:56.667600 19225 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "01db06fcfffe4b259ac4240c0df38047" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152"
I20260130 02:57:56.667845 19758 tablet_replica.cc:333] T c2a9675427754d0db1578637a37440c6 P 78c2c46c7520492db35d51c244fb4f74: stopping tablet replica
I20260130 02:57:56.667903 19224 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4035b8a330844dd28af9f3244bf8eb9a" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152" is_pre_election: true
I20260130 02:57:56.668165 19225 raft_consensus.cc:3060] T 01db06fcfffe4b259ac4240c0df38047 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:57:56.668670 19758 raft_consensus.cc:2243] T c2a9675427754d0db1578637a37440c6 P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260130 02:57:56.669077 19758 pending_rounds.cc:70] T c2a9675427754d0db1578637a37440c6 P 78c2c46c7520492db35d51c244fb4f74: Trying to abort 1 pending ops.
W20260130 02:57:56.650561 18870 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5d29239ce1984d2fad02607a2bdd4dd0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260130 02:57:56.675777 19313 leader_election.cc:304] T 30d8136b5ea742bca20ee46b5634d04e P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78c2c46c7520492db35d51c244fb4f74, d870164263bb44b6b780a557daea3656; no voters:
I20260130 02:57:56.676563 19663 raft_consensus.cc:2804] T 30d8136b5ea742bca20ee46b5634d04e P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Leader election won for term 1
I20260130 02:57:56.676982 19663 raft_consensus.cc:697] T 30d8136b5ea742bca20ee46b5634d04e P 78c2c46c7520492db35d51c244fb4f74 [term 1 LEADER]: Becoming Leader. State: Replica: 78c2c46c7520492db35d51c244fb4f74, State: Running, Role: LEADER
I20260130 02:57:56.677699 19663 consensus_queue.cc:237] T 30d8136b5ea742bca20ee46b5634d04e P 78c2c46c7520492db35d51c244fb4f74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:57:56.669282 19758 pending_rounds.cc:77] T c2a9675427754d0db1578637a37440c6 P 78c2c46c7520492db35d51c244fb4f74: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7248862721173012480 op_type: NO_OP noop_request { }
I20260130 02:57:56.678926 19758 raft_consensus.cc:2889] T c2a9675427754d0db1578637a37440c6 P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260130 02:57:56.680506 19758 raft_consensus.cc:2272] T c2a9675427754d0db1578637a37440c6 P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260130 02:57:56.682832 18860 catalog_manager.cc:4985] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767): tablet 2806ae64d7324b918397c3c6b72ebcfe (table default.Table_d1b81d6a8e704777b0fa35cf23dc91e0 [id=78a1af9c84574331a9112fdea3f67fcf]) successfully deleted
I20260130 02:57:56.683948 19758 ts_tablet_manager.cc:1916] T c2a9675427754d0db1578637a37440c6 P 78c2c46c7520492db35d51c244fb4f74: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260130 02:57:56.681155 19783 raft_consensus.cc:515] T 4035b8a330844dd28af9f3244bf8eb9a P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:57:56.686070 19783 leader_election.cc:290] T 4035b8a330844dd28af9f3244bf8eb9a P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 election: Requested vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), d870164263bb44b6b780a557daea3656 (127.18.100.67:39571)
W20260130 02:57:56.654714 18868 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2f60fd0292ab40c8a3ae9b842cae8faf in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260130 02:57:56.689649 19528 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4035b8a330844dd28af9f3244bf8eb9a" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656"
I20260130 02:57:56.690199 19528 raft_consensus.cc:3060] T 4035b8a330844dd28af9f3244bf8eb9a P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:57:56.693118 19221 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "01db06fcfffe4b259ac4240c0df38047" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152"
W20260130 02:57:56.694514 19314 leader_election.cc:343] T 4035b8a330844dd28af9f3244bf8eb9a P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:57:56.695475 19314 leader_election.cc:304] T 01db06fcfffe4b259ac4240c0df38047 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78c2c46c7520492db35d51c244fb4f74; no voters: 371a36d219164974abdf848841528152, d870164263bb44b6b780a557daea3656
I20260130 02:57:56.696509 19663 raft_consensus.cc:2749] T 01db06fcfffe4b259ac4240c0df38047 P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260130 02:57:56.697890 19754 raft_consensus.cc:2243] T c2a9675427754d0db1578637a37440c6 P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260130 02:57:56.698429 19754 raft_consensus.cc:2272] T c2a9675427754d0db1578637a37440c6 P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260130 02:57:56.700408 19754 ts_tablet_manager.cc:1916] T c2a9675427754d0db1578637a37440c6 P 371a36d219164974abdf848841528152: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260130 02:57:56.707441 19811 consensus_queue.cc:1048] T f2d01bfdc9c4494386bd02bed655fe63 P d870164263bb44b6b780a557daea3656 [LEADER]: Connected to new peer: Peer: permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260130 02:57:56.710968 19221 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4035b8a330844dd28af9f3244bf8eb9a" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152"
W20260130 02:57:56.715126 19314 leader_election.cc:343] T 4035b8a330844dd28af9f3244bf8eb9a P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:57:56.715518 19377 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "01db06fcfffe4b259ac4240c0df38047" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74"
I20260130 02:57:56.716282 19377 raft_consensus.cc:2393] T 01db06fcfffe4b259ac4240c0df38047 P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate d870164263bb44b6b780a557daea3656 in current term 1: Already voted for candidate 78c2c46c7520492db35d51c244fb4f74 in this term.
I20260130 02:57:56.719985 19794 tablet.cc:1722] T b078934be4bb4fd1915c691eeea9d00f P d870164263bb44b6b780a557daea3656: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260130 02:57:56.725703 19785 tablet.cc:1722] T b078934be4bb4fd1915c691eeea9d00f P 78c2c46c7520492db35d51c244fb4f74: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260130 02:57:56.733958 19788 tablet.cc:1722] T f2d01bfdc9c4494386bd02bed655fe63 P d870164263bb44b6b780a557daea3656: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20260130 02:57:56.739573 18867 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2112b0402de5464cb1caf2c2abb67d78 has no consensus state
W20260130 02:57:56.741931 18872 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ceda64471a92476f9ba183d2beab30d1 consensus state has no leader
W20260130 02:57:56.742739 19159 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.100.66:40767, user_credentials={real_user=slave}} blocked reactor thread for 292685us
I20260130 02:57:56.743494 19159 leader_election.cc:304] T 878f9346d3044d8d8b15c61f86869212 P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 371a36d219164974abdf848841528152, 78c2c46c7520492db35d51c244fb4f74; no voters:
I20260130 02:57:56.744391 19635 raft_consensus.cc:2804] T 878f9346d3044d8d8b15c61f86869212 P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Leader election won for term 1
I20260130 02:57:56.745075 19635 raft_consensus.cc:697] T 878f9346d3044d8d8b15c61f86869212 P 371a36d219164974abdf848841528152 [term 1 LEADER]: Becoming Leader. State: Replica: 371a36d219164974abdf848841528152, State: Running, Role: LEADER
I20260130 02:57:56.746114 19635 consensus_queue.cc:237] T 878f9346d3044d8d8b15c61f86869212 P 371a36d219164974abdf848841528152 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:57:56.756153 19225 raft_consensus.cc:2468] T 01db06fcfffe4b259ac4240c0df38047 P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d870164263bb44b6b780a557daea3656 in term 1.
I20260130 02:57:56.758780 19222 raft_consensus.cc:2468] T 30d8136b5ea742bca20ee46b5634d04e P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78c2c46c7520492db35d51c244fb4f74 in term 1.
I20260130 02:57:56.762804 19710 ts_tablet_manager.cc:1929] T 2806ae64d7324b918397c3c6b72ebcfe P d870164263bb44b6b780a557daea3656: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260130 02:57:56.763007 19463 leader_election.cc:304] T 01db06fcfffe4b259ac4240c0df38047 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 371a36d219164974abdf848841528152, d870164263bb44b6b780a557daea3656; no voters:
I20260130 02:57:56.763232 19710 log.cc:1199] T 2806ae64d7324b918397c3c6b72ebcfe P d870164263bb44b6b780a557daea3656: Deleting WAL directory at /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-2/wal/wals/2806ae64d7324b918397c3c6b72ebcfe
I20260130 02:57:56.763789 19811 raft_consensus.cc:2804] T 01db06fcfffe4b259ac4240c0df38047 P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Leader election won for term 1
I20260130 02:57:56.763996 19710 ts_tablet_manager.cc:1950] T 2806ae64d7324b918397c3c6b72ebcfe P d870164263bb44b6b780a557daea3656: Deleting consensus metadata
I20260130 02:57:56.764631 19811 raft_consensus.cc:697] T 01db06fcfffe4b259ac4240c0df38047 P d870164263bb44b6b780a557daea3656 [term 1 LEADER]: Becoming Leader. State: Replica: d870164263bb44b6b780a557daea3656, State: Running, Role: LEADER
I20260130 02:57:56.764631 19781 tablet.cc:1722] T 38a7a10e36874183bf9fe55a38d829f6 P 371a36d219164974abdf848841528152: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20260130 02:57:56.765631 19811 consensus_queue.cc:237] T 01db06fcfffe4b259ac4240c0df38047 P d870164263bb44b6b780a557daea3656 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:57:56.774734 19635 consensus_queue.cc:1048] T a46702bef0e6442d85e4ca49a5f7cfbb P 371a36d219164974abdf848841528152 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260130 02:57:56.777240 19528 raft_consensus.cc:2468] T 4035b8a330844dd28af9f3244bf8eb9a P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78c2c46c7520492db35d51c244fb4f74 in term 1.
I20260130 02:57:56.778717 19313 leader_election.cc:304] T 4035b8a330844dd28af9f3244bf8eb9a P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 78c2c46c7520492db35d51c244fb4f74, d870164263bb44b6b780a557daea3656; no voters: 371a36d219164974abdf848841528152
I20260130 02:57:56.781683 19663 raft_consensus.cc:2804] T 4035b8a330844dd28af9f3244bf8eb9a P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Leader election won for term 1
I20260130 02:57:56.782475 19663 raft_consensus.cc:697] T 4035b8a330844dd28af9f3244bf8eb9a P 78c2c46c7520492db35d51c244fb4f74 [term 1 LEADER]: Becoming Leader. State: Replica: 78c2c46c7520492db35d51c244fb4f74, State: Running, Role: LEADER
I20260130 02:57:56.785282 19816 tablet.cc:1722] T f2d01bfdc9c4494386bd02bed655fe63 P 371a36d219164974abdf848841528152: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260130 02:57:56.785696 19663 consensus_queue.cc:237] T 4035b8a330844dd28af9f3244bf8eb9a P 78c2c46c7520492db35d51c244fb4f74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:57:56.787878 19813 tablet.cc:1722] T b078934be4bb4fd1915c691eeea9d00f P 371a36d219164974abdf848841528152: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260130 02:57:56.791394 19635 consensus_queue.cc:1048] T a46702bef0e6442d85e4ca49a5f7cfbb P 371a36d219164974abdf848841528152 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260130 02:57:56.805606 18872 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50284:
name: "default.Table_0658e13596f44598bb5a108797bc06fc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260130 02:57:56.807518 18872 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0658e13596f44598bb5a108797bc06fc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260130 02:57:56.815198 19462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.100.66:40767, user_credentials={real_user=slave}} blocked reactor thread for 193997us
I20260130 02:57:56.830945 19807 tablet_bootstrap.cc:492] T bac739bf4cac4272975f5587ad9c71ff P 371a36d219164974abdf848841528152: Bootstrap starting.
I20260130 02:57:56.845957 19625 tablet_bootstrap.cc:654] T bac739bf4cac4272975f5587ad9c71ff P d870164263bb44b6b780a557daea3656: Neither blocks nor log segments found. Creating new log.
I20260130 02:57:56.847275 19791 tablet.cc:1717] T 38a7a10e36874183bf9fe55a38d829f6 P 78c2c46c7520492db35d51c244fb4f74: Skipping requested alter to schema version 2, tablet already version 2
I20260130 02:57:56.852432 19758 ts_tablet_manager.cc:1929] T c2a9675427754d0db1578637a37440c6 P 78c2c46c7520492db35d51c244fb4f74: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260130 02:57:56.852820 19758 log.cc:1199] T c2a9675427754d0db1578637a37440c6 P 78c2c46c7520492db35d51c244fb4f74: Deleting WAL directory at /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-1/wal/wals/c2a9675427754d0db1578637a37440c6
I20260130 02:57:56.853500 19758 ts_tablet_manager.cc:1950] T c2a9675427754d0db1578637a37440c6 P 78c2c46c7520492db35d51c244fb4f74: Deleting consensus metadata
W20260130 02:57:56.848783 19462 consensus_peers.cc:597] T c2a9675427754d0db1578637a37440c6 P d870164263bb44b6b780a557daea3656 -> Peer 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767): Couldn't send request to peer 78c2c46c7520492db35d51c244fb4f74. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20260130 02:57:56.859869 19781 tablet.cc:1717] T 38a7a10e36874183bf9fe55a38d829f6 P 371a36d219164974abdf848841528152: Skipping requested alter to schema version 2, tablet already version 2
I20260130 02:57:56.866801 19710 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59068 (request call id 30) took 1068 ms. Trace:
I20260130 02:57:56.867247 19710 rpcz_store.cc:276] 0130 02:57:55.798440 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:57:55.801771 (+ 3331us) service_pool.cc:225] Handling call
0130 02:57:56.172426 (+370655us) ts_tablet_manager.cc:1151] Deleting tablet 2806ae64d7324b918397c3c6b72ebcfe
0130 02:57:56.172846 (+ 420us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0130 02:57:56.568464 (+395618us) tablet_metadata.cc:641] Metadata flushed
0130 02:57:56.762462 (+193998us) tablet_metadata.cc:641] Metadata flushed
0130 02:57:56.866726 (+104264us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":370074}
I20260130 02:57:56.870060 19710 tablet_replica.cc:333] T c2a9675427754d0db1578637a37440c6 P d870164263bb44b6b780a557daea3656: stopping tablet replica
I20260130 02:57:56.870921 19710 raft_consensus.cc:2243] T c2a9675427754d0db1578637a37440c6 P d870164263bb44b6b780a557daea3656 [term 1 LEADER]: Raft consensus shutting down.
I20260130 02:57:56.871882 19710 raft_consensus.cc:2272] T c2a9675427754d0db1578637a37440c6 P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260130 02:57:56.872897 18860 catalog_manager.cc:4985] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): tablet 2806ae64d7324b918397c3c6b72ebcfe (table default.Table_d1b81d6a8e704777b0fa35cf23dc91e0 [id=78a1af9c84574331a9112fdea3f67fcf]) successfully deleted
I20260130 02:57:56.878116 19807 tablet_bootstrap.cc:654] T bac739bf4cac4272975f5587ad9c71ff P 371a36d219164974abdf848841528152: Neither blocks nor log segments found. Creating new log.
I20260130 02:57:56.888295 19811 consensus_queue.cc:1048] T 3b8cdac4bf4b46efa26236fc930eb9a6 P d870164263bb44b6b780a557daea3656 [LEADER]: Connected to new peer: Peer: permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260130 02:57:56.895702 19807 tablet_bootstrap.cc:492] T bac739bf4cac4272975f5587ad9c71ff P 371a36d219164974abdf848841528152: No bootstrap required, opened a new log
I20260130 02:57:56.897497 19807 ts_tablet_manager.cc:1403] T bac739bf4cac4272975f5587ad9c71ff P 371a36d219164974abdf848841528152: Time spent bootstrapping tablet: real 0.105s user 0.003s sys 0.009s
I20260130 02:57:56.897842 18860 catalog_manager.cc:4985] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767): tablet c2a9675427754d0db1578637a37440c6 (table default.Table_69ffd00165c6422292d9976b685d5d0d [id=f4829d67e1394959b7a7335fd067a826]) successfully deleted
I20260130 02:57:56.903699 19807 raft_consensus.cc:359] T bac739bf4cac4272975f5587ad9c71ff P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:57:56.906154 19710 ts_tablet_manager.cc:1916] T c2a9675427754d0db1578637a37440c6 P d870164263bb44b6b780a557daea3656: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260130 02:57:56.917594 19807 raft_consensus.cc:385] T bac739bf4cac4272975f5587ad9c71ff P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:57:56.917959 19807 raft_consensus.cc:740] T bac739bf4cac4272975f5587ad9c71ff P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 371a36d219164974abdf848841528152, State: Initialized, Role: FOLLOWER
I20260130 02:57:56.918718 19807 consensus_queue.cc:260] T bac739bf4cac4272975f5587ad9c71ff P 371a36d219164974abdf848841528152 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:57:56.921058 19807 ts_tablet_manager.cc:1434] T bac739bf4cac4272975f5587ad9c71ff P 371a36d219164974abdf848841528152: Time spent starting tablet: real 0.020s user 0.004s sys 0.000s
I20260130 02:57:56.924975 19807 tablet_bootstrap.cc:492] T 4035b8a330844dd28af9f3244bf8eb9a P 371a36d219164974abdf848841528152: Bootstrap starting.
I20260130 02:57:56.944171 18871 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50284:
table { table_name: "default.Table_a601373812c04ecf9924ecb3ea056a8c" } new_table_name: "default.Table_4d43592dc5b24ff28710e62ad4d87ce0" modify_external_catalogs: true
I20260130 02:57:56.947604 18868 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50284:
table { table_name: "default.Table_fcce594e0ed443158293215c25e4e5c3" } new_table_name: "default.Table_604228ee4a914be8a2f2f49f2f51e3f3" modify_external_catalogs: true
I20260130 02:57:56.948791 19679 raft_consensus.cc:493] T bac739bf4cac4272975f5587ad9c71ff P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:57:56.949229 19679 raft_consensus.cc:515] T bac739bf4cac4272975f5587ad9c71ff P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:57:56.951015 19679 leader_election.cc:290] T bac739bf4cac4272975f5587ad9c71ff P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767), d870164263bb44b6b780a557daea3656 (127.18.100.67:39571)
I20260130 02:57:56.952795 19528 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bac739bf4cac4272975f5587ad9c71ff" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
W20260130 02:57:56.956240 19159 leader_election.cc:343] T bac739bf4cac4272975f5587ad9c71ff P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:57:56.959179 19380 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bac739bf4cac4272975f5587ad9c71ff" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74" is_pre_election: true
I20260130 02:57:56.959895 19380 raft_consensus.cc:2468] T bac739bf4cac4272975f5587ad9c71ff P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 371a36d219164974abdf848841528152 in term 0.
I20260130 02:57:56.967329 19159 leader_election.cc:304] T bac739bf4cac4272975f5587ad9c71ff P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 371a36d219164974abdf848841528152, 78c2c46c7520492db35d51c244fb4f74; no voters: d870164263bb44b6b780a557daea3656
I20260130 02:57:56.968891 19679 raft_consensus.cc:2804] T bac739bf4cac4272975f5587ad9c71ff P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260130 02:57:56.969213 19679 raft_consensus.cc:493] T bac739bf4cac4272975f5587ad9c71ff P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260130 02:57:56.969497 19679 raft_consensus.cc:3060] T bac739bf4cac4272975f5587ad9c71ff P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:57:57.018487 19806 consensus_queue.cc:1048] T 93809949599a449ca09b13e9b8bbdef6 P d870164263bb44b6b780a557daea3656 [LEADER]: Connected to new peer: Peer: permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260130 02:57:56.941668 19625 tablet_bootstrap.cc:492] T bac739bf4cac4272975f5587ad9c71ff P d870164263bb44b6b780a557daea3656: No bootstrap required, opened a new log
I20260130 02:57:57.030745 19625 ts_tablet_manager.cc:1403] T bac739bf4cac4272975f5587ad9c71ff P d870164263bb44b6b780a557daea3656: Time spent bootstrapping tablet: real 0.420s user 0.007s sys 0.004s
I20260130 02:57:57.034075 19625 raft_consensus.cc:359] T bac739bf4cac4272975f5587ad9c71ff P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:57:57.034840 19625 raft_consensus.cc:385] T bac739bf4cac4272975f5587ad9c71ff P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:57:57.035128 19625 raft_consensus.cc:740] T bac739bf4cac4272975f5587ad9c71ff P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d870164263bb44b6b780a557daea3656, State: Initialized, Role: FOLLOWER
I20260130 02:57:57.035797 19625 consensus_queue.cc:260] T bac739bf4cac4272975f5587ad9c71ff P d870164263bb44b6b780a557daea3656 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:57:57.037871 19625 ts_tablet_manager.cc:1434] T bac739bf4cac4272975f5587ad9c71ff P d870164263bb44b6b780a557daea3656: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20260130 02:57:57.038720 19625 tablet_bootstrap.cc:492] T d4f2b54930cc4f5c825562e3e0a79f4d P d870164263bb44b6b780a557daea3656: Bootstrap starting.
I20260130 02:57:57.043516 19775 mvcc.cc:204] Tried to move back new op lower bound from 7248862726063476736 to 7248862722498981888. Current Snapshot: MvccSnapshot[applied={T|T < 7248862726063476736}]
I20260130 02:57:57.045130 19785 tablet.cc:1722] T f2d01bfdc9c4494386bd02bed655fe63 P 78c2c46c7520492db35d51c244fb4f74: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260130 02:57:57.047132 19679 raft_consensus.cc:515] T bac739bf4cac4272975f5587ad9c71ff P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:57:57.048834 19679 leader_election.cc:290] T bac739bf4cac4272975f5587ad9c71ff P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 election: Requested vote from peers 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767), d870164263bb44b6b780a557daea3656 (127.18.100.67:39571)
I20260130 02:57:57.050967 19754 ts_tablet_manager.cc:1929] T c2a9675427754d0db1578637a37440c6 P 371a36d219164974abdf848841528152: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260130 02:57:57.051345 19754 log.cc:1199] T c2a9675427754d0db1578637a37440c6 P 371a36d219164974abdf848841528152: Deleting WAL directory at /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-0/wal/wals/c2a9675427754d0db1578637a37440c6
I20260130 02:57:57.052086 19754 ts_tablet_manager.cc:1950] T c2a9675427754d0db1578637a37440c6 P 371a36d219164974abdf848841528152: Deleting consensus metadata
I20260130 02:57:57.053390 19528 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bac739bf4cac4272975f5587ad9c71ff" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656"
I20260130 02:57:57.053970 19528 raft_consensus.cc:3060] T bac739bf4cac4272975f5587ad9c71ff P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:57:57.056867 19380 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bac739bf4cac4272975f5587ad9c71ff" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74"
I20260130 02:57:57.057379 19380 raft_consensus.cc:3060] T bac739bf4cac4272975f5587ad9c71ff P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:57:57.060470 19625 tablet_bootstrap.cc:654] T d4f2b54930cc4f5c825562e3e0a79f4d P d870164263bb44b6b780a557daea3656: Neither blocks nor log segments found. Creating new log.
I20260130 02:57:57.086429 18870 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50284:
name: "default.Table_71ecfab2cca249b49070a8340586ce5e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260130 02:57:57.087054 18872 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50284:
name: "default.Table_f246882197f44c69b72ca421cbdf201f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260130 02:57:57.088124 18870 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_71ecfab2cca249b49070a8340586ce5e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260130 02:57:57.088735 18872 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f246882197f44c69b72ca421cbdf201f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260130 02:57:57.091028 18867 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50284:
name: "default.Table_2c0d85ead4ad406880e5796612629f18"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260130 02:57:57.092628 18867 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2c0d85ead4ad406880e5796612629f18 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260130 02:57:57.096832 19625 tablet_bootstrap.cc:492] T d4f2b54930cc4f5c825562e3e0a79f4d P d870164263bb44b6b780a557daea3656: No bootstrap required, opened a new log
I20260130 02:57:57.097234 19625 ts_tablet_manager.cc:1403] T d4f2b54930cc4f5c825562e3e0a79f4d P d870164263bb44b6b780a557daea3656: Time spent bootstrapping tablet: real 0.059s user 0.013s sys 0.004s
I20260130 02:57:57.099958 19625 raft_consensus.cc:359] T d4f2b54930cc4f5c825562e3e0a79f4d P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:57:57.100584 19625 raft_consensus.cc:385] T d4f2b54930cc4f5c825562e3e0a79f4d P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:57:57.100847 19625 raft_consensus.cc:740] T d4f2b54930cc4f5c825562e3e0a79f4d P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d870164263bb44b6b780a557daea3656, State: Initialized, Role: FOLLOWER
I20260130 02:57:57.101481 19625 consensus_queue.cc:260] T d4f2b54930cc4f5c825562e3e0a79f4d P d870164263bb44b6b780a557daea3656 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:57:57.103987 19625 ts_tablet_manager.cc:1434] T d4f2b54930cc4f5c825562e3e0a79f4d P d870164263bb44b6b780a557daea3656: Time spent starting tablet: real 0.006s user 0.006s sys 0.001s
I20260130 02:57:57.104836 19625 tablet_bootstrap.cc:492] T ceda64471a92476f9ba183d2beab30d1 P d870164263bb44b6b780a557daea3656: Bootstrap starting.
I20260130 02:57:57.122514 19742 consensus_queue.cc:1048] T 3b8cdac4bf4b46efa26236fc930eb9a6 P d870164263bb44b6b780a557daea3656 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260130 02:57:57.128556 19625 tablet_bootstrap.cc:654] T ceda64471a92476f9ba183d2beab30d1 P d870164263bb44b6b780a557daea3656: Neither blocks nor log segments found. Creating new log.
I20260130 02:57:57.130120 19742 consensus_queue.cc:1048] T 93809949599a449ca09b13e9b8bbdef6 P d870164263bb44b6b780a557daea3656 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260130 02:57:57.134809 19807 tablet_bootstrap.cc:654] T 4035b8a330844dd28af9f3244bf8eb9a P 371a36d219164974abdf848841528152: Neither blocks nor log segments found. Creating new log.
I20260130 02:57:57.143131 18861 catalog_manager.cc:4985] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021): tablet c2a9675427754d0db1578637a37440c6 (table default.Table_69ffd00165c6422292d9976b685d5d0d [id=f4829d67e1394959b7a7335fd067a826]) successfully deleted
I20260130 02:57:57.144428 19754 tablet_replica.cc:333] T 79e28c3a566c4521ac2453bbaa3567ed P 371a36d219164974abdf848841528152: stopping tablet replica
I20260130 02:57:57.145282 19754 raft_consensus.cc:2243] T 79e28c3a566c4521ac2453bbaa3567ed P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260130 02:57:57.145817 19754 raft_consensus.cc:2272] T 79e28c3a566c4521ac2453bbaa3567ed P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260130 02:57:57.151127 19380 raft_consensus.cc:2468] T bac739bf4cac4272975f5587ad9c71ff P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 371a36d219164974abdf848841528152 in term 1.
I20260130 02:57:57.152470 19159 leader_election.cc:304] T bac739bf4cac4272975f5587ad9c71ff P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 371a36d219164974abdf848841528152, 78c2c46c7520492db35d51c244fb4f74; no voters:
I20260130 02:57:57.153304 19679 raft_consensus.cc:2804] T bac739bf4cac4272975f5587ad9c71ff P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Leader election won for term 1
I20260130 02:57:57.153740 19679 raft_consensus.cc:697] T bac739bf4cac4272975f5587ad9c71ff P 371a36d219164974abdf848841528152 [term 1 LEADER]: Becoming Leader. State: Replica: 371a36d219164974abdf848841528152, State: Running, Role: LEADER
I20260130 02:57:57.154510 19679 consensus_queue.cc:237] T bac739bf4cac4272975f5587ad9c71ff P 371a36d219164974abdf848841528152 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:57:57.156787 19754 ts_tablet_manager.cc:1916] T 79e28c3a566c4521ac2453bbaa3567ed P 371a36d219164974abdf848841528152: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260130 02:57:57.169103 19528 raft_consensus.cc:2468] T bac739bf4cac4272975f5587ad9c71ff P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 371a36d219164974abdf848841528152 in term 1.
I20260130 02:57:57.175005 19710 ts_tablet_manager.cc:1929] T c2a9675427754d0db1578637a37440c6 P d870164263bb44b6b780a557daea3656: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260130 02:57:57.175483 19710 log.cc:1199] T c2a9675427754d0db1578637a37440c6 P d870164263bb44b6b780a557daea3656: Deleting WAL directory at /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-2/wal/wals/c2a9675427754d0db1578637a37440c6
I20260130 02:57:57.176072 19807 tablet_bootstrap.cc:492] T 4035b8a330844dd28af9f3244bf8eb9a P 371a36d219164974abdf848841528152: No bootstrap required, opened a new log
I20260130 02:57:57.176240 19710 ts_tablet_manager.cc:1950] T c2a9675427754d0db1578637a37440c6 P d870164263bb44b6b780a557daea3656: Deleting consensus metadata
I20260130 02:57:57.176499 19807 ts_tablet_manager.cc:1403] T 4035b8a330844dd28af9f3244bf8eb9a P 371a36d219164974abdf848841528152: Time spent bootstrapping tablet: real 0.255s user 0.010s sys 0.002s
W20260130 02:57:57.188880 18860 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.100.124:34629, user_credentials={real_user=slave}} blocked reactor thread for 71627.9us
I20260130 02:57:57.212949 19625 tablet_bootstrap.cc:492] T ceda64471a92476f9ba183d2beab30d1 P d870164263bb44b6b780a557daea3656: No bootstrap required, opened a new log
I20260130 02:57:57.213518 19625 ts_tablet_manager.cc:1403] T ceda64471a92476f9ba183d2beab30d1 P d870164263bb44b6b780a557daea3656: Time spent bootstrapping tablet: real 0.109s user 0.006s sys 0.007s
I20260130 02:57:57.216063 19625 raft_consensus.cc:359] T ceda64471a92476f9ba183d2beab30d1 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:57:57.216653 18869 catalog_manager.cc:5746] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021) does not have the latest schema for tablet 927c90b84d7449efa36bc5e2035bf0da (table default.Table_ebc8b8379c274c18b0a06a919dbe0ee0 [id=28156e0843674fce88f5914a3e4195ae]). Expected version 1 got 0
I20260130 02:57:57.216461 19506 tablet_service.cc:1505] Processing CreateTablet for tablet 2112b0402de5464cb1caf2c2abb67d78 (DEFAULT_TABLE table=default.Table_da3d581b15a94840898777f74611994e [id=b7e241ced53e4160ac6fa1f65ef3674e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:57:57.217847 18869 catalog_manager.cc:5654] T a46702bef0e6442d85e4ca49a5f7cfbb P 371a36d219164974abdf848841528152 reported cstate change: term changed from 0 to 1, leader changed from <none> to 371a36d219164974abdf848841528152 (127.18.100.65). New cstate: current_term: 1 leader_uuid: "371a36d219164974abdf848841528152" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } health_report { overall_health: HEALTHY } } }
I20260130 02:57:57.219115 19360 tablet_service.cc:1505] Processing CreateTablet for tablet 2112b0402de5464cb1caf2c2abb67d78 (DEFAULT_TABLE table=default.Table_da3d581b15a94840898777f74611994e [id=b7e241ced53e4160ac6fa1f65ef3674e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:57:57.219413 19506 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2112b0402de5464cb1caf2c2abb67d78. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:57:57.219621 18869 catalog_manager.cc:5746] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021) does not have the latest schema for tablet 37470ce4356f4542b621e21698551209 (table default.Table_007e969f70d345a7bcb3dfbaabbc025d [id=77d4fa8705a3419cb0a84d3ca1bb4ce3]). Expected version 1 got 0
I20260130 02:57:57.220069 19360 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2112b0402de5464cb1caf2c2abb67d78. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:57:57.220203 18869 catalog_manager.cc:5746] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021) does not have the latest schema for tablet b078934be4bb4fd1915c691eeea9d00f (table default.Table_fcce594e0ed443158293215c25e4e5c3 [id=720decfb55c24f34983ef636f06e3169]). Expected version 1 got 0
I20260130 02:57:57.220754 18869 catalog_manager.cc:5746] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021) does not have the latest schema for tablet 38a7a10e36874183bf9fe55a38d829f6 (table default.Table_a601373812c04ecf9924ecb3ea056a8c [id=27adf356fd004597a54c7a9887dd3ca8]). Expected version 2 got 0
I20260130 02:57:57.221907 18869 catalog_manager.cc:5746] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021) does not have the latest schema for tablet f2d01bfdc9c4494386bd02bed655fe63 (table default.Table_14c76bd65820453fb5cade67cf66e752 [id=d128767e303145a885eec3c5f714a27f]). Expected version 1 got 0
I20260130 02:57:57.222035 19358 tablet_service.cc:1505] Processing CreateTablet for tablet 073b5941850c4140b1751956dd7cf5ef (DEFAULT_TABLE table=default.Table_6eb0a92b1926487cbf0b978d26d6a7b1 [id=de7a420d40864f55b156e149308ba12f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:57:57.222966 19357 tablet_service.cc:1505] Processing CreateTablet for tablet 0b1877e5f9244913bcd639f1336df7ee (DEFAULT_TABLE table=default.Table_19677b801d1f4a63b22139b9fc80c4ef [id=3992cda3a7bb49ae86b8f84ee12f1c8b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:57:57.224076 19357 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0b1877e5f9244913bcd639f1336df7ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:57:57.229118 19507 tablet_service.cc:1505] Processing CreateTablet for tablet 0b1877e5f9244913bcd639f1336df7ee (DEFAULT_TABLE table=default.Table_19677b801d1f4a63b22139b9fc80c4ef [id=3992cda3a7bb49ae86b8f84ee12f1c8b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:57:57.230105 19507 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0b1877e5f9244913bcd639f1336df7ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:57:57.230680 19625 raft_consensus.cc:385] T ceda64471a92476f9ba183d2beab30d1 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:57:57.230962 19625 raft_consensus.cc:740] T ceda64471a92476f9ba183d2beab30d1 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d870164263bb44b6b780a557daea3656, State: Initialized, Role: FOLLOWER
I20260130 02:57:57.231309 19508 tablet_service.cc:1505] Processing CreateTablet for tablet 073b5941850c4140b1751956dd7cf5ef (DEFAULT_TABLE table=default.Table_6eb0a92b1926487cbf0b978d26d6a7b1 [id=de7a420d40864f55b156e149308ba12f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:57:57.231547 18871 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50284:
name: "default.Table_22d10332a83f4700ac21892bab67f46b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260130 02:57:57.232306 19508 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 073b5941850c4140b1751956dd7cf5ef. 1 dirs total, 0 dirs full, 0 dirs failed
W20260130 02:57:57.233171 18871 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_22d10332a83f4700ac21892bab67f46b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260130 02:57:57.239082 18875 catalog_manager.cc:5941] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4: Sending DeleteTablet for 3 replicas of tablet 01db06fcfffe4b259ac4240c0df38047
I20260130 02:57:57.238809 19625 consensus_queue.cc:260] T ceda64471a92476f9ba183d2beab30d1 P d870164263bb44b6b780a557daea3656 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:57:57.240546 18875 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:50284 (request call id 66) took 1012 ms. Trace:
I20260130 02:57:57.244993 19625 ts_tablet_manager.cc:1434] T ceda64471a92476f9ba183d2beab30d1 P d870164263bb44b6b780a557daea3656: Time spent starting tablet: real 0.031s user 0.005s sys 0.000s
I20260130 02:57:57.240886 18875 rpcz_store.cc:276] 0130 02:57:56.227716 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:57:56.227851 (+ 135us) service_pool.cc:225] Handling call
0130 02:57:56.229163 (+ 1312us) catalog_manager.cc:2393] Looking up, locking, and authorizing table
0130 02:57:56.229741 (+ 578us) catalog_manager.cc:2815] Modifying in-memory table state
0130 02:57:56.229862 (+ 121us) catalog_manager.cc:2824] Locking tablets
0130 02:57:56.239722 (+ 9860us) catalog_manager.cc:2838] Removing table and tablets from system table
0130 02:57:56.613220 (+373498us) catalog_manager.cc:2856] Removing table from by-name map
0130 02:57:56.613459 (+ 239us) catalog_manager.cc:2872] Committing in-memory state
0130 02:57:57.238986 (+625527us) catalog_manager.cc:2876] Aborting table tasks
0130 02:57:57.240488 (+ 1502us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0130 02:57:56.578939 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0130 02:57:56.579309 (+ 370us) write_op.cc:432] Acquiring schema lock in shared mode
0130 02:57:56.579331 (+ 22us) write_op.cc:435] Acquired schema lock
0130 02:57:56.579343 (+ 12us) tablet.cc:660] Decoding operations
0130 02:57:56.579645 (+ 302us) write_op.cc:620] Acquiring the partition lock for write op
0130 02:57:56.579684 (+ 39us) write_op.cc:641] Partition lock acquired for write op
0130 02:57:56.579705 (+ 21us) tablet.cc:683] Acquiring locks for 2 operations
0130 02:57:56.579968 (+ 263us) tablet.cc:699] Row locks acquired
0130 02:57:56.579981 (+ 13us) write_op.cc:260] PREPARE: finished
0130 02:57:56.580078 (+ 97us) write_op.cc:270] Start()
0130 02:57:56.580154 (+ 76us) write_op.cc:276] Timestamp: P: 1769741876580009 usec, L: 0
0130 02:57:56.601059 (+ 20905us) op_driver.cc:348] REPLICATION: starting
0130 02:57:56.601499 (+ 440us) log.cc:844] Serialized 730 byte log entry
0130 02:57:56.611116 (+ 9617us) op_driver.cc:464] REPLICATION: finished
0130 02:57:56.611478 (+ 362us) write_op.cc:301] APPLY: starting
0130 02:57:56.611532 (+ 54us) tablet.cc:1365] starting BulkCheckPresence
0130 02:57:56.611635 (+ 103us) tablet.cc:1368] finished BulkCheckPresence
0130 02:57:56.611726 (+ 91us) tablet.cc:1370] starting ApplyRowOperation cycle
0130 02:57:56.611925 (+ 199us) tablet.cc:1381] finished ApplyRowOperation cycle
0130 02:57:56.612055 (+ 130us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0130 02:57:56.612093 (+ 38us) write_op.cc:312] APPLY: finished
0130 02:57:56.612254 (+ 161us) log.cc:844] Serialized 28 byte log entry
0130 02:57:56.612390 (+ 136us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:57:56.612452 (+ 62us) write_op.cc:454] Released schema lock
0130 02:57:56.612481 (+ 29us) write_op.cc:341] FINISH: Updating metrics
0130 02:57:56.614312 (+ 1831us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:57:56.614337 (+ 25us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":229,"apply.run_cpu_time_us":1213,"apply.run_wall_time_us":2795,"mutex_wait_us":44304,"num_ops":2,"prepare.queue_time_us":337380,"prepare.run_cpu_time_us":1989,"prepare.run_wall_time_us":23245,"raft.queue_time_us":314,"raft.run_cpu_time_us":786,"raft.run_wall_time_us":1256,"replication_time_us":9970,"spinlock_wait_cycles":1355136}]]}
W20260130 02:57:57.247841 18874 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.18.100.66:59691 (request call id 16) took 1186 ms (client timeout 1000 ms). Trace:
I20260130 02:57:57.248149 19356 tablet_service.cc:1552] Processing DeleteTablet for tablet 01db06fcfffe4b259ac4240c0df38047 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:57:56 UTC) from {username='slave'} at 127.0.0.1:60110
I20260130 02:57:57.250268 19358 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 073b5941850c4140b1751956dd7cf5ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:57:57.250468 19354 tablet_service.cc:1552] Processing DeleteTablet for tablet 3b8cdac4bf4b46efa26236fc930eb9a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:57:54 UTC) from {username='slave'} at 127.0.0.1:60110
I20260130 02:57:57.251210 19758 tablet_replica.cc:333] T 01db06fcfffe4b259ac4240c0df38047 P 78c2c46c7520492db35d51c244fb4f74: stopping tablet replica
W20260130 02:57:57.251891 18874 rpcz_store.cc:269] 0130 02:57:56.061300 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:57:56.061440 (+ 140us) service_pool.cc:225] Handling call
0130 02:57:57.247769 (+1186329us) inbound_call.cc:173] Queueing success response
Metrics: {"reported_tablets":13}
I20260130 02:57:57.255064 19625 tablet_bootstrap.cc:492] T 2112b0402de5464cb1caf2c2abb67d78 P d870164263bb44b6b780a557daea3656: Bootstrap starting.
I20260130 02:57:57.256595 19353 tablet_service.cc:1552] Processing DeleteTablet for tablet 2806ae64d7324b918397c3c6b72ebcfe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:57:55 UTC) from {username='slave'} at 127.0.0.1:60110
I20260130 02:57:57.260370 19625 tablet_bootstrap.cc:654] T 2112b0402de5464cb1caf2c2abb67d78 P d870164263bb44b6b780a557daea3656: Neither blocks nor log segments found. Creating new log.
I20260130 02:57:57.260985 19710 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59068 (request call id 31) took 1139 ms. Trace:
I20260130 02:57:57.261340 19710 rpcz_store.cc:276] 0130 02:57:56.121362 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:57:56.122207 (+ 845us) service_pool.cc:225] Handling call
0130 02:57:56.869872 (+747665us) ts_tablet_manager.cc:1151] Deleting tablet c2a9675427754d0db1578637a37440c6
0130 02:57:56.869920 (+ 48us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0130 02:57:57.051410 (+181490us) tablet_metadata.cc:641] Metadata flushed
0130 02:57:57.174679 (+123269us) tablet_metadata.cc:641] Metadata flushed
0130 02:57:57.260917 (+ 86238us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":747069}
I20260130 02:57:57.266171 19710 tablet_replica.cc:333] T 3b8cdac4bf4b46efa26236fc930eb9a6 P d870164263bb44b6b780a557daea3656: stopping tablet replica
I20260130 02:57:57.267038 19710 raft_consensus.cc:2243] T 3b8cdac4bf4b46efa26236fc930eb9a6 P d870164263bb44b6b780a557daea3656 [term 1 LEADER]: Raft consensus shutting down.
I20260130 02:57:57.271763 18860 catalog_manager.cc:4985] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): tablet c2a9675427754d0db1578637a37440c6 (table default.Table_69ffd00165c6422292d9976b685d5d0d [id=f4829d67e1394959b7a7335fd067a826]) successfully deleted
W20260130 02:57:57.272611 18860 catalog_manager.cc:4952] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): delete failed for tablet 06b31d020b864c418476e6dcc1e4eafb because the tablet was not found. No further retry: Not found: Tablet not found: 06b31d020b864c418476e6dcc1e4eafb
I20260130 02:57:57.282418 19356 tablet_service.cc:1552] Processing DeleteTablet for tablet 06b31d020b864c418476e6dcc1e4eafb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:57:55 UTC) from {username='slave'} at 127.0.0.1:60110
I20260130 02:57:57.252025 19758 raft_consensus.cc:2243] T 01db06fcfffe4b259ac4240c0df38047 P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260130 02:57:57.285939 19758 raft_consensus.cc:2272] T 01db06fcfffe4b259ac4240c0df38047 P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260130 02:57:57.296885 19758 ts_tablet_manager.cc:1916] T 01db06fcfffe4b259ac4240c0df38047 P 78c2c46c7520492db35d51c244fb4f74: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260130 02:57:57.300216 19787 raft_consensus.cc:493] T d4f2b54930cc4f5c825562e3e0a79f4d P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:57:57.300644 19787 raft_consensus.cc:515] T d4f2b54930cc4f5c825562e3e0a79f4d P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:57:57.302438 19787 leader_election.cc:290] T d4f2b54930cc4f5c825562e3e0a79f4d P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), d870164263bb44b6b780a557daea3656 (127.18.100.67:39571)
I20260130 02:57:57.306903 18866 catalog_manager.cc:5941] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4: Sending DeleteTablet for 3 replicas of tablet 30d8136b5ea742bca20ee46b5634d04e
I20260130 02:57:57.308727 19785 tablet.cc:1722] T 38a7a10e36874183bf9fe55a38d829f6 P 78c2c46c7520492db35d51c244fb4f74: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20260130 02:57:57.310168 19791 tablet.cc:1722] T b078934be4bb4fd1915c691eeea9d00f P 78c2c46c7520492db35d51c244fb4f74: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20260130 02:57:57.321372 19356 tablet_service.cc:1552] Processing DeleteTablet for tablet 30d8136b5ea742bca20ee46b5634d04e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:57:56 UTC) from {username='slave'} at 127.0.0.1:60110
I20260130 02:57:57.267832 19710 raft_consensus.cc:2272] T 3b8cdac4bf4b46efa26236fc930eb9a6 P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260130 02:57:57.375976 19840 tablet_bootstrap.cc:492] T 2112b0402de5464cb1caf2c2abb67d78 P 78c2c46c7520492db35d51c244fb4f74: Bootstrap starting.
I20260130 02:57:57.182552 19807 raft_consensus.cc:359] T 4035b8a330844dd28af9f3244bf8eb9a P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:57:57.384538 19807 raft_consensus.cc:385] T 4035b8a330844dd28af9f3244bf8eb9a P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:57:57.384876 19807 raft_consensus.cc:740] T 4035b8a330844dd28af9f3244bf8eb9a P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 371a36d219164974abdf848841528152, State: Initialized, Role: FOLLOWER
I20260130 02:57:57.385529 19807 consensus_queue.cc:260] T 4035b8a330844dd28af9f3244bf8eb9a P 371a36d219164974abdf848841528152 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:57:57.390038 19840 tablet_bootstrap.cc:654] T 2112b0402de5464cb1caf2c2abb67d78 P 78c2c46c7520492db35d51c244fb4f74: Neither blocks nor log segments found. Creating new log.
I20260130 02:57:57.392585 19221 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d4f2b54930cc4f5c825562e3e0a79f4d" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152" is_pre_election: true
W20260130 02:57:57.395928 19314 leader_election.cc:343] T d4f2b54930cc4f5c825562e3e0a79f4d P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:57:57.419238 19204 tablet_service.cc:1505] Processing CreateTablet for tablet 2112b0402de5464cb1caf2c2abb67d78 (DEFAULT_TABLE table=default.Table_da3d581b15a94840898777f74611994e [id=b7e241ced53e4160ac6fa1f65ef3674e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:57:57.418144 19205 tablet_service.cc:1505] Processing CreateTablet for tablet 073b5941850c4140b1751956dd7cf5ef (DEFAULT_TABLE table=default.Table_6eb0a92b1926487cbf0b978d26d6a7b1 [id=de7a420d40864f55b156e149308ba12f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:57:57.420965 19204 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2112b0402de5464cb1caf2c2abb67d78. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:57:57.421687 19205 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 073b5941850c4140b1751956dd7cf5ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:57:57.425225 19202 tablet_service.cc:1552] Processing DeleteTablet for tablet 30d8136b5ea742bca20ee46b5634d04e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:57:56 UTC) from {username='slave'} at 127.0.0.1:42870
I20260130 02:57:57.428218 19203 tablet_service.cc:1552] Processing DeleteTablet for tablet 01db06fcfffe4b259ac4240c0df38047 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:57:56 UTC) from {username='slave'} at 127.0.0.1:42870
I20260130 02:57:57.430894 19201 tablet_service.cc:1505] Processing CreateTablet for tablet 0b1877e5f9244913bcd639f1336df7ee (DEFAULT_TABLE table=default.Table_19677b801d1f4a63b22139b9fc80c4ef [id=3992cda3a7bb49ae86b8f84ee12f1c8b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:57:57.431973 19201 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0b1877e5f9244913bcd639f1336df7ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:57:57.437247 18871 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50284:
table { table_name: "default.Table_a8b2bef1c4564242b8d5fc95a3a35f85" } new_table_name: "default.Table_a31136243e6241a19ae437271d1cf283" modify_external_catalogs: true
I20260130 02:57:57.272681 19506 tablet_service.cc:1552] Processing DeleteTablet for tablet 2806ae64d7324b918397c3c6b72ebcfe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:57:55 UTC) from {username='slave'} at 127.0.0.1:59068
I20260130 02:57:57.345538 19794 tablet.cc:1722] T b078934be4bb4fd1915c691eeea9d00f P d870164263bb44b6b780a557daea3656: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20260130 02:57:57.309931 19504 tablet_service.cc:1552] Processing DeleteTablet for tablet 30d8136b5ea742bca20ee46b5634d04e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:57:56 UTC) from {username='slave'} at 127.0.0.1:59068
I20260130 02:57:57.289580 19625 tablet_bootstrap.cc:492] T 2112b0402de5464cb1caf2c2abb67d78 P d870164263bb44b6b780a557daea3656: No bootstrap required, opened a new log
I20260130 02:57:57.443125 19625 ts_tablet_manager.cc:1403] T 2112b0402de5464cb1caf2c2abb67d78 P d870164263bb44b6b780a557daea3656: Time spent bootstrapping tablet: real 0.188s user 0.009s sys 0.026s
I20260130 02:57:57.445441 19625 raft_consensus.cc:359] T 2112b0402de5464cb1caf2c2abb67d78 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:57:57.446040 19625 raft_consensus.cc:385] T 2112b0402de5464cb1caf2c2abb67d78 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:57:57.446280 19625 raft_consensus.cc:740] T 2112b0402de5464cb1caf2c2abb67d78 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d870164263bb44b6b780a557daea3656, State: Initialized, Role: FOLLOWER
I20260130 02:57:57.446779 18872 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50284:
table { table_name: "default.Table_604228ee4a914be8a2f2f49f2f51e3f3" } new_table_name: "default.Table_b63ca4e4060f48e893b0099fb6433f04" modify_external_catalogs: true
I20260130 02:57:57.446861 19625 consensus_queue.cc:260] T 2112b0402de5464cb1caf2c2abb67d78 P d870164263bb44b6b780a557daea3656 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:57:57.448786 19625 ts_tablet_manager.cc:1434] T 2112b0402de5464cb1caf2c2abb67d78 P d870164263bb44b6b780a557daea3656: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260130 02:57:57.448814 19807 ts_tablet_manager.cc:1434] T 4035b8a330844dd28af9f3244bf8eb9a P 371a36d219164974abdf848841528152: Time spent starting tablet: real 0.272s user 0.006s sys 0.001s
I20260130 02:57:57.449702 19807 tablet_bootstrap.cc:492] T d4f2b54930cc4f5c825562e3e0a79f4d P 371a36d219164974abdf848841528152: Bootstrap starting.
I20260130 02:57:57.452728 19625 tablet_bootstrap.cc:492] T 0b1877e5f9244913bcd639f1336df7ee P d870164263bb44b6b780a557daea3656: Bootstrap starting.
I20260130 02:57:57.455008 18867 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50284:
table { table_name: "default.Table_298b1af220674b6ab20d3652e8a637a1" } modify_external_catalogs: true
I20260130 02:57:57.455551 18867 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50284:
table { table_name: "default.Table_298b1af220674b6ab20d3652e8a637a1" } modify_external_catalogs: true
I20260130 02:57:57.456143 19200 tablet_service.cc:1552] Processing DeleteTablet for tablet 2806ae64d7324b918397c3c6b72ebcfe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:57:55 UTC) from {username='slave'} at 127.0.0.1:42870
I20260130 02:57:57.456738 19202 tablet_service.cc:1552] Processing DeleteTablet for tablet c2a9675427754d0db1578637a37440c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:57:56 UTC) from {username='slave'} at 127.0.0.1:42870
I20260130 02:57:57.457958 19625 tablet_bootstrap.cc:654] T 0b1877e5f9244913bcd639f1336df7ee P d870164263bb44b6b780a557daea3656: Neither blocks nor log segments found. Creating new log.
I20260130 02:57:57.307318 19528 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d4f2b54930cc4f5c825562e3e0a79f4d" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
I20260130 02:57:57.467571 19528 raft_consensus.cc:2468] T d4f2b54930cc4f5c825562e3e0a79f4d P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78c2c46c7520492db35d51c244fb4f74 in term 0.
I20260130 02:57:57.284922 19505 tablet_service.cc:1552] Processing DeleteTablet for tablet 01db06fcfffe4b259ac4240c0df38047 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:57:56 UTC) from {username='slave'} at 127.0.0.1:59068
I20260130 02:57:57.292025 19788 tablet.cc:1722] T 38a7a10e36874183bf9fe55a38d829f6 P d870164263bb44b6b780a557daea3656: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
W20260130 02:57:57.483095 18869 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.18.100.65:51021 (request call id 20) took 1019 ms (client timeout 1000 ms). Trace:
I20260130 02:57:57.486961 19710 ts_tablet_manager.cc:1916] T 3b8cdac4bf4b46efa26236fc930eb9a6 P d870164263bb44b6b780a557daea3656: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260130 02:57:57.487212 18869 rpcz_store.cc:269] 0130 02:57:56.463381 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:57:56.463523 (+ 142us) service_pool.cc:225] Handling call
0130 02:57:56.510816 (+ 47293us) spinlock_profiling.cc:243] Waited 44.3 ms on lock 0x7b5400030554. stack: 00007fb0bd3f7373 00007fb0bd3f7270 00007fb0bcf6c574 0000000000508744 00000000005086d9 00007fb0c44490c0 00007fb0c44e7bba 00007fb0c61e6eac 00007fb0c62e110e 00007fb0c3efe4a4 00007fb0c3efe432 00007fb0c3efe361 00007fb0c3efe2dc 00007fb0c3efd592 00007fb0c21dd34c 00007fb0c21dc786
0130 02:57:57.483024 (+972208us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0130 02:57:57.331099 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0130 02:57:57.331492 (+ 393us) write_op.cc:432] Acquiring schema lock in shared mode
0130 02:57:57.331515 (+ 23us) write_op.cc:435] Acquired schema lock
0130 02:57:57.331526 (+ 11us) tablet.cc:660] Decoding operations
0130 02:57:57.331788 (+ 262us) write_op.cc:620] Acquiring the partition lock for write op
0130 02:57:57.331825 (+ 37us) write_op.cc:641] Partition lock acquired for write op
0130 02:57:57.331847 (+ 22us) tablet.cc:683] Acquiring locks for 1 operations
0130 02:57:57.331985 (+ 138us) tablet.cc:699] Row locks acquired
0130 02:57:57.331999 (+ 14us) write_op.cc:260] PREPARE: finished
0130 02:57:57.332109 (+ 110us) write_op.cc:270] Start()
0130 02:57:57.332191 (+ 82us) write_op.cc:276] Timestamp: P: 1769741877332074 usec, L: 0
0130 02:57:57.332206 (+ 15us) op_driver.cc:348] REPLICATION: starting
0130 02:57:57.332704 (+ 498us) log.cc:844] Serialized 532 byte log entry
0130 02:57:57.414154 (+ 81450us) op_driver.cc:464] REPLICATION: finished
0130 02:57:57.416253 (+ 2099us) write_op.cc:301] APPLY: starting
0130 02:57:57.416301 (+ 48us) tablet.cc:1365] starting BulkCheckPresence
0130 02:57:57.416382 (+ 81us) tablet.cc:1368] finished BulkCheckPresence
0130 02:57:57.416462 (+ 80us) tablet.cc:1370] starting ApplyRowOperation cycle
0130 02:57:57.416594 (+ 132us) tablet.cc:1381] finished ApplyRowOperation cycle
0130 02:57:57.416720 (+ 126us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0130 02:57:57.416760 (+ 40us) write_op.cc:312] APPLY: finished
0130 02:57:57.416907 (+ 147us) log.cc:844] Serialized 22 byte log entry
0130 02:57:57.417011 (+ 104us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:57:57.417067 (+ 56us) write_op.cc:454] Released schema lock
0130 02:57:57.417097 (+ 30us) write_op.cc:341] FINISH: Updating metrics
0130 02:57:57.417299 (+ 202us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:57:57.417322 (+ 23us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":16,"spinlock_wait_cycles":101969280,"child_traces":[["op",{"apply.queue_time_us":1971,"apply.run_cpu_time_us":1023,"apply.run_wall_time_us":1023,"num_ops":1,"prepare.queue_time_us":14636,"prepare.run_cpu_time_us":1889,"prepare.run_wall_time_us":1890,"raft.queue_time_us":91,"raft.run_cpu_time_us":13,"raft.run_wall_time_us":15,"replication_time_us":81882}]]}
I20260130 02:57:57.486500 19625 tablet_bootstrap.cc:492] T 0b1877e5f9244913bcd639f1336df7ee P d870164263bb44b6b780a557daea3656: No bootstrap required, opened a new log
I20260130 02:57:57.491072 19625 ts_tablet_manager.cc:1403] T 0b1877e5f9244913bcd639f1336df7ee P d870164263bb44b6b780a557daea3656: Time spent bootstrapping tablet: real 0.039s user 0.009s sys 0.004s
I20260130 02:57:57.493410 19625 raft_consensus.cc:359] T 0b1877e5f9244913bcd639f1336df7ee P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:57:57.494083 19625 raft_consensus.cc:385] T 0b1877e5f9244913bcd639f1336df7ee P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:57:57.494331 19625 raft_consensus.cc:740] T 0b1877e5f9244913bcd639f1336df7ee P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d870164263bb44b6b780a557daea3656, State: Initialized, Role: FOLLOWER
I20260130 02:57:57.503513 19625 consensus_queue.cc:260] T 0b1877e5f9244913bcd639f1336df7ee P d870164263bb44b6b780a557daea3656 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
W20260130 02:57:57.505841 19314 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.100.65:42021, user_credentials={real_user=slave}} blocked reactor thread for 54505.4us
W20260130 02:57:57.506706 19313 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.100.67:39571, user_credentials={real_user=slave}} blocked reactor thread for 126912us
I20260130 02:57:57.508062 19787 raft_consensus.cc:493] T ceda64471a92476f9ba183d2beab30d1 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:57:57.508481 19787 raft_consensus.cc:515] T ceda64471a92476f9ba183d2beab30d1 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:57:57.510164 19787 leader_election.cc:290] T ceda64471a92476f9ba183d2beab30d1 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), d870164263bb44b6b780a557daea3656 (127.18.100.67:39571)
I20260130 02:57:57.510836 19313 leader_election.cc:304] T d4f2b54930cc4f5c825562e3e0a79f4d P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 78c2c46c7520492db35d51c244fb4f74, d870164263bb44b6b780a557daea3656; no voters: 371a36d219164974abdf848841528152
I20260130 02:57:57.511512 19625 ts_tablet_manager.cc:1434] T 0b1877e5f9244913bcd639f1336df7ee P d870164263bb44b6b780a557daea3656: Time spent starting tablet: real 0.020s user 0.006s sys 0.000s
I20260130 02:57:57.512651 19528 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ceda64471a92476f9ba183d2beab30d1" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
I20260130 02:57:57.513247 19528 raft_consensus.cc:2468] T ceda64471a92476f9ba183d2beab30d1 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78c2c46c7520492db35d51c244fb4f74 in term 0.
I20260130 02:57:57.514243 19787 raft_consensus.cc:2804] T d4f2b54930cc4f5c825562e3e0a79f4d P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260130 02:57:57.514592 19787 raft_consensus.cc:493] T d4f2b54930cc4f5c825562e3e0a79f4d P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260130 02:57:57.514907 19787 raft_consensus.cc:3060] T d4f2b54930cc4f5c825562e3e0a79f4d P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:57:57.515008 18869 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50284:
table { table_name: "default.Table_1c77605822574b10a05074a093ad30a3" } modify_external_catalogs: true
I20260130 02:57:57.515566 18869 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50284:
table { table_name: "default.Table_1c77605822574b10a05074a093ad30a3" } modify_external_catalogs: true
I20260130 02:57:57.580715 19203 tablet_service.cc:1552] Processing DeleteTablet for tablet 3b8cdac4bf4b46efa26236fc930eb9a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:57:54 UTC) from {username='slave'} at 127.0.0.1:42870
I20260130 02:57:57.592640 18874 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50284:
name: "default.Table_9cb9a9f9e5d145a6a42d7ccbc42aced9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260130 02:57:57.594344 18874 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9cb9a9f9e5d145a6a42d7ccbc42aced9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260130 02:57:57.598644 18875 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50284:
name: "default.Table_b2d2da7aae8b430aba32bac2944aa931"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260130 02:57:57.598861 18873 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50284:
name: "default.Table_f7aab7c003284493810119a94032eca0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260130 02:57:57.600276 18875 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b2d2da7aae8b430aba32bac2944aa931 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260130 02:57:57.600440 18873 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f7aab7c003284493810119a94032eca0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260130 02:57:57.602807 19635 consensus_queue.cc:1048] T 37470ce4356f4542b621e21698551209 P 371a36d219164974abdf848841528152 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260130 02:57:57.608019 18859 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet 37470ce4356f4542b621e21698551209 (table default.Table_007e969f70d345a7bcb3dfbaabbc025d [id=77d4fa8705a3419cb0a84d3ca1bb4ce3], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 37470ce4356f4542b621e21698551209 consensus state has no leader
I20260130 02:57:57.616168 19313 leader_election.cc:304] T ceda64471a92476f9ba183d2beab30d1 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78c2c46c7520492db35d51c244fb4f74, d870164263bb44b6b780a557daea3656; no voters:
I20260130 02:57:57.619040 19663 raft_consensus.cc:2804] T ceda64471a92476f9ba183d2beab30d1 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260130 02:57:57.619388 19663 raft_consensus.cc:493] T ceda64471a92476f9ba183d2beab30d1 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260130 02:57:57.619726 19663 raft_consensus.cc:3060] T ceda64471a92476f9ba183d2beab30d1 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:57:57.629694 18868 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50284:
name: "default.Table_a78ce7ab85924506b184fb74db3ed0c3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260130 02:57:57.631392 18868 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a78ce7ab85924506b184fb74db3ed0c3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260130 02:57:57.631574 19221 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ceda64471a92476f9ba183d2beab30d1" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152" is_pre_election: true
I20260130 02:57:57.633332 19813 tablet.cc:1722] T b078934be4bb4fd1915c691eeea9d00f P 371a36d219164974abdf848841528152: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20260130 02:57:57.635264 19357 tablet_service.cc:1505] Processing CreateTablet for tablet 1bbc55bbf2254d8bb6b721d7e8f69d85 (DEFAULT_TABLE table=default.Table_0658e13596f44598bb5a108797bc06fc [id=b16908b75587459597a978c71756451c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:57:57.636327 19357 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1bbc55bbf2254d8bb6b721d7e8f69d85. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:57:57.622268 19787 raft_consensus.cc:515] T d4f2b54930cc4f5c825562e3e0a79f4d P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:57:57.637555 19625 tablet_bootstrap.cc:492] T 073b5941850c4140b1751956dd7cf5ef P d870164263bb44b6b780a557daea3656: Bootstrap starting.
W20260130 02:57:57.638358 19314 leader_election.cc:343] T ceda64471a92476f9ba183d2beab30d1 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:57:57.638686 19787 leader_election.cc:290] T d4f2b54930cc4f5c825562e3e0a79f4d P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 election: Requested vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), d870164263bb44b6b780a557daea3656 (127.18.100.67:39571)
I20260130 02:57:57.640646 19528 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d4f2b54930cc4f5c825562e3e0a79f4d" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656"
I20260130 02:57:57.641171 19528 raft_consensus.cc:3060] T d4f2b54930cc4f5c825562e3e0a79f4d P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:57:57.641634 19758 ts_tablet_manager.cc:1929] T 01db06fcfffe4b259ac4240c0df38047 P 78c2c46c7520492db35d51c244fb4f74: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260130 02:57:57.642064 19758 log.cc:1199] T 01db06fcfffe4b259ac4240c0df38047 P 78c2c46c7520492db35d51c244fb4f74: Deleting WAL directory at /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-1/wal/wals/01db06fcfffe4b259ac4240c0df38047
I20260130 02:57:57.642697 19758 ts_tablet_manager.cc:1950] T 01db06fcfffe4b259ac4240c0df38047 P 78c2c46c7520492db35d51c244fb4f74: Deleting consensus metadata
I20260130 02:57:57.645392 19758 tablet_replica.cc:333] T 3b8cdac4bf4b46efa26236fc930eb9a6 P 78c2c46c7520492db35d51c244fb4f74: stopping tablet replica
I20260130 02:57:57.646221 19758 raft_consensus.cc:2243] T 3b8cdac4bf4b46efa26236fc930eb9a6 P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260130 02:57:57.646730 19758 raft_consensus.cc:2272] T 3b8cdac4bf4b46efa26236fc930eb9a6 P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260130 02:57:57.648258 18860 catalog_manager.cc:4985] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767): tablet 01db06fcfffe4b259ac4240c0df38047 (table default.Table_e67bd1859b5846f386097cf33c5f1bcb [id=952ccb90477f47e4a352623247a8a6f2]) successfully deleted
I20260130 02:57:57.649788 18866 catalog_manager.cc:5746] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767) does not have the latest schema for tablet 37470ce4356f4542b621e21698551209 (table default.Table_007e969f70d345a7bcb3dfbaabbc025d [id=77d4fa8705a3419cb0a84d3ca1bb4ce3]). Expected version 1 got 0
I20260130 02:57:57.651422 19663 raft_consensus.cc:515] T ceda64471a92476f9ba183d2beab30d1 P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:57:57.653234 19663 leader_election.cc:290] T ceda64471a92476f9ba183d2beab30d1 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 election: Requested vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), d870164263bb44b6b780a557daea3656 (127.18.100.67:39571)
I20260130 02:57:57.657127 19807 tablet_bootstrap.cc:654] T d4f2b54930cc4f5c825562e3e0a79f4d P 371a36d219164974abdf848841528152: Neither blocks nor log segments found. Creating new log.
I20260130 02:57:57.658185 18866 catalog_manager.cc:5746] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767) does not have the latest schema for tablet 38a7a10e36874183bf9fe55a38d829f6 (table default.Table_4d43592dc5b24ff28710e62ad4d87ce0 [id=27adf356fd004597a54c7a9887dd3ca8]). Expected version 3 got 2
I20260130 02:57:57.659255 19358 tablet_service.cc:1505] Processing CreateTablet for tablet ffa1142fc7274aecbdbee1a0802b95da (DEFAULT_TABLE table=default.Table_2f60fd0292ab40c8a3ae9b842cae8faf [id=e648e521ef5044a0b1967452cc3a31b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:57:57.659960 19526 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ceda64471a92476f9ba183d2beab30d1" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656"
I20260130 02:57:57.660337 19358 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ffa1142fc7274aecbdbee1a0802b95da. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:57:57.660499 19526 raft_consensus.cc:3060] T ceda64471a92476f9ba183d2beab30d1 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:57:57.669194 19758 ts_tablet_manager.cc:1916] T 3b8cdac4bf4b46efa26236fc930eb9a6 P 78c2c46c7520492db35d51c244fb4f74: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260130 02:57:57.669740 19505 tablet_service.cc:1505] Processing CreateTablet for tablet 1bbc55bbf2254d8bb6b721d7e8f69d85 (DEFAULT_TABLE table=default.Table_0658e13596f44598bb5a108797bc06fc [id=b16908b75587459597a978c71756451c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:57:57.670805 19505 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1bbc55bbf2254d8bb6b721d7e8f69d85. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:57:57.675349 19504 tablet_service.cc:1505] Processing CreateTablet for tablet ffa1142fc7274aecbdbee1a0802b95da (DEFAULT_TABLE table=default.Table_2f60fd0292ab40c8a3ae9b842cae8faf [id=e648e521ef5044a0b1967452cc3a31b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:57:57.676613 19811 raft_consensus.cc:493] T 0b1877e5f9244913bcd639f1336df7ee P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:57:57.680853 19811 raft_consensus.cc:515] T 0b1877e5f9244913bcd639f1336df7ee P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:57:57.688306 19204 tablet_service.cc:1505] Processing CreateTablet for tablet 1bbc55bbf2254d8bb6b721d7e8f69d85 (DEFAULT_TABLE table=default.Table_0658e13596f44598bb5a108797bc06fc [id=b16908b75587459597a978c71756451c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:57:57.692982 19742 consensus_queue.cc:1048] T 01db06fcfffe4b259ac4240c0df38047 P d870164263bb44b6b780a557daea3656 [LEADER]: Connected to new peer: Peer: permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260130 02:57:57.695487 19204 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1bbc55bbf2254d8bb6b721d7e8f69d85. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:57:57.697047 19222 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d4f2b54930cc4f5c825562e3e0a79f4d" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152"
I20260130 02:57:57.697824 19225 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ceda64471a92476f9ba183d2beab30d1" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152"
W20260130 02:57:57.698971 19314 leader_election.cc:343] T d4f2b54930cc4f5c825562e3e0a79f4d P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Illegal state: must be running to vote when last-logged opid is not known
W20260130 02:57:57.699642 19314 leader_election.cc:343] T ceda64471a92476f9ba183d2beab30d1 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:57:57.702219 19504 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ffa1142fc7274aecbdbee1a0802b95da. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:57:57.702843 19223 raft_consensus.cc:3060] T 4035b8a330844dd28af9f3244bf8eb9a P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Advancing to term 1
W20260130 02:57:57.715796 19463 leader_election.cc:343] T 0b1877e5f9244913bcd639f1336df7ee P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Not found: Tablet not found: 0b1877e5f9244913bcd639f1336df7ee
I20260130 02:57:57.717118 19679 consensus_queue.cc:1048] T 37470ce4356f4542b621e21698551209 P 371a36d219164974abdf848841528152 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260130 02:57:57.719717 19679 consensus_queue.cc:1048] T 878f9346d3044d8d8b15c61f86869212 P 371a36d219164974abdf848841528152 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260130 02:57:57.704022 19205 tablet_service.cc:1505] Processing CreateTablet for tablet ffa1142fc7274aecbdbee1a0802b95da (DEFAULT_TABLE table=default.Table_2f60fd0292ab40c8a3ae9b842cae8faf [id=e648e521ef5044a0b1967452cc3a31b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:57:57.720643 19811 leader_election.cc:290] T 0b1877e5f9244913bcd639f1336df7ee P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767)
I20260130 02:57:57.720942 19205 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ffa1142fc7274aecbdbee1a0802b95da. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:57:57.724290 19528 raft_consensus.cc:2468] T d4f2b54930cc4f5c825562e3e0a79f4d P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78c2c46c7520492db35d51c244fb4f74 in term 1.
I20260130 02:57:57.725446 19313 leader_election.cc:304] T d4f2b54930cc4f5c825562e3e0a79f4d P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 78c2c46c7520492db35d51c244fb4f74, d870164263bb44b6b780a557daea3656; no voters: 371a36d219164974abdf848841528152
I20260130 02:57:57.726517 19787 raft_consensus.cc:2804] T d4f2b54930cc4f5c825562e3e0a79f4d P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Leader election won for term 1
I20260130 02:57:57.726972 19787 raft_consensus.cc:697] T d4f2b54930cc4f5c825562e3e0a79f4d P 78c2c46c7520492db35d51c244fb4f74 [term 1 LEADER]: Becoming Leader. State: Replica: 78c2c46c7520492db35d51c244fb4f74, State: Running, Role: LEADER
I20260130 02:57:57.727701 19787 consensus_queue.cc:237] T d4f2b54930cc4f5c825562e3e0a79f4d P 78c2c46c7520492db35d51c244fb4f74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:57:57.732686 19377 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0b1877e5f9244913bcd639f1336df7ee" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74" is_pre_election: true
I20260130 02:57:57.734529 19625 tablet_bootstrap.cc:654] T 073b5941850c4140b1751956dd7cf5ef P d870164263bb44b6b780a557daea3656: Neither blocks nor log segments found. Creating new log.
I20260130 02:57:57.738703 19754 ts_tablet_manager.cc:1929] T 79e28c3a566c4521ac2453bbaa3567ed P 371a36d219164974abdf848841528152: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260130 02:57:57.739151 19754 log.cc:1199] T 79e28c3a566c4521ac2453bbaa3567ed P 371a36d219164974abdf848841528152: Deleting WAL directory at /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-0/wal/wals/79e28c3a566c4521ac2453bbaa3567ed
I20260130 02:57:57.739876 19754 ts_tablet_manager.cc:1950] T 79e28c3a566c4521ac2453bbaa3567ed P 371a36d219164974abdf848841528152: Deleting consensus metadata
I20260130 02:57:57.740809 19754 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42870 (request call id 38) took 1451 ms. Trace:
I20260130 02:57:57.741210 19754 rpcz_store.cc:276] 0130 02:57:56.289833 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:57:56.299510 (+ 9677us) service_pool.cc:225] Handling call
0130 02:57:57.142440 (+842930us) ts_tablet_manager.cc:1151] Deleting tablet 79e28c3a566c4521ac2453bbaa3567ed
0130 02:57:57.142506 (+ 66us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0130 02:57:57.592701 (+450195us) tablet_metadata.cc:641] Metadata flushed
0130 02:57:57.738411 (+145710us) tablet_metadata.cc:641] Metadata flushed
0130 02:57:57.740751 (+ 2340us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":842247}
W20260130 02:57:57.744551 19462 leader_election.cc:343] T 0b1877e5f9244913bcd639f1336df7ee P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:57:57.745131 19462 leader_election.cc:304] T 0b1877e5f9244913bcd639f1336df7ee P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d870164263bb44b6b780a557daea3656; no voters: 371a36d219164974abdf848841528152, 78c2c46c7520492db35d51c244fb4f74
I20260130 02:57:57.748039 19506 tablet_service.cc:1505] Processing CreateTablet for tablet c3fce8cc99fb451ebd532fa37390e862 (DEFAULT_TABLE table=default.Table_5d29239ce1984d2fad02607a2bdd4dd0 [id=7e271b81bb064a949685b2411b485e3e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:57:57.749075 19506 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c3fce8cc99fb451ebd532fa37390e862. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:57:57.749322 19360 tablet_service.cc:1505] Processing CreateTablet for tablet c3fce8cc99fb451ebd532fa37390e862 (DEFAULT_TABLE table=default.Table_5d29239ce1984d2fad02607a2bdd4dd0 [id=7e271b81bb064a949685b2411b485e3e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:57:57.750209 19360 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c3fce8cc99fb451ebd532fa37390e862. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:57:57.698418 19526 raft_consensus.cc:2468] T ceda64471a92476f9ba183d2beab30d1 P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78c2c46c7520492db35d51c244fb4f74 in term 1.
I20260130 02:57:57.752466 19201 tablet_service.cc:1505] Processing CreateTablet for tablet c3fce8cc99fb451ebd532fa37390e862 (DEFAULT_TABLE table=default.Table_5d29239ce1984d2fad02607a2bdd4dd0 [id=7e271b81bb064a949685b2411b485e3e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:57:57.755026 19203 tablet_service.cc:1505] Processing CreateTablet for tablet e364f4e70e5749e083e14e7fa7416ff0 (DEFAULT_TABLE table=default.Table_495035e92963454c8690f28fb4557c00 [id=df7d3d68e36142ff8564e63ffe0b8797]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:57:57.759615 19203 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e364f4e70e5749e083e14e7fa7416ff0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:57:57.762382 19356 tablet_service.cc:1505] Processing CreateTablet for tablet e364f4e70e5749e083e14e7fa7416ff0 (DEFAULT_TABLE table=default.Table_495035e92963454c8690f28fb4557c00 [id=df7d3d68e36142ff8564e63ffe0b8797]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:57:57.763509 19356 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e364f4e70e5749e083e14e7fa7416ff0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:57:57.763543 19201 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c3fce8cc99fb451ebd532fa37390e862. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:57:57.769480 19663 consensus_queue.cc:1048] T 4035b8a330844dd28af9f3244bf8eb9a P 78c2c46c7520492db35d51c244fb4f74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260130 02:57:57.769853 19507 tablet_service.cc:1505] Processing CreateTablet for tablet e364f4e70e5749e083e14e7fa7416ff0 (DEFAULT_TABLE table=default.Table_495035e92963454c8690f28fb4557c00 [id=df7d3d68e36142ff8564e63ffe0b8797]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:57:57.770833 19507 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e364f4e70e5749e083e14e7fa7416ff0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:57:57.787454 18861 catalog_manager.cc:4985] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021): tablet 79e28c3a566c4521ac2453bbaa3567ed (table default.Table_b8f4e9cf27f44a479e1adad0b7ce4e33 [id=26422816af0c430880cb44e1b0cce534]) successfully deleted
I20260130 02:57:57.793975 19353 tablet_service.cc:1505] Processing CreateTablet for tablet 7a902d0d571a42bb82ebffd48f7eca3f (DEFAULT_TABLE table=default.Table_298b1af220674b6ab20d3652e8a637a1 [id=742a870a99554fc48fb4f369e68fc6d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:57:57.795220 19353 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7a902d0d571a42bb82ebffd48f7eca3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:57:57.806102 19754 tablet_replica.cc:333] T 30d8136b5ea742bca20ee46b5634d04e P 371a36d219164974abdf848841528152: stopping tablet replica
I20260130 02:57:57.806963 19754 raft_consensus.cc:2243] T 30d8136b5ea742bca20ee46b5634d04e P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260130 02:57:57.807463 19754 raft_consensus.cc:2272] T 30d8136b5ea742bca20ee46b5634d04e P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260130 02:57:57.783918 19811 raft_consensus.cc:2749] T 0b1877e5f9244913bcd639f1336df7ee P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260130 02:57:57.814031 19313 leader_election.cc:304] T ceda64471a92476f9ba183d2beab30d1 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 78c2c46c7520492db35d51c244fb4f74, d870164263bb44b6b780a557daea3656; no voters: 371a36d219164974abdf848841528152
I20260130 02:57:57.816105 19508 tablet_service.cc:1505] Processing CreateTablet for tablet 7a902d0d571a42bb82ebffd48f7eca3f (DEFAULT_TABLE table=default.Table_298b1af220674b6ab20d3652e8a637a1 [id=742a870a99554fc48fb4f369e68fc6d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:57:57.817214 19508 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7a902d0d571a42bb82ebffd48f7eca3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:57:57.782691 18866 catalog_manager.cc:5654] T 4035b8a330844dd28af9f3244bf8eb9a P 78c2c46c7520492db35d51c244fb4f74 reported cstate change: term changed from 0 to 1, leader changed from <none> to 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66). New cstate: current_term: 1 leader_uuid: "78c2c46c7520492db35d51c244fb4f74" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } health_report { overall_health: UNKNOWN } } }
I20260130 02:57:57.833740 18866 catalog_manager.cc:5746] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767) does not have the latest schema for tablet ceda64471a92476f9ba183d2beab30d1 (table default.Table_90ad2e09fa9b4952a1564cee70ab8c1c [id=4f59d133b0cd4773bb012145927a92a4]). Expected version 1 got 0
I20260130 02:57:57.834460 19200 tablet_service.cc:1505] Processing CreateTablet for tablet 15918c90ffdc43a5a38b2950e9834188 (DEFAULT_TABLE table=default.Table_b87dd1ee48f143538e487968bbea94bb [id=8b7db40afb894ae99625a756c75ef1de]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:57:57.835678 19200 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 15918c90ffdc43a5a38b2950e9834188. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:57:57.792487 19202 tablet_service.cc:1505] Processing CreateTablet for tablet 7a902d0d571a42bb82ebffd48f7eca3f (DEFAULT_TABLE table=default.Table_298b1af220674b6ab20d3652e8a637a1 [id=742a870a99554fc48fb4f369e68fc6d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:57:57.845613 19202 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7a902d0d571a42bb82ebffd48f7eca3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:57:57.858920 19625 tablet_bootstrap.cc:492] T 073b5941850c4140b1751956dd7cf5ef P d870164263bb44b6b780a557daea3656: No bootstrap required, opened a new log
I20260130 02:57:57.859416 19625 ts_tablet_manager.cc:1403] T 073b5941850c4140b1751956dd7cf5ef P d870164263bb44b6b780a557daea3656: Time spent bootstrapping tablet: real 0.222s user 0.006s sys 0.008s
I20260130 02:57:57.859647 19813 tablet.cc:1722] T 38a7a10e36874183bf9fe55a38d829f6 P 371a36d219164974abdf848841528152: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20260130 02:57:57.861757 19625 raft_consensus.cc:359] T 073b5941850c4140b1751956dd7cf5ef P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:57:57.857266 18867 catalog_manager.cc:5941] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4: Sending DeleteTablet for 3 replicas of tablet 7a902d0d571a42bb82ebffd48f7eca3f
I20260130 02:57:57.863595 19754 ts_tablet_manager.cc:1916] T 30d8136b5ea742bca20ee46b5634d04e P 371a36d219164974abdf848841528152: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260130 02:57:57.866950 19503 tablet_service.cc:1552] Processing DeleteTablet for tablet 7a902d0d571a42bb82ebffd48f7eca3f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:57:57 UTC) from {username='slave'} at 127.0.0.1:59068
I20260130 02:57:57.867655 19199 tablet_service.cc:1552] Processing DeleteTablet for tablet 7a902d0d571a42bb82ebffd48f7eca3f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:57:57 UTC) from {username='slave'} at 127.0.0.1:42870
W20260130 02:57:57.872408 18871 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 878f9346d3044d8d8b15c61f86869212 consensus state has no leader
I20260130 02:57:57.882680 19508 tablet_service.cc:1505] Processing CreateTablet for tablet 15918c90ffdc43a5a38b2950e9834188 (DEFAULT_TABLE table=default.Table_b87dd1ee48f143538e487968bbea94bb [id=8b7db40afb894ae99625a756c75ef1de]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:57:57.883771 19508 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 15918c90ffdc43a5a38b2950e9834188. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:57:57.896274 19807 tablet_bootstrap.cc:492] T d4f2b54930cc4f5c825562e3e0a79f4d P 371a36d219164974abdf848841528152: No bootstrap required, opened a new log
I20260130 02:57:57.896755 19807 ts_tablet_manager.cc:1403] T d4f2b54930cc4f5c825562e3e0a79f4d P 371a36d219164974abdf848841528152: Time spent bootstrapping tablet: real 0.447s user 0.017s sys 0.000s
I20260130 02:57:57.896576 19199 tablet_service.cc:1505] Processing CreateTablet for tablet d701152960b14396a6ac1bd10480d2b6 (DEFAULT_TABLE table=default.Table_e3b63d7d5d874900aac850654d98d62e [id=bc8cc55916fa4a41b78019f59e6fbe39]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:57:57.897675 19199 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d701152960b14396a6ac1bd10480d2b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:57:57.904877 19807 raft_consensus.cc:359] T d4f2b54930cc4f5c825562e3e0a79f4d P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:57:57.905743 19807 raft_consensus.cc:385] T d4f2b54930cc4f5c825562e3e0a79f4d P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:57:57.906030 19807 raft_consensus.cc:740] T d4f2b54930cc4f5c825562e3e0a79f4d P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 371a36d219164974abdf848841528152, State: Initialized, Role: FOLLOWER
I20260130 02:57:57.913129 19504 tablet_service.cc:1505] Processing CreateTablet for tablet d701152960b14396a6ac1bd10480d2b6 (DEFAULT_TABLE table=default.Table_e3b63d7d5d874900aac850654d98d62e [id=bc8cc55916fa4a41b78019f59e6fbe39]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:57:57.914326 19504 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d701152960b14396a6ac1bd10480d2b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:57:57.942117 19204 tablet_service.cc:1505] Processing CreateTablet for tablet e026dee22493424692d00f14848ab57c (DEFAULT_TABLE table=default.Table_85611716686c4097bacb64df677d5793 [id=122fe719b1d64ec8b531783539f85df1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:57:57.943121 19204 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e026dee22493424692d00f14848ab57c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:57:57.943107 19507 tablet_service.cc:1505] Processing CreateTablet for tablet e026dee22493424692d00f14848ab57c (DEFAULT_TABLE table=default.Table_85611716686c4097bacb64df677d5793 [id=122fe719b1d64ec8b531783539f85df1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:57:57.944085 19507 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e026dee22493424692d00f14848ab57c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:57:57.947877 19625 raft_consensus.cc:385] T 073b5941850c4140b1751956dd7cf5ef P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:57:57.948194 19625 raft_consensus.cc:740] T 073b5941850c4140b1751956dd7cf5ef P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d870164263bb44b6b780a557daea3656, State: Initialized, Role: FOLLOWER
I20260130 02:57:57.951566 19357 tablet_service.cc:1505] Processing CreateTablet for tablet d701152960b14396a6ac1bd10480d2b6 (DEFAULT_TABLE table=default.Table_e3b63d7d5d874900aac850654d98d62e [id=bc8cc55916fa4a41b78019f59e6fbe39]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:57:57.952723 19358 tablet_service.cc:1552] Processing DeleteTablet for tablet 7a902d0d571a42bb82ebffd48f7eca3f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:57:57 UTC) from {username='slave'} at 127.0.0.1:60110
I20260130 02:57:57.953215 19357 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d701152960b14396a6ac1bd10480d2b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:57:57.955551 19625 consensus_queue.cc:260] T 073b5941850c4140b1751956dd7cf5ef P d870164263bb44b6b780a557daea3656 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:57:57.958120 19807 consensus_queue.cc:260] T d4f2b54930cc4f5c825562e3e0a79f4d P 371a36d219164974abdf848841528152 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:57:57.971577 19710 ts_tablet_manager.cc:1929] T 3b8cdac4bf4b46efa26236fc930eb9a6 P d870164263bb44b6b780a557daea3656: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260130 02:57:57.973248 19710 log.cc:1199] T 3b8cdac4bf4b46efa26236fc930eb9a6 P d870164263bb44b6b780a557daea3656: Deleting WAL directory at /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-2/wal/wals/3b8cdac4bf4b46efa26236fc930eb9a6
I20260130 02:57:57.972891 19625 ts_tablet_manager.cc:1434] T 073b5941850c4140b1751956dd7cf5ef P d870164263bb44b6b780a557daea3656: Time spent starting tablet: real 0.113s user 0.005s sys 0.000s
I20260130 02:57:57.977617 19625 tablet_bootstrap.cc:492] T c3fce8cc99fb451ebd532fa37390e862 P d870164263bb44b6b780a557daea3656: Bootstrap starting.
I20260130 02:57:57.978197 19710 ts_tablet_manager.cc:1950] T 3b8cdac4bf4b46efa26236fc930eb9a6 P d870164263bb44b6b780a557daea3656: Deleting consensus metadata
I20260130 02:57:57.981797 19710 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59068 (request call id 39) took 1495 ms. Trace:
I20260130 02:57:57.983794 19360 tablet_service.cc:1505] Processing CreateTablet for tablet 15918c90ffdc43a5a38b2950e9834188 (DEFAULT_TABLE table=default.Table_b87dd1ee48f143538e487968bbea94bb [id=8b7db40afb894ae99625a756c75ef1de]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:57:57.984799 19360 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 15918c90ffdc43a5a38b2950e9834188. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:57:57.984544 19710 rpcz_store.cc:276] 0130 02:57:56.486321 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:57:56.493164 (+ 6843us) service_pool.cc:225] Handling call
0130 02:57:57.266014 (+772850us) ts_tablet_manager.cc:1151] Deleting tablet 3b8cdac4bf4b46efa26236fc930eb9a6
0130 02:57:57.266055 (+ 41us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0130 02:57:57.494409 (+228354us) tablet_metadata.cc:641] Metadata flushed
0130 02:57:57.951075 (+456666us) tablet_metadata.cc:641] Metadata flushed
0130 02:57:57.981742 (+ 30667us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":772316}
I20260130 02:57:57.991948 18860 catalog_manager.cc:4985] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): tablet 3b8cdac4bf4b46efa26236fc930eb9a6 (table default.Table_9428a7751159470793188f91cdcba4f3 [id=eaed0f6841e94faa858449fbb74bd5c4]) successfully deleted
W20260130 02:57:57.994678 18860 catalog_manager.cc:4952] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): delete failed for tablet 2806ae64d7324b918397c3c6b72ebcfe because the tablet was not found. No further retry: Not found: Tablet not found: 2806ae64d7324b918397c3c6b72ebcfe
I20260130 02:57:57.994987 19807 ts_tablet_manager.cc:1434] T d4f2b54930cc4f5c825562e3e0a79f4d P 371a36d219164974abdf848841528152: Time spent starting tablet: real 0.098s user 0.009s sys 0.000s
I20260130 02:57:57.999393 19710 tablet_replica.cc:333] T 30d8136b5ea742bca20ee46b5634d04e P d870164263bb44b6b780a557daea3656: stopping tablet replica
I20260130 02:57:58.000655 19807 tablet_bootstrap.cc:492] T ceda64471a92476f9ba183d2beab30d1 P 371a36d219164974abdf848841528152: Bootstrap starting.
I20260130 02:57:58.001550 19710 raft_consensus.cc:2243] T 30d8136b5ea742bca20ee46b5634d04e P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260130 02:57:58.002425 19710 raft_consensus.cc:2272] T 30d8136b5ea742bca20ee46b5634d04e P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260130 02:57:58.004537 19710 ts_tablet_manager.cc:1916] T 30d8136b5ea742bca20ee46b5634d04e P d870164263bb44b6b780a557daea3656: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260130 02:57:58.005861 19625 tablet_bootstrap.cc:654] T c3fce8cc99fb451ebd532fa37390e862 P d870164263bb44b6b780a557daea3656: Neither blocks nor log segments found. Creating new log.
I20260130 02:57:58.019613 19635 consensus_queue.cc:1048] T bac739bf4cac4272975f5587ad9c71ff P 371a36d219164974abdf848841528152 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260130 02:57:58.022080 19625 tablet_bootstrap.cc:492] T c3fce8cc99fb451ebd532fa37390e862 P d870164263bb44b6b780a557daea3656: No bootstrap required, opened a new log
I20260130 02:57:58.023209 19625 ts_tablet_manager.cc:1403] T c3fce8cc99fb451ebd532fa37390e862 P d870164263bb44b6b780a557daea3656: Time spent bootstrapping tablet: real 0.046s user 0.008s sys 0.002s
I20260130 02:57:58.030146 19754 ts_tablet_manager.cc:1929] T 30d8136b5ea742bca20ee46b5634d04e P 371a36d219164974abdf848841528152: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260130 02:57:58.034171 19742 raft_consensus.cc:493] T 2112b0402de5464cb1caf2c2abb67d78 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:57:58.034651 19742 raft_consensus.cc:515] T 2112b0402de5464cb1caf2c2abb67d78 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:57:58.036461 19742 leader_election.cc:290] T 2112b0402de5464cb1caf2c2abb67d78 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767)
I20260130 02:57:58.037041 18870 catalog_manager.cc:5746] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571) does not have the latest schema for tablet 38a7a10e36874183bf9fe55a38d829f6 (table default.Table_4d43592dc5b24ff28710e62ad4d87ce0 [id=27adf356fd004597a54c7a9887dd3ca8]). Expected version 3 got 2
I20260130 02:57:58.037806 18870 catalog_manager.cc:5746] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571) does not have the latest schema for tablet b078934be4bb4fd1915c691eeea9d00f (table default.Table_604228ee4a914be8a2f2f49f2f51e3f3 [id=720decfb55c24f34983ef636f06e3169]). Expected version 2 got 1
I20260130 02:57:58.039670 18870 catalog_manager.cc:5746] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571) does not have the latest schema for tablet ceda64471a92476f9ba183d2beab30d1 (table default.Table_90ad2e09fa9b4952a1564cee70ab8c1c [id=4f59d133b0cd4773bb012145927a92a4]). Expected version 1 got 0
I20260130 02:57:58.039565 19625 raft_consensus.cc:359] T c3fce8cc99fb451ebd532fa37390e862 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:57:58.040159 19625 raft_consensus.cc:385] T c3fce8cc99fb451ebd532fa37390e862 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:57:58.040408 19625 raft_consensus.cc:740] T c3fce8cc99fb451ebd532fa37390e862 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d870164263bb44b6b780a557daea3656, State: Initialized, Role: FOLLOWER
I20260130 02:57:58.042639 19625 consensus_queue.cc:260] T c3fce8cc99fb451ebd532fa37390e862 P d870164263bb44b6b780a557daea3656 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:57:58.048274 18868 catalog_manager.cc:5654] T bac739bf4cac4272975f5587ad9c71ff P 371a36d219164974abdf848841528152 reported cstate change: term changed from 0 to 1, leader changed from <none> to 371a36d219164974abdf848841528152 (127.18.100.65). New cstate: current_term: 1 leader_uuid: "371a36d219164974abdf848841528152" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } health_report { overall_health: UNKNOWN } } }
I20260130 02:57:58.050112 19625 ts_tablet_manager.cc:1434] T c3fce8cc99fb451ebd532fa37390e862 P d870164263bb44b6b780a557daea3656: Time spent starting tablet: real 0.026s user 0.005s sys 0.000s
I20260130 02:57:58.051002 19625 tablet_bootstrap.cc:492] T ffa1142fc7274aecbdbee1a0802b95da P d870164263bb44b6b780a557daea3656: Bootstrap starting.
I20260130 02:57:58.050650 18868 catalog_manager.cc:5654] T 37470ce4356f4542b621e21698551209 P 371a36d219164974abdf848841528152 reported cstate change: term changed from 0 to 1, leader changed from <none> to 371a36d219164974abdf848841528152 (127.18.100.65). New cstate: current_term: 1 leader_uuid: "371a36d219164974abdf848841528152" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } health_report { overall_health: UNKNOWN } } }
I20260130 02:57:58.052090 18868 catalog_manager.cc:5746] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021) does not have the latest schema for tablet 37470ce4356f4542b621e21698551209 (table default.Table_007e969f70d345a7bcb3dfbaabbc025d [id=77d4fa8705a3419cb0a84d3ca1bb4ce3]). Expected version 1 got 0
I20260130 02:57:58.052965 18868 catalog_manager.cc:5746] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021) does not have the latest schema for tablet 38a7a10e36874183bf9fe55a38d829f6 (table default.Table_4d43592dc5b24ff28710e62ad4d87ce0 [id=27adf356fd004597a54c7a9887dd3ca8]). Expected version 3 got 2
I20260130 02:57:58.056412 18868 catalog_manager.cc:5654] T 878f9346d3044d8d8b15c61f86869212 P 371a36d219164974abdf848841528152 reported cstate change: term changed from 0 to 1, leader changed from <none> to 371a36d219164974abdf848841528152 (127.18.100.65). New cstate: current_term: 1 leader_uuid: "371a36d219164974abdf848841528152" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } health_report { overall_health: UNKNOWN } } }
I20260130 02:57:58.058470 18875 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50284:
name: "default.Table_9d0cd9756483406081c8bdb18565a42f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260130 02:57:58.076247 18871 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50284:
table { table_name: "default.Table_2c0d85ead4ad406880e5796612629f18" } new_table_name: "default.Table_a9476edf8a5445c883f8d6ce786554a2" modify_external_catalogs: true
W20260130 02:57:58.077072 18875 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9d0cd9756483406081c8bdb18565a42f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260130 02:57:58.068259 18873 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50284:
table { table_name: "default.Table_e7b6f8ca4ed143888d55e87c1b157efd" } modify_external_catalogs: true
I20260130 02:57:58.079016 18873 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50284:
table { table_name: "default.Table_e7b6f8ca4ed143888d55e87c1b157efd" } modify_external_catalogs: true
I20260130 02:57:58.082376 18868 catalog_manager.cc:5746] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021) does not have the latest schema for tablet 878f9346d3044d8d8b15c61f86869212 (table default.Table_a31136243e6241a19ae437271d1cf283 [id=2009496525c94d7f8ee6ce59694b652d]). Expected version 1 got 0
I20260130 02:57:58.089063 18869 catalog_manager.cc:5941] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4: Sending DeleteTablet for 3 replicas of tablet 93809949599a449ca09b13e9b8bbdef6
I20260130 02:57:58.040118 19221 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2112b0402de5464cb1caf2c2abb67d78" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152" is_pre_election: true
W20260130 02:57:58.095788 19463 leader_election.cc:343] T 2112b0402de5464cb1caf2c2abb67d78 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:57:58.091985 19202 tablet_service.cc:1552] Processing DeleteTablet for tablet 93809949599a449ca09b13e9b8bbdef6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:57:57 UTC) from {username='slave'} at 127.0.0.1:42870
I20260130 02:57:58.085906 19807 tablet_bootstrap.cc:654] T ceda64471a92476f9ba183d2beab30d1 P 371a36d219164974abdf848841528152: Neither blocks nor log segments found. Creating new log.
I20260130 02:57:58.030556 19754 log.cc:1199] T 30d8136b5ea742bca20ee46b5634d04e P 371a36d219164974abdf848841528152: Deleting WAL directory at /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-0/wal/wals/30d8136b5ea742bca20ee46b5634d04e
I20260130 02:57:58.101990 19754 ts_tablet_manager.cc:1950] T 30d8136b5ea742bca20ee46b5634d04e P 371a36d219164974abdf848841528152: Deleting consensus metadata
I20260130 02:57:58.105471 19504 tablet_service.cc:1552] Processing DeleteTablet for tablet 93809949599a449ca09b13e9b8bbdef6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:57:57 UTC) from {username='slave'} at 127.0.0.1:59068
W20260130 02:57:58.106401 18860 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.18.100.124:34629: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20260130 02:57:58.109413 18861 catalog_manager.cc:4985] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021): tablet 30d8136b5ea742bca20ee46b5634d04e (table default.Table_4a74769a488b4760b12165bbffa7cac8 [id=d4c5313ba6e443e0af44b805d21617a4]) successfully deleted
I20260130 02:57:58.121510 19811 raft_consensus.cc:493] T c3fce8cc99fb451ebd532fa37390e862 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:57:58.121981 19811 raft_consensus.cc:515] T c3fce8cc99fb451ebd532fa37390e862 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:57:58.124657 19221 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c3fce8cc99fb451ebd532fa37390e862" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152" is_pre_election: true
W20260130 02:57:58.125833 19463 leader_election.cc:343] T c3fce8cc99fb451ebd532fa37390e862 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:57:58.127506 19754 tablet_replica.cc:333] T 01db06fcfffe4b259ac4240c0df38047 P 371a36d219164974abdf848841528152: stopping tablet replica
I20260130 02:57:58.128870 19754 raft_consensus.cc:2243] T 01db06fcfffe4b259ac4240c0df38047 P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260130 02:57:58.131747 19754 raft_consensus.cc:2272] T 01db06fcfffe4b259ac4240c0df38047 P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260130 02:57:58.134867 19462 consensus_peers.cc:597] T 01db06fcfffe4b259ac4240c0df38047 P d870164263bb44b6b780a557daea3656 -> Peer 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767): Couldn't send request to peer 78c2c46c7520492db35d51c244fb4f74. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 01db06fcfffe4b259ac4240c0df38047. This is attempt 1: this message will repeat every 5th retry.
I20260130 02:57:58.124902 19807 tablet_bootstrap.cc:492] T ceda64471a92476f9ba183d2beab30d1 P 371a36d219164974abdf848841528152: No bootstrap required, opened a new log
I20260130 02:57:58.135142 19811 leader_election.cc:290] T c3fce8cc99fb451ebd532fa37390e862 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767)
I20260130 02:57:58.135584 19807 ts_tablet_manager.cc:1403] T ceda64471a92476f9ba183d2beab30d1 P 371a36d219164974abdf848841528152: Time spent bootstrapping tablet: real 0.135s user 0.011s sys 0.000s
I20260130 02:57:58.137267 19358 tablet_service.cc:1552] Processing DeleteTablet for tablet 93809949599a449ca09b13e9b8bbdef6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:57:57 UTC) from {username='slave'} at 127.0.0.1:60110
I20260130 02:57:58.137792 19850 raft_consensus.cc:2804] T ceda64471a92476f9ba183d2beab30d1 P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Leader election won for term 1
I20260130 02:57:58.138224 19850 raft_consensus.cc:697] T ceda64471a92476f9ba183d2beab30d1 P 78c2c46c7520492db35d51c244fb4f74 [term 1 LEADER]: Becoming Leader. State: Replica: 78c2c46c7520492db35d51c244fb4f74, State: Running, Role: LEADER
I20260130 02:57:58.138875 19625 tablet_bootstrap.cc:654] T ffa1142fc7274aecbdbee1a0802b95da P d870164263bb44b6b780a557daea3656: Neither blocks nor log segments found. Creating new log.
I20260130 02:57:58.138976 19850 consensus_queue.cc:237] T ceda64471a92476f9ba183d2beab30d1 P 78c2c46c7520492db35d51c244fb4f74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:57:58.140388 19354 tablet_service.cc:1505] Processing CreateTablet for tablet e026dee22493424692d00f14848ab57c (DEFAULT_TABLE table=default.Table_85611716686c4097bacb64df677d5793 [id=122fe719b1d64ec8b531783539f85df1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:57:58.141467 19354 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e026dee22493424692d00f14848ab57c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:57:58.141956 19710 ts_tablet_manager.cc:1929] T 30d8136b5ea742bca20ee46b5634d04e P d870164263bb44b6b780a557daea3656: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260130 02:57:58.142364 19710 log.cc:1199] T 30d8136b5ea742bca20ee46b5634d04e P d870164263bb44b6b780a557daea3656: Deleting WAL directory at /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-2/wal/wals/30d8136b5ea742bca20ee46b5634d04e
I20260130 02:57:58.143007 19710 ts_tablet_manager.cc:1950] T 30d8136b5ea742bca20ee46b5634d04e P d870164263bb44b6b780a557daea3656: Deleting consensus metadata
W20260130 02:57:58.134407 19840 thread.cc:641] log-alloc [worker] (thread pool) Time spent creating pthread: real 0.733s user 0.032s sys 0.032s
I20260130 02:57:58.138017 19807 raft_consensus.cc:359] T ceda64471a92476f9ba183d2beab30d1 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:57:58.159634 19710 tablet_replica.cc:333] T 01db06fcfffe4b259ac4240c0df38047 P d870164263bb44b6b780a557daea3656: stopping tablet replica
I20260130 02:57:58.160387 19710 raft_consensus.cc:2243] T 01db06fcfffe4b259ac4240c0df38047 P d870164263bb44b6b780a557daea3656 [term 1 LEADER]: Raft consensus shutting down.
W20260130 02:57:58.161619 19840 thread.cc:608] log-alloc [worker] (thread pool) Time spent starting thread: real 0.760s user 0.032s sys 0.032s
I20260130 02:57:58.161760 19710 raft_consensus.cc:2272] T 01db06fcfffe4b259ac4240c0df38047 P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260130 02:57:58.162714 19807 raft_consensus.cc:385] T ceda64471a92476f9ba183d2beab30d1 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:57:58.163015 19807 raft_consensus.cc:740] T ceda64471a92476f9ba183d2beab30d1 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 371a36d219164974abdf848841528152, State: Initialized, Role: FOLLOWER
I20260130 02:57:58.166320 19807 consensus_queue.cc:260] T ceda64471a92476f9ba183d2beab30d1 P 371a36d219164974abdf848841528152 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:57:58.170111 19377 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2112b0402de5464cb1caf2c2abb67d78" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74" is_pre_election: true
I20260130 02:57:58.173396 19377 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c3fce8cc99fb451ebd532fa37390e862" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74" is_pre_election: true
W20260130 02:57:58.176075 19462 leader_election.cc:343] T 2112b0402de5464cb1caf2c2abb67d78 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:57:58.176695 19807 ts_tablet_manager.cc:1434] T ceda64471a92476f9ba183d2beab30d1 P 371a36d219164974abdf848841528152: Time spent starting tablet: real 0.041s user 0.005s sys 0.000s
I20260130 02:57:58.176602 19462 leader_election.cc:304] T 2112b0402de5464cb1caf2c2abb67d78 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d870164263bb44b6b780a557daea3656; no voters: 371a36d219164974abdf848841528152, 78c2c46c7520492db35d51c244fb4f74
I20260130 02:57:58.178184 19811 raft_consensus.cc:2749] T 2112b0402de5464cb1caf2c2abb67d78 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260130 02:57:58.185376 19462 leader_election.cc:343] T c3fce8cc99fb451ebd532fa37390e862 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:57:58.186479 19462 leader_election.cc:304] T c3fce8cc99fb451ebd532fa37390e862 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d870164263bb44b6b780a557daea3656; no voters: 371a36d219164974abdf848841528152, 78c2c46c7520492db35d51c244fb4f74
I20260130 02:57:58.187835 19811 raft_consensus.cc:2749] T c3fce8cc99fb451ebd532fa37390e862 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260130 02:57:58.190292 19807 tablet_bootstrap.cc:492] T 073b5941850c4140b1751956dd7cf5ef P 371a36d219164974abdf848841528152: Bootstrap starting.
I20260130 02:57:58.202957 19840 tablet_bootstrap.cc:492] T 2112b0402de5464cb1caf2c2abb67d78 P 78c2c46c7520492db35d51c244fb4f74: No bootstrap required, opened a new log
I20260130 02:57:58.203462 19840 ts_tablet_manager.cc:1403] T 2112b0402de5464cb1caf2c2abb67d78 P 78c2c46c7520492db35d51c244fb4f74: Time spent bootstrapping tablet: real 0.828s user 0.037s sys 0.037s
I20260130 02:57:58.209117 19840 raft_consensus.cc:359] T 2112b0402de5464cb1caf2c2abb67d78 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:57:58.212273 19635 consensus_queue.cc:1048] T 878f9346d3044d8d8b15c61f86869212 P 371a36d219164974abdf848841528152 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260130 02:57:58.216180 19635 consensus_queue.cc:1048] T bac739bf4cac4272975f5587ad9c71ff P 371a36d219164974abdf848841528152 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260130 02:57:58.219159 19710 ts_tablet_manager.cc:1916] T 01db06fcfffe4b259ac4240c0df38047 P d870164263bb44b6b780a557daea3656: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260130 02:57:58.228374 19807 tablet_bootstrap.cc:654] T 073b5941850c4140b1751956dd7cf5ef P 371a36d219164974abdf848841528152: Neither blocks nor log segments found. Creating new log.
W20260130 02:57:58.110900 19314 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.100.65:42021, user_credentials={real_user=slave}} blocked reactor thread for 372786us
I20260130 02:57:58.235808 19635 raft_consensus.cc:493] T 4035b8a330844dd28af9f3244bf8eb9a P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:57:58.247970 19635 raft_consensus.cc:515] T 4035b8a330844dd28af9f3244bf8eb9a P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:57:58.249814 19635 leader_election.cc:290] T 4035b8a330844dd28af9f3244bf8eb9a P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767), d870164263bb44b6b780a557daea3656 (127.18.100.67:39571)
I20260130 02:57:58.250761 19527 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4035b8a330844dd28af9f3244bf8eb9a" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
I20260130 02:57:58.251335 19527 raft_consensus.cc:2468] T 4035b8a330844dd28af9f3244bf8eb9a P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 371a36d219164974abdf848841528152 in term 1.
I20260130 02:57:58.253010 19159 leader_election.cc:304] T 4035b8a330844dd28af9f3244bf8eb9a P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 371a36d219164974abdf848841528152, d870164263bb44b6b780a557daea3656; no voters:
I20260130 02:57:58.264966 19754 ts_tablet_manager.cc:1916] T 01db06fcfffe4b259ac4240c0df38047 P 371a36d219164974abdf848841528152: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260130 02:57:58.268823 19635 raft_consensus.cc:2804] T 4035b8a330844dd28af9f3244bf8eb9a P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260130 02:57:58.231936 19840 raft_consensus.cc:385] T 2112b0402de5464cb1caf2c2abb67d78 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:57:58.270066 19635 raft_consensus.cc:493] T 4035b8a330844dd28af9f3244bf8eb9a P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 78c2c46c7520492db35d51c244fb4f74)
I20260130 02:57:58.270520 19635 raft_consensus.cc:3060] T 4035b8a330844dd28af9f3244bf8eb9a P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Advancing to term 2
I20260130 02:57:58.272707 19710 ts_tablet_manager.cc:1929] T 01db06fcfffe4b259ac4240c0df38047 P d870164263bb44b6b780a557daea3656: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260130 02:57:58.273089 19710 log.cc:1199] T 01db06fcfffe4b259ac4240c0df38047 P d870164263bb44b6b780a557daea3656: Deleting WAL directory at /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-2/wal/wals/01db06fcfffe4b259ac4240c0df38047
I20260130 02:57:58.273752 19710 ts_tablet_manager.cc:1950] T 01db06fcfffe4b259ac4240c0df38047 P d870164263bb44b6b780a557daea3656: Deleting consensus metadata
I20260130 02:57:58.277944 19377 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4035b8a330844dd28af9f3244bf8eb9a" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74" is_pre_election: true
I20260130 02:57:58.274983 19840 raft_consensus.cc:740] T 2112b0402de5464cb1caf2c2abb67d78 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78c2c46c7520492db35d51c244fb4f74, State: Initialized, Role: FOLLOWER
W20260130 02:57:58.282891 18860 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.100.124:34629, user_credentials={real_user=slave}} blocked reactor thread for 176488us
I20260130 02:57:58.284319 18869 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50284:
name: "default.Table_e37f8d618ff244a6af90cc83a9035328"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260130 02:57:58.286155 18869 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e37f8d618ff244a6af90cc83a9035328 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260130 02:57:58.292089 18872 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50284:
name: "default.Table_0480357296ac4a27812d802ee4566351"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260130 02:57:58.293298 19710 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59068 (request call id 45) took 1045 ms. Trace:
W20260130 02:57:58.293721 18872 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0480357296ac4a27812d802ee4566351 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260130 02:57:58.293597 19710 rpcz_store.cc:276] 0130 02:57:57.248381 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:57:57.284755 (+ 36374us) service_pool.cc:225] Handling call
0130 02:57:58.159480 (+874725us) ts_tablet_manager.cc:1151] Deleting tablet 01db06fcfffe4b259ac4240c0df38047
0130 02:57:58.159517 (+ 37us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0130 02:57:58.225741 (+ 66224us) tablet_metadata.cc:641] Metadata flushed
0130 02:57:58.272493 (+ 46752us) tablet_metadata.cc:641] Metadata flushed
0130 02:57:58.293236 (+ 20743us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":689361}
I20260130 02:57:58.295189 19710 tablet_replica.cc:333] T 93809949599a449ca09b13e9b8bbdef6 P d870164263bb44b6b780a557daea3656: stopping tablet replica
I20260130 02:57:58.295019 18867 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50284:
name: "default.Table_7d1c27ab3515406ba5b5333c90df093e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260130 02:57:58.295850 19710 raft_consensus.cc:2243] T 93809949599a449ca09b13e9b8bbdef6 P d870164263bb44b6b780a557daea3656 [term 1 LEADER]: Raft consensus shutting down.
I20260130 02:57:58.296630 19710 raft_consensus.cc:2272] T 93809949599a449ca09b13e9b8bbdef6 P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260130 02:57:58.296635 18867 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7d1c27ab3515406ba5b5333c90df093e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260130 02:57:58.297151 18833 external_mini_cluster.cc:1366] Running /tmp/dist-test-tasktGe2wI/build/tsan/bin/kudu
/tmp/dist-test-tasktGe2wI/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.100.124:34629
--webserver_interface=127.18.100.124
--webserver_port=37741
--builtin_ntp_servers=127.18.100.84:46033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.100.126:38191,127.18.100.125:39711,127.18.100.124:34629
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20260130 02:57:58.300745 18871 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet bd43c5d148514836869a8ac648c8ad11 has no consensus state
I20260130 02:57:58.308957 19710 ts_tablet_manager.cc:1916] T 93809949599a449ca09b13e9b8bbdef6 P d870164263bb44b6b780a557daea3656: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260130 02:57:58.311669 18860 catalog_manager.cc:4985] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): tablet 30d8136b5ea742bca20ee46b5634d04e (table default.Table_4a74769a488b4760b12165bbffa7cac8 [id=d4c5313ba6e443e0af44b805d21617a4]) successfully deleted
I20260130 02:57:58.312225 18860 catalog_manager.cc:4985] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): tablet 01db06fcfffe4b259ac4240c0df38047 (table default.Table_e67bd1859b5846f386097cf33c5f1bcb [id=952ccb90477f47e4a352623247a8a6f2]) successfully deleted
W20260130 02:57:58.312783 18860 catalog_manager.cc:4965] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): delete failed for tablet 7a902d0d571a42bb82ebffd48f7eca3f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7a902d0d571a42bb82ebffd48f7eca3f already in progress: creating tablet
I20260130 02:57:58.277094 19635 raft_consensus.cc:515] T 4035b8a330844dd28af9f3244bf8eb9a P 371a36d219164974abdf848841528152 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:57:58.323280 19625 tablet_bootstrap.cc:492] T ffa1142fc7274aecbdbee1a0802b95da P d870164263bb44b6b780a557daea3656: No bootstrap required, opened a new log
I20260130 02:57:58.324419 19625 ts_tablet_manager.cc:1403] T ffa1142fc7274aecbdbee1a0802b95da P d870164263bb44b6b780a557daea3656: Time spent bootstrapping tablet: real 0.274s user 0.041s sys 0.024s
I20260130 02:57:58.335770 19625 raft_consensus.cc:359] T ffa1142fc7274aecbdbee1a0802b95da P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:57:58.337131 19625 raft_consensus.cc:385] T ffa1142fc7274aecbdbee1a0802b95da P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:57:58.337750 19625 raft_consensus.cc:740] T ffa1142fc7274aecbdbee1a0802b95da P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d870164263bb44b6b780a557daea3656, State: Initialized, Role: FOLLOWER
I20260130 02:57:58.338912 19625 consensus_queue.cc:260] T ffa1142fc7274aecbdbee1a0802b95da P d870164263bb44b6b780a557daea3656 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:57:58.282255 19840 consensus_queue.cc:260] T 2112b0402de5464cb1caf2c2abb67d78 P 78c2c46c7520492db35d51c244fb4f74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:57:58.341652 19840 ts_tablet_manager.cc:1434] T 2112b0402de5464cb1caf2c2abb67d78 P 78c2c46c7520492db35d51c244fb4f74: Time spent starting tablet: real 0.138s user 0.005s sys 0.001s
I20260130 02:57:58.341821 19625 ts_tablet_manager.cc:1434] T ffa1142fc7274aecbdbee1a0802b95da P d870164263bb44b6b780a557daea3656: Time spent starting tablet: real 0.016s user 0.004s sys 0.000s
I20260130 02:57:58.342543 19840 tablet_bootstrap.cc:492] T 073b5941850c4140b1751956dd7cf5ef P 78c2c46c7520492db35d51c244fb4f74: Bootstrap starting.
I20260130 02:57:58.345817 19625 tablet_bootstrap.cc:492] T 7a902d0d571a42bb82ebffd48f7eca3f P d870164263bb44b6b780a557daea3656: Bootstrap starting.
I20260130 02:57:58.342779 19758 ts_tablet_manager.cc:1929] T 3b8cdac4bf4b46efa26236fc930eb9a6 P 78c2c46c7520492db35d51c244fb4f74: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260130 02:57:58.347940 19635 leader_election.cc:290] T 4035b8a330844dd28af9f3244bf8eb9a P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 2 election: Requested vote from peers 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767), d870164263bb44b6b780a557daea3656 (127.18.100.67:39571)
I20260130 02:57:58.348160 19525 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4035b8a330844dd28af9f3244bf8eb9a" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656"
I20260130 02:57:58.348673 19525 raft_consensus.cc:3060] T 4035b8a330844dd28af9f3244bf8eb9a P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Advancing to term 2
I20260130 02:57:58.347225 19758 log.cc:1199] T 3b8cdac4bf4b46efa26236fc930eb9a6 P 78c2c46c7520492db35d51c244fb4f74: Deleting WAL directory at /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-1/wal/wals/3b8cdac4bf4b46efa26236fc930eb9a6
I20260130 02:57:58.354593 19758 ts_tablet_manager.cc:1950] T 3b8cdac4bf4b46efa26236fc930eb9a6 P 78c2c46c7520492db35d51c244fb4f74: Deleting consensus metadata
I20260130 02:57:58.355432 19758 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60110 (request call id 41) took 1106 ms. Trace:
W20260130 02:57:58.356551 19314 consensus_peers.cc:597] T 30d8136b5ea742bca20ee46b5634d04e P 78c2c46c7520492db35d51c244fb4f74 -> Peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Couldn't send request to peer 371a36d219164974abdf848841528152. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 30d8136b5ea742bca20ee46b5634d04e. This is attempt 1: this message will repeat every 5th retry.
I20260130 02:57:58.357865 19625 tablet_bootstrap.cc:654] T 7a902d0d571a42bb82ebffd48f7eca3f P d870164263bb44b6b780a557daea3656: Neither blocks nor log segments found. Creating new log.
I20260130 02:57:58.358609 19224 raft_consensus.cc:1240] T 4035b8a330844dd28af9f3244bf8eb9a P 371a36d219164974abdf848841528152 [term 2 FOLLOWER]: Rejecting Update request from peer 78c2c46c7520492db35d51c244fb4f74 for earlier term 1. Current term is 2. Ops: []
I20260130 02:57:58.360431 19787 consensus_queue.cc:1059] T 4035b8a330844dd28af9f3244bf8eb9a P 78c2c46c7520492db35d51c244fb4f74 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20260130 02:57:58.361729 19377 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4035b8a330844dd28af9f3244bf8eb9a" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74"
I20260130 02:57:58.361645 19813 tablet.cc:1722] T b078934be4bb4fd1915c691eeea9d00f P 371a36d219164974abdf848841528152: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20260130 02:57:58.355798 19758 rpcz_store.cc:276] 0130 02:57:57.248557 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:57:57.250191 (+ 1634us) service_pool.cc:225] Handling call
0130 02:57:57.645234 (+395043us) ts_tablet_manager.cc:1151] Deleting tablet 3b8cdac4bf4b46efa26236fc930eb9a6
0130 02:57:57.645277 (+ 43us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0130 02:57:57.818929 (+173652us) tablet_metadata.cc:641] Metadata flushed
0130 02:57:58.284244 (+465315us) tablet_metadata.cc:641] Metadata flushed
0130 02:57:58.355365 (+ 71121us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":394296}
I20260130 02:57:58.364741 19758 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60110 (request call id 43) took 1115 ms. Trace:
I20260130 02:57:58.365063 19758 rpcz_store.cc:276] 0130 02:57:57.249145 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:57:57.256291 (+ 7146us) service_pool.cc:225] Handling call
0130 02:57:58.364499 (+1108208us) ts_tablet_manager.cc:1151] Deleting tablet 2806ae64d7324b918397c3c6b72ebcfe
0130 02:57:58.364543 (+ 44us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0130 02:57:58.364689 (+ 146us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1107374}
I20260130 02:57:58.367412 19758 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60110 (request call id 42) took 1118 ms. Trace:
I20260130 02:57:58.367736 19758 rpcz_store.cc:276] 0130 02:57:57.248846 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:57:57.282250 (+ 33404us) service_pool.cc:225] Handling call
0130 02:57:58.367154 (+1084904us) ts_tablet_manager.cc:1151] Deleting tablet 06b31d020b864c418476e6dcc1e4eafb
0130 02:57:58.367203 (+ 49us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0130 02:57:58.367358 (+ 155us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1084238}
I20260130 02:57:58.371838 19840 tablet_bootstrap.cc:654] T 073b5941850c4140b1751956dd7cf5ef P 78c2c46c7520492db35d51c244fb4f74: Neither blocks nor log segments found. Creating new log.
I20260130 02:57:58.377656 19874 tablet.cc:1722] T b078934be4bb4fd1915c691eeea9d00f P 78c2c46c7520492db35d51c244fb4f74: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20260130 02:57:58.380440 19807 tablet_bootstrap.cc:492] T 073b5941850c4140b1751956dd7cf5ef P 371a36d219164974abdf848841528152: No bootstrap required, opened a new log
I20260130 02:57:58.381090 19625 tablet_bootstrap.cc:492] T 7a902d0d571a42bb82ebffd48f7eca3f P d870164263bb44b6b780a557daea3656: No bootstrap required, opened a new log
I20260130 02:57:58.381525 19807 ts_tablet_manager.cc:1403] T 073b5941850c4140b1751956dd7cf5ef P 371a36d219164974abdf848841528152: Time spent bootstrapping tablet: real 0.191s user 0.001s sys 0.010s
I20260130 02:57:58.382006 19625 ts_tablet_manager.cc:1403] T 7a902d0d571a42bb82ebffd48f7eca3f P d870164263bb44b6b780a557daea3656: Time spent bootstrapping tablet: real 0.036s user 0.007s sys 0.004s
I20260130 02:57:58.382977 19758 tablet_replica.cc:333] T 30d8136b5ea742bca20ee46b5634d04e P 78c2c46c7520492db35d51c244fb4f74: stopping tablet replica
I20260130 02:57:58.383827 19758 raft_consensus.cc:2243] T 30d8136b5ea742bca20ee46b5634d04e P 78c2c46c7520492db35d51c244fb4f74 [term 1 LEADER]: Raft consensus shutting down.
I20260130 02:57:58.384546 19758 pending_rounds.cc:70] T 30d8136b5ea742bca20ee46b5634d04e P 78c2c46c7520492db35d51c244fb4f74: Trying to abort 1 pending ops.
I20260130 02:57:58.384708 19525 raft_consensus.cc:2468] T 4035b8a330844dd28af9f3244bf8eb9a P d870164263bb44b6b780a557daea3656 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 371a36d219164974abdf848841528152 in term 2.
I20260130 02:57:58.384789 19758 pending_rounds.cc:77] T 30d8136b5ea742bca20ee46b5634d04e P 78c2c46c7520492db35d51c244fb4f74: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7248862726881599488 op_type: NO_OP noop_request { }
I20260130 02:57:58.385111 19758 raft_consensus.cc:2889] T 30d8136b5ea742bca20ee46b5634d04e P 78c2c46c7520492db35d51c244fb4f74 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20260130 02:57:58.385464 19758 raft_consensus.cc:2272] T 30d8136b5ea742bca20ee46b5634d04e P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260130 02:57:58.385962 19159 leader_election.cc:304] T 4035b8a330844dd28af9f3244bf8eb9a P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 371a36d219164974abdf848841528152, d870164263bb44b6b780a557daea3656; no voters: 78c2c46c7520492db35d51c244fb4f74
I20260130 02:57:58.389829 19635 raft_consensus.cc:2804] T 4035b8a330844dd28af9f3244bf8eb9a P 371a36d219164974abdf848841528152 [term 2 FOLLOWER]: Leader election won for term 2
I20260130 02:57:58.392146 19663 raft_consensus.cc:3055] T 4035b8a330844dd28af9f3244bf8eb9a P 78c2c46c7520492db35d51c244fb4f74 [term 1 LEADER]: Stepping down as leader of term 1
I20260130 02:57:58.392601 19663 raft_consensus.cc:740] T 4035b8a330844dd28af9f3244bf8eb9a P 78c2c46c7520492db35d51c244fb4f74 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 78c2c46c7520492db35d51c244fb4f74, State: Running, Role: LEADER
I20260130 02:57:58.392886 19710 ts_tablet_manager.cc:1929] T 93809949599a449ca09b13e9b8bbdef6 P d870164263bb44b6b780a557daea3656: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260130 02:57:58.393265 19710 log.cc:1199] T 93809949599a449ca09b13e9b8bbdef6 P d870164263bb44b6b780a557daea3656: Deleting WAL directory at /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-2/wal/wals/93809949599a449ca09b13e9b8bbdef6
I20260130 02:57:58.393970 19710 ts_tablet_manager.cc:1950] T 93809949599a449ca09b13e9b8bbdef6 P d870164263bb44b6b780a557daea3656: Deleting consensus metadata
I20260130 02:57:58.393639 19663 consensus_queue.cc:260] T 4035b8a330844dd28af9f3244bf8eb9a P 78c2c46c7520492db35d51c244fb4f74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:57:58.395316 19663 raft_consensus.cc:3060] T 4035b8a330844dd28af9f3244bf8eb9a P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Advancing to term 2
I20260130 02:57:58.396020 19635 raft_consensus.cc:697] T 4035b8a330844dd28af9f3244bf8eb9a P 371a36d219164974abdf848841528152 [term 2 LEADER]: Becoming Leader. State: Replica: 371a36d219164974abdf848841528152, State: Running, Role: LEADER
I20260130 02:57:58.395807 19625 raft_consensus.cc:359] T 7a902d0d571a42bb82ebffd48f7eca3f P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:57:58.396757 19625 raft_consensus.cc:385] T 7a902d0d571a42bb82ebffd48f7eca3f P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:57:58.396991 19625 raft_consensus.cc:740] T 7a902d0d571a42bb82ebffd48f7eca3f P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d870164263bb44b6b780a557daea3656, State: Initialized, Role: FOLLOWER
I20260130 02:57:58.396821 19635 consensus_queue.cc:237] T 4035b8a330844dd28af9f3244bf8eb9a P 371a36d219164974abdf848841528152 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:57:58.401398 19625 consensus_queue.cc:260] T 7a902d0d571a42bb82ebffd48f7eca3f P d870164263bb44b6b780a557daea3656 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:57:58.401664 19807 raft_consensus.cc:359] T 073b5941850c4140b1751956dd7cf5ef P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:57:58.402303 19807 raft_consensus.cc:385] T 073b5941850c4140b1751956dd7cf5ef P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:57:58.402591 19807 raft_consensus.cc:740] T 073b5941850c4140b1751956dd7cf5ef P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 371a36d219164974abdf848841528152, State: Initialized, Role: FOLLOWER
W20260130 02:57:58.411038 18860 consensus_peers.cc:597] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4 -> Peer 3749baa287ed4f249329a74071781edd (127.18.100.124:34629): Couldn't send request to peer 3749baa287ed4f249329a74071781edd. Status: Network error: Client connection negotiation failed: client connection to 127.18.100.124:34629: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20260130 02:57:58.413034 19758 ts_tablet_manager.cc:1916] T 30d8136b5ea742bca20ee46b5634d04e P 78c2c46c7520492db35d51c244fb4f74: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260130 02:57:58.416114 19807 consensus_queue.cc:260] T 073b5941850c4140b1751956dd7cf5ef P 371a36d219164974abdf848841528152 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:57:58.418506 19807 ts_tablet_manager.cc:1434] T 073b5941850c4140b1751956dd7cf5ef P 371a36d219164974abdf848841528152: Time spent starting tablet: real 0.037s user 0.007s sys 0.000s
I20260130 02:57:58.422048 19807 tablet_bootstrap.cc:492] T 2112b0402de5464cb1caf2c2abb67d78 P 371a36d219164974abdf848841528152: Bootstrap starting.
I20260130 02:57:58.422104 18860 catalog_manager.cc:4985] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): tablet 93809949599a449ca09b13e9b8bbdef6 (table default.Table_1c77605822574b10a05074a093ad30a3 [id=aeba5f5a21564264a8cd975244774ab8]) successfully deleted
I20260130 02:57:58.429415 19840 tablet_bootstrap.cc:492] T 073b5941850c4140b1751956dd7cf5ef P 78c2c46c7520492db35d51c244fb4f74: No bootstrap required, opened a new log
I20260130 02:57:58.429903 19840 ts_tablet_manager.cc:1403] T 073b5941850c4140b1751956dd7cf5ef P 78c2c46c7520492db35d51c244fb4f74: Time spent bootstrapping tablet: real 0.088s user 0.007s sys 0.003s
I20260130 02:57:58.432451 19840 raft_consensus.cc:359] T 073b5941850c4140b1751956dd7cf5ef P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:57:58.433130 19840 raft_consensus.cc:385] T 073b5941850c4140b1751956dd7cf5ef P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:57:58.433411 19840 raft_consensus.cc:740] T 073b5941850c4140b1751956dd7cf5ef P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78c2c46c7520492db35d51c244fb4f74, State: Initialized, Role: FOLLOWER
I20260130 02:57:58.439335 18874 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50284:
table { table_name: "default.Table_a78ce7ab85924506b184fb74db3ed0c3" } modify_external_catalogs: true
I20260130 02:57:58.439898 18874 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50284:
table { table_name: "default.Table_a78ce7ab85924506b184fb74db3ed0c3" } modify_external_catalogs: true
I20260130 02:57:58.440045 18869 master_service.cc:954] ReplaceTablet: received request to replace tablet d701152960b14396a6ac1bd10480d2b6 from {username='slave'} at 127.0.0.1:50284
I20260130 02:57:58.438367 18875 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50284:
table { table_name: "default.Table_90ad2e09fa9b4952a1564cee70ab8c1c" } new_table_name: "default.Table_be9aa687d9744ac6a2aa335772c6cc9a" modify_external_catalogs: true
I20260130 02:57:58.448247 19807 tablet_bootstrap.cc:654] T 2112b0402de5464cb1caf2c2abb67d78 P 371a36d219164974abdf848841528152: Neither blocks nor log segments found. Creating new log.
I20260130 02:57:58.450250 19635 raft_consensus.cc:493] T 073b5941850c4140b1751956dd7cf5ef P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:57:58.450712 19635 raft_consensus.cc:515] T 073b5941850c4140b1751956dd7cf5ef P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:57:58.452555 19635 leader_election.cc:290] T 073b5941850c4140b1751956dd7cf5ef P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d870164263bb44b6b780a557daea3656 (127.18.100.67:39571), 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767)
I20260130 02:57:58.457625 19380 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "073b5941850c4140b1751956dd7cf5ef" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74" is_pre_election: true
I20260130 02:57:58.460028 18860 catalog_manager.cc:4985] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767): tablet 3b8cdac4bf4b46efa26236fc930eb9a6 (table default.Table_9428a7751159470793188f91cdcba4f3 [id=eaed0f6841e94faa858449fbb74bd5c4]) successfully deleted
W20260130 02:57:58.460628 18860 catalog_manager.cc:4952] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767): delete failed for tablet 2806ae64d7324b918397c3c6b72ebcfe because the tablet was not found. No further retry: Not found: Tablet not found: 2806ae64d7324b918397c3c6b72ebcfe
I20260130 02:57:58.419102 19625 ts_tablet_manager.cc:1434] T 7a902d0d571a42bb82ebffd48f7eca3f P d870164263bb44b6b780a557daea3656: Time spent starting tablet: real 0.036s user 0.005s sys 0.000s
I20260130 02:57:58.454869 19742 raft_consensus.cc:493] T 7a902d0d571a42bb82ebffd48f7eca3f P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260130 02:57:58.461426 18860 catalog_manager.cc:4952] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767): delete failed for tablet 06b31d020b864c418476e6dcc1e4eafb because the tablet was not found. No further retry: Not found: Tablet not found: 06b31d020b864c418476e6dcc1e4eafb
I20260130 02:57:58.461372 19742 raft_consensus.cc:515] T 7a902d0d571a42bb82ebffd48f7eca3f P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:57:58.458964 19525 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "073b5941850c4140b1751956dd7cf5ef" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
I20260130 02:57:58.436909 19880 tablet.cc:1722] T b078934be4bb4fd1915c691eeea9d00f P d870164263bb44b6b780a557daea3656: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20260130 02:57:58.462929 19525 raft_consensus.cc:2468] T 073b5941850c4140b1751956dd7cf5ef P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 371a36d219164974abdf848841528152 in term 0.
I20260130 02:57:58.464208 19159 leader_election.cc:304] T 073b5941850c4140b1751956dd7cf5ef P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 371a36d219164974abdf848841528152, d870164263bb44b6b780a557daea3656; no voters:
I20260130 02:57:58.464939 19635 raft_consensus.cc:2804] T 073b5941850c4140b1751956dd7cf5ef P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260130 02:57:58.465267 19635 raft_consensus.cc:493] T 073b5941850c4140b1751956dd7cf5ef P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260130 02:57:58.465583 19635 raft_consensus.cc:3060] T 073b5941850c4140b1751956dd7cf5ef P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:57:58.465734 19742 leader_election.cc:290] T 7a902d0d571a42bb82ebffd48f7eca3f P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767)
I20260130 02:57:58.472530 19224 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7a902d0d571a42bb82ebffd48f7eca3f" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152" is_pre_election: true
I20260130 02:57:58.473485 19625 tablet_bootstrap.cc:492] T e364f4e70e5749e083e14e7fa7416ff0 P d870164263bb44b6b780a557daea3656: Bootstrap starting.
I20260130 02:57:58.473773 19377 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7a902d0d571a42bb82ebffd48f7eca3f" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74" is_pre_election: true
W20260130 02:57:58.475387 19463 leader_election.cc:343] T 7a902d0d571a42bb82ebffd48f7eca3f P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Illegal state: must be running to vote when last-logged opid is not known
W20260130 02:57:58.477990 19462 leader_election.cc:343] T 7a902d0d571a42bb82ebffd48f7eca3f P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:57:58.479735 19635 raft_consensus.cc:515] T 073b5941850c4140b1751956dd7cf5ef P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:57:58.481647 19635 leader_election.cc:290] T 073b5941850c4140b1751956dd7cf5ef P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 election: Requested vote from peers d870164263bb44b6b780a557daea3656 (127.18.100.67:39571), 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767)
I20260130 02:57:58.483502 19525 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "073b5941850c4140b1751956dd7cf5ef" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656"
I20260130 02:57:58.484036 19525 raft_consensus.cc:3060] T 073b5941850c4140b1751956dd7cf5ef P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:57:58.484020 19377 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "073b5941850c4140b1751956dd7cf5ef" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74"
I20260130 02:57:58.479751 19462 leader_election.cc:304] T 7a902d0d571a42bb82ebffd48f7eca3f P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d870164263bb44b6b780a557daea3656; no voters: 371a36d219164974abdf848841528152, 78c2c46c7520492db35d51c244fb4f74
I20260130 02:57:58.486829 19742 raft_consensus.cc:2749] T 7a902d0d571a42bb82ebffd48f7eca3f P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260130 02:57:58.434046 19840 consensus_queue.cc:260] T 073b5941850c4140b1751956dd7cf5ef P 78c2c46c7520492db35d51c244fb4f74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:57:58.490375 19663 raft_consensus.cc:493] T 073b5941850c4140b1751956dd7cf5ef P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:57:58.490538 19525 raft_consensus.cc:2468] T 073b5941850c4140b1751956dd7cf5ef P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 371a36d219164974abdf848841528152 in term 1.
I20260130 02:57:58.490849 19663 raft_consensus.cc:515] T 073b5941850c4140b1751956dd7cf5ef P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:57:58.491837 19159 leader_election.cc:304] T 073b5941850c4140b1751956dd7cf5ef P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 371a36d219164974abdf848841528152, d870164263bb44b6b780a557daea3656; no voters: 78c2c46c7520492db35d51c244fb4f74
I20260130 02:57:58.493024 19807 tablet_bootstrap.cc:492] T 2112b0402de5464cb1caf2c2abb67d78 P 371a36d219164974abdf848841528152: No bootstrap required, opened a new log
I20260130 02:57:58.493170 19635 raft_consensus.cc:2804] T 073b5941850c4140b1751956dd7cf5ef P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Leader election won for term 1
I20260130 02:57:58.493429 19807 ts_tablet_manager.cc:1403] T 2112b0402de5464cb1caf2c2abb67d78 P 371a36d219164974abdf848841528152: Time spent bootstrapping tablet: real 0.072s user 0.006s sys 0.009s
I20260130 02:57:58.493835 19635 raft_consensus.cc:697] T 073b5941850c4140b1751956dd7cf5ef P 371a36d219164974abdf848841528152 [term 1 LEADER]: Becoming Leader. State: Replica: 371a36d219164974abdf848841528152, State: Running, Role: LEADER
I20260130 02:57:58.494199 19380 raft_consensus.cc:2468] T 073b5941850c4140b1751956dd7cf5ef P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 371a36d219164974abdf848841528152 in term 0.
I20260130 02:57:58.494798 19527 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "073b5941850c4140b1751956dd7cf5ef" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
I20260130 02:57:58.494614 19635 consensus_queue.cc:237] T 073b5941850c4140b1751956dd7cf5ef P 371a36d219164974abdf848841528152 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:57:58.495494 19527 raft_consensus.cc:2393] T 073b5941850c4140b1751956dd7cf5ef P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 78c2c46c7520492db35d51c244fb4f74 in current term 1: Already voted for candidate 371a36d219164974abdf848841528152 in this term.
I20260130 02:57:58.493773 19663 leader_election.cc:290] T 073b5941850c4140b1751956dd7cf5ef P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d870164263bb44b6b780a557daea3656 (127.18.100.67:39571), 371a36d219164974abdf848841528152 (127.18.100.65:42021)
I20260130 02:57:58.496433 19807 raft_consensus.cc:359] T 2112b0402de5464cb1caf2c2abb67d78 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:57:58.496984 19224 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "073b5941850c4140b1751956dd7cf5ef" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152" is_pre_election: true
I20260130 02:57:58.497212 19807 raft_consensus.cc:385] T 2112b0402de5464cb1caf2c2abb67d78 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:57:58.497481 19807 raft_consensus.cc:740] T 2112b0402de5464cb1caf2c2abb67d78 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 371a36d219164974abdf848841528152, State: Initialized, Role: FOLLOWER
I20260130 02:57:58.498327 19840 ts_tablet_manager.cc:1434] T 073b5941850c4140b1751956dd7cf5ef P 78c2c46c7520492db35d51c244fb4f74: Time spent starting tablet: real 0.068s user 0.006s sys 0.000s
I20260130 02:57:58.498040 19807 consensus_queue.cc:260] T 2112b0402de5464cb1caf2c2abb67d78 P 371a36d219164974abdf848841528152 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:57:58.498992 19314 leader_election.cc:304] T 073b5941850c4140b1751956dd7cf5ef P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78c2c46c7520492db35d51c244fb4f74; no voters: 371a36d219164974abdf848841528152, d870164263bb44b6b780a557daea3656
I20260130 02:57:58.499990 19850 raft_consensus.cc:3060] T 073b5941850c4140b1751956dd7cf5ef P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:57:58.501113 19807 ts_tablet_manager.cc:1434] T 2112b0402de5464cb1caf2c2abb67d78 P 371a36d219164974abdf848841528152: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260130 02:57:58.507558 19807 tablet_bootstrap.cc:492] T 0b1877e5f9244913bcd639f1336df7ee P 371a36d219164974abdf848841528152: Bootstrap starting.
I20260130 02:57:58.513587 19807 tablet_bootstrap.cc:654] T 0b1877e5f9244913bcd639f1336df7ee P 371a36d219164974abdf848841528152: Neither blocks nor log segments found. Creating new log.
I20260130 02:57:58.514824 19754 ts_tablet_manager.cc:1929] T 01db06fcfffe4b259ac4240c0df38047 P 371a36d219164974abdf848841528152: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260130 02:57:58.515179 19754 log.cc:1199] T 01db06fcfffe4b259ac4240c0df38047 P 371a36d219164974abdf848841528152: Deleting WAL directory at /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-0/wal/wals/01db06fcfffe4b259ac4240c0df38047
I20260130 02:57:58.515589 18872 master_service.cc:954] ReplaceTablet: received request to replace tablet 4035b8a330844dd28af9f3244bf8eb9a from {username='slave'} at 127.0.0.1:50284
I20260130 02:57:58.515877 19754 ts_tablet_manager.cc:1950] T 01db06fcfffe4b259ac4240c0df38047 P 371a36d219164974abdf848841528152: Deleting consensus metadata
I20260130 02:57:58.506500 19850 raft_consensus.cc:2749] T 073b5941850c4140b1751956dd7cf5ef P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260130 02:57:58.517820 19754 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42870 (request call id 47) took 1128 ms. Trace:
I20260130 02:57:58.535615 19758 ts_tablet_manager.cc:1929] T 30d8136b5ea742bca20ee46b5634d04e P 78c2c46c7520492db35d51c244fb4f74: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260130 02:57:58.536020 19758 log.cc:1199] T 30d8136b5ea742bca20ee46b5634d04e P 78c2c46c7520492db35d51c244fb4f74: Deleting WAL directory at /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-1/wal/wals/30d8136b5ea742bca20ee46b5634d04e
I20260130 02:57:58.536701 19758 ts_tablet_manager.cc:1950] T 30d8136b5ea742bca20ee46b5634d04e P 78c2c46c7520492db35d51c244fb4f74: Deleting consensus metadata
I20260130 02:57:58.518118 19754 rpcz_store.cc:276] 0130 02:57:57.389734 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:57:57.422245 (+ 32511us) service_pool.cc:225] Handling call
0130 02:57:58.127333 (+705088us) ts_tablet_manager.cc:1151] Deleting tablet 01db06fcfffe4b259ac4240c0df38047
0130 02:57:58.127376 (+ 43us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0130 02:57:58.439325 (+311949us) tablet_metadata.cc:641] Metadata flushed
0130 02:57:58.478530 (+ 39205us) tablet_metadata.cc:641] Metadata flushed
0130 02:57:58.517764 (+ 39234us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":698200}
I20260130 02:57:58.540191 18861 catalog_manager.cc:4985] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021): tablet 01db06fcfffe4b259ac4240c0df38047 (table default.Table_e67bd1859b5846f386097cf33c5f1bcb [id=952ccb90477f47e4a352623247a8a6f2]) successfully deleted
I20260130 02:57:58.543186 19754 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42870 (request call id 51) took 1091 ms. Trace:
I20260130 02:57:58.543557 19754 rpcz_store.cc:276] 0130 02:57:57.451690 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:57:57.456001 (+ 4311us) service_pool.cc:225] Handling call
0130 02:57:58.542884 (+1086883us) ts_tablet_manager.cc:1151] Deleting tablet 2806ae64d7324b918397c3c6b72ebcfe
0130 02:57:58.542936 (+ 52us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0130 02:57:58.543123 (+ 187us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1086327}
W20260130 02:57:58.544656 18861 catalog_manager.cc:4952] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021): delete failed for tablet 2806ae64d7324b918397c3c6b72ebcfe because the tablet was not found. No further retry: Not found: Tablet not found: 2806ae64d7324b918397c3c6b72ebcfe
I20260130 02:57:58.548604 19754 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42870 (request call id 50) took 1097 ms. Trace:
I20260130 02:57:58.549364 18869 catalog_manager.cc:5941] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4: Sending DeleteTablet for 3 replicas of tablet d701152960b14396a6ac1bd10480d2b6
I20260130 02:57:58.549023 19754 rpcz_store.cc:276] 0130 02:57:57.451399 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:57:57.452185 (+ 786us) service_pool.cc:225] Handling call
0130 02:57:58.548311 (+1096126us) ts_tablet_manager.cc:1151] Deleting tablet c2a9675427754d0db1578637a37440c6
0130 02:57:58.548359 (+ 48us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0130 02:57:58.548538 (+ 179us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1091165}
W20260130 02:57:58.550159 18861 catalog_manager.cc:4952] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021): delete failed for tablet c2a9675427754d0db1578637a37440c6 because the tablet was not found. No further retry: Not found: Tablet not found: c2a9675427754d0db1578637a37440c6
I20260130 02:57:58.555400 19754 tablet_replica.cc:333] T 3b8cdac4bf4b46efa26236fc930eb9a6 P 371a36d219164974abdf848841528152: stopping tablet replica
I20260130 02:57:58.559384 19204 tablet_service.cc:1552] Processing DeleteTablet for tablet d701152960b14396a6ac1bd10480d2b6 with delete_type TABLET_DATA_DELETED (replaced by tablet 481bd4cca6ee4c178873bf12392e3441) from {username='slave'} at 127.0.0.1:42870
I20260130 02:57:58.562599 19754 raft_consensus.cc:2243] T 3b8cdac4bf4b46efa26236fc930eb9a6 P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260130 02:57:58.558082 19758 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60110 (request call id 44) took 1236 ms. Trace:
I20260130 02:57:58.563585 19758 rpcz_store.cc:276] 0130 02:57:57.321061 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:57:57.321230 (+ 169us) service_pool.cc:225] Handling call
0130 02:57:58.382825 (+1061595us) ts_tablet_manager.cc:1151] Deleting tablet 30d8136b5ea742bca20ee46b5634d04e
0130 02:57:58.382864 (+ 39us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0130 02:57:58.454084 (+ 71220us) tablet_metadata.cc:641] Metadata flushed
0130 02:57:58.535368 (+ 81284us) tablet_metadata.cc:641] Metadata flushed
0130 02:57:58.557104 (+ 21736us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1060997}
I20260130 02:57:58.564467 18869 catalog_manager.cc:6724] ReplaceTablet: tablet d701152960b14396a6ac1bd10480d2b6 deleted and replaced by tablet 481bd4cca6ee4c178873bf12392e3441
I20260130 02:57:58.565407 18869 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50284:
name: "default.Table_d764cbaecfee476ba5e734dd1e2f50fd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260130 02:57:58.567355 18869 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d764cbaecfee476ba5e734dd1e2f50fd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260130 02:57:58.573403 19679 raft_consensus.cc:493] T d4f2b54930cc4f5c825562e3e0a79f4d P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:57:58.573899 19679 raft_consensus.cc:515] T d4f2b54930cc4f5c825562e3e0a79f4d P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:57:58.575899 19679 leader_election.cc:290] T d4f2b54930cc4f5c825562e3e0a79f4d P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767), d870164263bb44b6b780a557daea3656 (127.18.100.67:39571)
I20260130 02:57:58.577600 19525 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d4f2b54930cc4f5c825562e3e0a79f4d" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
I20260130 02:57:58.578379 19525 raft_consensus.cc:2393] T d4f2b54930cc4f5c825562e3e0a79f4d P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 371a36d219164974abdf848841528152 in current term 1: Already voted for candidate 78c2c46c7520492db35d51c244fb4f74 in this term.
W20260130 02:57:58.587733 18860 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.100.125:39711, user_credentials={real_user=slave}} blocked reactor thread for 43835.5us
I20260130 02:57:58.589389 19754 raft_consensus.cc:2272] T 3b8cdac4bf4b46efa26236fc930eb9a6 P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260130 02:57:58.591564 19754 ts_tablet_manager.cc:1916] T 3b8cdac4bf4b46efa26236fc930eb9a6 P 371a36d219164974abdf848841528152: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260130 02:57:58.596632 19758 tablet_replica.cc:333] T 93809949599a449ca09b13e9b8bbdef6 P 78c2c46c7520492db35d51c244fb4f74: stopping tablet replica
I20260130 02:57:58.596921 19360 tablet_service.cc:1552] Processing DeleteTablet for tablet d701152960b14396a6ac1bd10480d2b6 with delete_type TABLET_DATA_DELETED (replaced by tablet 481bd4cca6ee4c178873bf12392e3441) from {username='slave'} at 127.0.0.1:60110
I20260130 02:57:58.603040 19504 tablet_service.cc:1552] Processing DeleteTablet for tablet d701152960b14396a6ac1bd10480d2b6 with delete_type TABLET_DATA_DELETED (replaced by tablet 481bd4cca6ee4c178873bf12392e3441) from {username='slave'} at 127.0.0.1:59068
I20260130 02:57:58.603466 19224 raft_consensus.cc:3060] T d4f2b54930cc4f5c825562e3e0a79f4d P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:57:58.611923 19787 consensus_queue.cc:1048] T d4f2b54930cc4f5c825562e3e0a79f4d P 78c2c46c7520492db35d51c244fb4f74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260130 02:57:58.612445 19380 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d4f2b54930cc4f5c825562e3e0a79f4d" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74" is_pre_election: true
I20260130 02:57:58.613807 19159 leader_election.cc:304] T d4f2b54930cc4f5c825562e3e0a79f4d P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 371a36d219164974abdf848841528152; no voters: 78c2c46c7520492db35d51c244fb4f74, d870164263bb44b6b780a557daea3656
I20260130 02:57:58.614190 18860 catalog_manager.cc:4985] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767): tablet 30d8136b5ea742bca20ee46b5634d04e (table default.Table_4a74769a488b4760b12165bbffa7cac8 [id=d4c5313ba6e443e0af44b805d21617a4]) successfully deleted
I20260130 02:57:58.614773 19635 raft_consensus.cc:2749] T d4f2b54930cc4f5c825562e3e0a79f4d P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260130 02:57:58.614874 18860 catalog_manager.cc:4965] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767): delete failed for tablet 7a902d0d571a42bb82ebffd48f7eca3f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7a902d0d571a42bb82ebffd48f7eca3f already in progress: creating tablet
W20260130 02:57:58.616819 18860 catalog_manager.cc:4965] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): delete failed for tablet d701152960b14396a6ac1bd10480d2b6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d701152960b14396a6ac1bd10480d2b6 already in progress: creating tablet
I20260130 02:57:58.623229 19758 raft_consensus.cc:2243] T 93809949599a449ca09b13e9b8bbdef6 P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260130 02:57:58.623786 19758 raft_consensus.cc:2272] T 93809949599a449ca09b13e9b8bbdef6 P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260130 02:57:58.625510 19758 ts_tablet_manager.cc:1916] T 93809949599a449ca09b13e9b8bbdef6 P 78c2c46c7520492db35d51c244fb4f74: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260130 02:57:58.631778 19625 tablet_bootstrap.cc:654] T e364f4e70e5749e083e14e7fa7416ff0 P d870164263bb44b6b780a557daea3656: Neither blocks nor log segments found. Creating new log.
I20260130 02:57:58.639389 19840 tablet_bootstrap.cc:492] T 0b1877e5f9244913bcd639f1336df7ee P 78c2c46c7520492db35d51c244fb4f74: Bootstrap starting.
W20260130 02:57:58.661412 18860 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet bd43c5d148514836869a8ac648c8ad11 (table default.Table_a9476edf8a5445c883f8d6ce786554a2 [id=7e29f588b41b491e8a9d1ad61e50b033], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet bd43c5d148514836869a8ac648c8ad11 has no consensus state
I20260130 02:57:58.662092 19840 tablet_bootstrap.cc:654] T 0b1877e5f9244913bcd639f1336df7ee P 78c2c46c7520492db35d51c244fb4f74: Neither blocks nor log segments found. Creating new log.
I20260130 02:57:58.668148 18869 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50284:
table { table_name: "default.Table_a31136243e6241a19ae437271d1cf283" } new_table_name: "default.Table_8edea1fae34d4d18b54b0f801a145225" modify_external_catalogs: true
I20260130 02:57:58.668313 19813 tablet.cc:1722] T 878f9346d3044d8d8b15c61f86869212 P 371a36d219164974abdf848841528152: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260130 02:57:58.678364 19880 tablet.cc:1722] T 878f9346d3044d8d8b15c61f86869212 P d870164263bb44b6b780a557daea3656: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260130 02:57:58.685067 19840 tablet_bootstrap.cc:492] T 0b1877e5f9244913bcd639f1336df7ee P 78c2c46c7520492db35d51c244fb4f74: No bootstrap required, opened a new log
I20260130 02:57:58.685523 19840 ts_tablet_manager.cc:1403] T 0b1877e5f9244913bcd639f1336df7ee P 78c2c46c7520492db35d51c244fb4f74: Time spent bootstrapping tablet: real 0.046s user 0.007s sys 0.004s
I20260130 02:57:58.687964 19840 raft_consensus.cc:359] T 0b1877e5f9244913bcd639f1336df7ee P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:57:58.688624 19840 raft_consensus.cc:385] T 0b1877e5f9244913bcd639f1336df7ee P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:57:58.688907 19840 raft_consensus.cc:740] T 0b1877e5f9244913bcd639f1336df7ee P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78c2c46c7520492db35d51c244fb4f74, State: Initialized, Role: FOLLOWER
I20260130 02:57:58.691860 19874 tablet.cc:1722] T 878f9346d3044d8d8b15c61f86869212 P 78c2c46c7520492db35d51c244fb4f74: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260130 02:57:58.694507 19840 consensus_queue.cc:260] T 0b1877e5f9244913bcd639f1336df7ee P 78c2c46c7520492db35d51c244fb4f74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:57:58.695988 19625 tablet_bootstrap.cc:492] T e364f4e70e5749e083e14e7fa7416ff0 P d870164263bb44b6b780a557daea3656: No bootstrap required, opened a new log
I20260130 02:57:58.702900 19840 ts_tablet_manager.cc:1434] T 0b1877e5f9244913bcd639f1336df7ee P 78c2c46c7520492db35d51c244fb4f74: Time spent starting tablet: real 0.017s user 0.007s sys 0.001s
I20260130 02:57:58.717566 19840 tablet_bootstrap.cc:492] T 1bbc55bbf2254d8bb6b721d7e8f69d85 P 78c2c46c7520492db35d51c244fb4f74: Bootstrap starting.
I20260130 02:57:58.717661 19807 tablet_bootstrap.cc:492] T 0b1877e5f9244913bcd639f1336df7ee P 371a36d219164974abdf848841528152: No bootstrap required, opened a new log
I20260130 02:57:58.718106 19807 ts_tablet_manager.cc:1403] T 0b1877e5f9244913bcd639f1336df7ee P 371a36d219164974abdf848841528152: Time spent bootstrapping tablet: real 0.211s user 0.048s sys 0.095s
I20260130 02:57:58.720811 19807 raft_consensus.cc:359] T 0b1877e5f9244913bcd639f1336df7ee P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:57:58.721453 19807 raft_consensus.cc:385] T 0b1877e5f9244913bcd639f1336df7ee P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:57:58.721740 19807 raft_consensus.cc:740] T 0b1877e5f9244913bcd639f1336df7ee P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 371a36d219164974abdf848841528152, State: Initialized, Role: FOLLOWER
I20260130 02:57:58.725025 19807 consensus_queue.cc:260] T 0b1877e5f9244913bcd639f1336df7ee P 371a36d219164974abdf848841528152 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:57:58.728365 19840 tablet_bootstrap.cc:654] T 1bbc55bbf2254d8bb6b721d7e8f69d85 P 78c2c46c7520492db35d51c244fb4f74: Neither blocks nor log segments found. Creating new log.
I20260130 02:57:58.730755 19807 ts_tablet_manager.cc:1434] T 0b1877e5f9244913bcd639f1336df7ee P 371a36d219164974abdf848841528152: Time spent starting tablet: real 0.012s user 0.007s sys 0.002s
I20260130 02:57:58.736857 19807 tablet_bootstrap.cc:492] T 1bbc55bbf2254d8bb6b721d7e8f69d85 P 371a36d219164974abdf848841528152: Bootstrap starting.
I20260130 02:57:58.748307 19807 tablet_bootstrap.cc:654] T 1bbc55bbf2254d8bb6b721d7e8f69d85 P 371a36d219164974abdf848841528152: Neither blocks nor log segments found. Creating new log.
I20260130 02:57:58.757887 19204 tablet_service.cc:1505] Processing CreateTablet for tablet 26560f746d244e58a182eb6bb24c8720 (DEFAULT_TABLE table=default.Table_b2d2da7aae8b430aba32bac2944aa931 [id=8d76f4328edc491291491c78ba205d30]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:57:58.758843 19204 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 26560f746d244e58a182eb6bb24c8720. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:57:58.762952 19856 consensus_queue.cc:1048] T d4f2b54930cc4f5c825562e3e0a79f4d P 78c2c46c7520492db35d51c244fb4f74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260130 02:57:58.767608 19840 tablet_bootstrap.cc:492] T 1bbc55bbf2254d8bb6b721d7e8f69d85 P 78c2c46c7520492db35d51c244fb4f74: No bootstrap required, opened a new log
I20260130 02:57:58.768038 19840 ts_tablet_manager.cc:1403] T 1bbc55bbf2254d8bb6b721d7e8f69d85 P 78c2c46c7520492db35d51c244fb4f74: Time spent bootstrapping tablet: real 0.051s user 0.007s sys 0.004s
I20260130 02:57:58.770426 19840 raft_consensus.cc:359] T 1bbc55bbf2254d8bb6b721d7e8f69d85 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:57:58.771106 19840 raft_consensus.cc:385] T 1bbc55bbf2254d8bb6b721d7e8f69d85 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:57:58.771373 19840 raft_consensus.cc:740] T 1bbc55bbf2254d8bb6b721d7e8f69d85 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78c2c46c7520492db35d51c244fb4f74, State: Initialized, Role: FOLLOWER
I20260130 02:57:58.696398 19625 ts_tablet_manager.cc:1403] T e364f4e70e5749e083e14e7fa7416ff0 P d870164263bb44b6b780a557daea3656: Time spent bootstrapping tablet: real 0.223s user 0.007s sys 0.001s
I20260130 02:57:58.777290 19840 consensus_queue.cc:260] T 1bbc55bbf2254d8bb6b721d7e8f69d85 P 78c2c46c7520492db35d51c244fb4f74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:57:58.780601 19840 ts_tablet_manager.cc:1434] T 1bbc55bbf2254d8bb6b721d7e8f69d85 P 78c2c46c7520492db35d51c244fb4f74: Time spent starting tablet: real 0.012s user 0.006s sys 0.000s
I20260130 02:57:58.781443 19840 tablet_bootstrap.cc:492] T ffa1142fc7274aecbdbee1a0802b95da P 78c2c46c7520492db35d51c244fb4f74: Bootstrap starting.
I20260130 02:57:58.798817 18872 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50284:
name: "default.Table_d11b53ac041740d8818edf7d2861dc6d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260130 02:57:58.803800 18872 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d11b53ac041740d8818edf7d2861dc6d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260130 02:57:58.815161 18871 catalog_manager.cc:5746] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767) does not have the latest schema for tablet 37470ce4356f4542b621e21698551209 (table default.Table_007e969f70d345a7bcb3dfbaabbc025d [id=77d4fa8705a3419cb0a84d3ca1bb4ce3]). Expected version 1 got 0
I20260130 02:57:58.816892 18871 catalog_manager.cc:5654] T d4f2b54930cc4f5c825562e3e0a79f4d P 78c2c46c7520492db35d51c244fb4f74 reported cstate change: term changed from 0 to 1, leader changed from <none> to 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66). New cstate: current_term: 1 leader_uuid: "78c2c46c7520492db35d51c244fb4f74" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } health_report { overall_health: UNKNOWN } } }
I20260130 02:57:58.823891 19202 tablet_service.cc:1505] Processing CreateTablet for tablet 5b2976108584478a98fd90fe6594af66 (DEFAULT_TABLE table=default.Table_a78ce7ab85924506b184fb74db3ed0c3 [id=5f4c327b216449a1a8b146d0f652de52]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:57:58.824486 19840 tablet_bootstrap.cc:654] T ffa1142fc7274aecbdbee1a0802b95da P 78c2c46c7520492db35d51c244fb4f74: Neither blocks nor log segments found. Creating new log.
I20260130 02:57:58.824996 19202 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5b2976108584478a98fd90fe6594af66. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:57:58.826244 19200 tablet_service.cc:1505] Processing CreateTablet for tablet c7c621b99deb457998c44ab1644d3939 (DEFAULT_TABLE table=default.Table_f7aab7c003284493810119a94032eca0 [id=7a08fa3986d0410d81de0235d2fb0f92]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:57:58.827271 19200 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c7c621b99deb457998c44ab1644d3939. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:57:58.827474 18871 catalog_manager.cc:5746] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767) does not have the latest schema for tablet 878f9346d3044d8d8b15c61f86869212 (table default.Table_a31136243e6241a19ae437271d1cf283 [id=2009496525c94d7f8ee6ce59694b652d]). Expected version 1 got 0
I20260130 02:57:58.828246 18871 catalog_manager.cc:5654] T ceda64471a92476f9ba183d2beab30d1 P 78c2c46c7520492db35d51c244fb4f74 reported cstate change: term changed from 0 to 1, leader changed from <none> to 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66). New cstate: current_term: 1 leader_uuid: "78c2c46c7520492db35d51c244fb4f74" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } health_report { overall_health: UNKNOWN } } }
I20260130 02:57:58.829620 18871 catalog_manager.cc:5746] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767) does not have the latest schema for tablet ceda64471a92476f9ba183d2beab30d1 (table default.Table_90ad2e09fa9b4952a1564cee70ab8c1c [id=4f59d133b0cd4773bb012145927a92a4]). Expected version 1 got 0
I20260130 02:57:58.837222 19360 tablet_service.cc:1505] Processing CreateTablet for tablet 26560f746d244e58a182eb6bb24c8720 (DEFAULT_TABLE table=default.Table_b2d2da7aae8b430aba32bac2944aa931 [id=8d76f4328edc491291491c78ba205d30]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:57:58.837425 19504 tablet_service.cc:1505] Processing CreateTablet for tablet 26560f746d244e58a182eb6bb24c8720 (DEFAULT_TABLE table=default.Table_b2d2da7aae8b430aba32bac2944aa931 [id=8d76f4328edc491291491c78ba205d30]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:57:58.838191 19360 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 26560f746d244e58a182eb6bb24c8720. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:57:58.838459 19504 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 26560f746d244e58a182eb6bb24c8720. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:57:58.843995 19860 raft_consensus.cc:493] T 2112b0402de5464cb1caf2c2abb67d78 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:57:58.844398 19860 raft_consensus.cc:515] T 2112b0402de5464cb1caf2c2abb67d78 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:57:58.846266 19860 leader_election.cc:290] T 2112b0402de5464cb1caf2c2abb67d78 P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d870164263bb44b6b780a557daea3656 (127.18.100.67:39571), 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767)
I20260130 02:57:58.846906 19527 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2112b0402de5464cb1caf2c2abb67d78" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
I20260130 02:57:58.847523 19527 raft_consensus.cc:2468] T 2112b0402de5464cb1caf2c2abb67d78 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 371a36d219164974abdf848841528152 in term 0.
I20260130 02:57:58.848912 19159 leader_election.cc:304] T 2112b0402de5464cb1caf2c2abb67d78 P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 371a36d219164974abdf848841528152, d870164263bb44b6b780a557daea3656; no voters:
I20260130 02:57:58.849577 19860 raft_consensus.cc:2804] T 2112b0402de5464cb1caf2c2abb67d78 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260130 02:57:58.849886 19860 raft_consensus.cc:493] T 2112b0402de5464cb1caf2c2abb67d78 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260130 02:57:58.850175 19860 raft_consensus.cc:3060] T 2112b0402de5464cb1caf2c2abb67d78 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:57:58.850422 19380 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2112b0402de5464cb1caf2c2abb67d78" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74" is_pre_election: true
I20260130 02:57:58.850999 19380 raft_consensus.cc:2468] T 2112b0402de5464cb1caf2c2abb67d78 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 371a36d219164974abdf848841528152 in term 0.
I20260130 02:57:58.853204 19625 raft_consensus.cc:359] T e364f4e70e5749e083e14e7fa7416ff0 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:57:58.853873 19625 raft_consensus.cc:385] T e364f4e70e5749e083e14e7fa7416ff0 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:57:58.854101 19625 raft_consensus.cc:740] T e364f4e70e5749e083e14e7fa7416ff0 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d870164263bb44b6b780a557daea3656, State: Initialized, Role: FOLLOWER
I20260130 02:57:58.856475 19860 raft_consensus.cc:515] T 2112b0402de5464cb1caf2c2abb67d78 P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:57:58.858628 19860 leader_election.cc:290] T 2112b0402de5464cb1caf2c2abb67d78 P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 election: Requested vote from peers d870164263bb44b6b780a557daea3656 (127.18.100.67:39571), 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767)
I20260130 02:57:58.858826 19625 consensus_queue.cc:260] T e364f4e70e5749e083e14e7fa7416ff0 P d870164263bb44b6b780a557daea3656 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:57:58.860088 19527 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2112b0402de5464cb1caf2c2abb67d78" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656"
I20260130 02:57:58.860646 19527 raft_consensus.cc:3060] T 2112b0402de5464cb1caf2c2abb67d78 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:57:58.863088 19354 tablet_service.cc:1505] Processing CreateTablet for tablet 5b2976108584478a98fd90fe6594af66 (DEFAULT_TABLE table=default.Table_a78ce7ab85924506b184fb74db3ed0c3 [id=5f4c327b216449a1a8b146d0f652de52]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:57:58.863197 19507 tablet_service.cc:1505] Processing CreateTablet for tablet 5b2976108584478a98fd90fe6594af66 (DEFAULT_TABLE table=default.Table_a78ce7ab85924506b184fb74db3ed0c3 [id=5f4c327b216449a1a8b146d0f652de52]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:57:58.864105 19507 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5b2976108584478a98fd90fe6594af66. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:57:58.864120 19354 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5b2976108584478a98fd90fe6594af66. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:57:58.869428 19380 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2112b0402de5464cb1caf2c2abb67d78" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74"
I20260130 02:57:58.869998 19380 raft_consensus.cc:3060] T 2112b0402de5464cb1caf2c2abb67d78 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:57:58.870786 19840 tablet_bootstrap.cc:492] T ffa1142fc7274aecbdbee1a0802b95da P 78c2c46c7520492db35d51c244fb4f74: No bootstrap required, opened a new log
I20260130 02:57:58.871205 19840 ts_tablet_manager.cc:1403] T ffa1142fc7274aecbdbee1a0802b95da P 78c2c46c7520492db35d51c244fb4f74: Time spent bootstrapping tablet: real 0.090s user 0.005s sys 0.006s
I20260130 02:57:58.873854 19527 raft_consensus.cc:2468] T 2112b0402de5464cb1caf2c2abb67d78 P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 371a36d219164974abdf848841528152 in term 1.
I20260130 02:57:58.877887 18874 catalog_manager.cc:5941] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4: Sending DeleteTablet for 3 replicas of tablet 5b2976108584478a98fd90fe6594af66
I20260130 02:57:58.879036 19380 raft_consensus.cc:2468] T 2112b0402de5464cb1caf2c2abb67d78 P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 371a36d219164974abdf848841528152 in term 1.
I20260130 02:57:58.880043 18874 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50284:
name: "default.Table_87dbbd65d92043b0b91c801517e56dfe"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260130 02:57:58.882529 19840 raft_consensus.cc:359] T ffa1142fc7274aecbdbee1a0802b95da P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:57:58.883265 19840 raft_consensus.cc:385] T ffa1142fc7274aecbdbee1a0802b95da P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:57:58.883011 18872 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50284:
name: "default.Table_92836e7fcf3a48d8bac2160c2ff7c482"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260130 02:57:58.883383 18874 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_87dbbd65d92043b0b91c801517e56dfe in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260130 02:57:58.883536 19840 raft_consensus.cc:740] T ffa1142fc7274aecbdbee1a0802b95da P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78c2c46c7520492db35d51c244fb4f74, State: Initialized, Role: FOLLOWER
I20260130 02:57:58.883999 19159 leader_election.cc:304] T 2112b0402de5464cb1caf2c2abb67d78 P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 371a36d219164974abdf848841528152, 78c2c46c7520492db35d51c244fb4f74; no voters:
W20260130 02:57:58.884737 18872 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_92836e7fcf3a48d8bac2160c2ff7c482 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260130 02:57:58.885640 19861 raft_consensus.cc:2804] T 2112b0402de5464cb1caf2c2abb67d78 P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Leader election won for term 1
I20260130 02:57:58.886073 19861 raft_consensus.cc:697] T 2112b0402de5464cb1caf2c2abb67d78 P 371a36d219164974abdf848841528152 [term 1 LEADER]: Becoming Leader. State: Replica: 371a36d219164974abdf848841528152, State: Running, Role: LEADER
I20260130 02:57:58.886516 19840 consensus_queue.cc:260] T ffa1142fc7274aecbdbee1a0802b95da P 78c2c46c7520492db35d51c244fb4f74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:57:58.886811 19861 consensus_queue.cc:237] T 2112b0402de5464cb1caf2c2abb67d78 P 371a36d219164974abdf848841528152 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:57:58.887696 19202 tablet_service.cc:1552] Processing DeleteTablet for tablet 5b2976108584478a98fd90fe6594af66 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:57:58 UTC) from {username='slave'} at 127.0.0.1:42870
I20260130 02:57:58.892212 19360 tablet_service.cc:1552] Processing DeleteTablet for tablet 5b2976108584478a98fd90fe6594af66 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:57:58 UTC) from {username='slave'} at 127.0.0.1:60110
I20260130 02:57:58.898378 19840 ts_tablet_manager.cc:1434] T ffa1142fc7274aecbdbee1a0802b95da P 78c2c46c7520492db35d51c244fb4f74: Time spent starting tablet: real 0.023s user 0.000s sys 0.006s
I20260130 02:57:58.901588 19754 ts_tablet_manager.cc:1929] T 3b8cdac4bf4b46efa26236fc930eb9a6 P 371a36d219164974abdf848841528152: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260130 02:57:58.830320 18871 catalog_manager.cc:5746] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767) does not have the latest schema for tablet 2112b0402de5464cb1caf2c2abb67d78 (table default.Table_da3d581b15a94840898777f74611994e [id=b7e241ced53e4160ac6fa1f65ef3674e]). Expected version 1 got 0
I20260130 02:57:58.902799 19840 tablet_bootstrap.cc:492] T c3fce8cc99fb451ebd532fa37390e862 P 78c2c46c7520492db35d51c244fb4f74: Bootstrap starting.
I20260130 02:57:58.889829 19508 tablet_service.cc:1552] Processing DeleteTablet for tablet 5b2976108584478a98fd90fe6594af66 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:57:58 UTC) from {username='slave'} at 127.0.0.1:59068
I20260130 02:57:58.916124 19508 tablet_service.cc:1505] Processing CreateTablet for tablet c7c621b99deb457998c44ab1644d3939 (DEFAULT_TABLE table=default.Table_f7aab7c003284493810119a94032eca0 [id=7a08fa3986d0410d81de0235d2fb0f92]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:57:58.916034 19354 tablet_service.cc:1505] Processing CreateTablet for tablet c7c621b99deb457998c44ab1644d3939 (DEFAULT_TABLE table=default.Table_f7aab7c003284493810119a94032eca0 [id=7a08fa3986d0410d81de0235d2fb0f92]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:57:58.916309 19360 tablet_service.cc:1505] Processing CreateTablet for tablet 1ca34772ef484c1f809a99d126e28eff (DEFAULT_TABLE table=default.Table_9cb9a9f9e5d145a6a42d7ccbc42aced9 [id=5f1f179fadc247da9382dd7796dcf2c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:57:58.917069 19508 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c7c621b99deb457998c44ab1644d3939. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:57:58.917079 19354 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c7c621b99deb457998c44ab1644d3939. 1 dirs total, 0 dirs full, 0 dirs failed
W20260130 02:57:58.917304 18860 catalog_manager.cc:4965] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): delete failed for tablet 5b2976108584478a98fd90fe6594af66 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5b2976108584478a98fd90fe6594af66 already in progress: creating tablet
I20260130 02:57:58.917596 19360 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1ca34772ef484c1f809a99d126e28eff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:57:58.924988 19204 tablet_service.cc:1505] Processing CreateTablet for tablet 1ca34772ef484c1f809a99d126e28eff (DEFAULT_TABLE table=default.Table_9cb9a9f9e5d145a6a42d7ccbc42aced9 [id=5f1f179fadc247da9382dd7796dcf2c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:57:58.926119 19204 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1ca34772ef484c1f809a99d126e28eff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:57:58.861502 19625 ts_tablet_manager.cc:1434] T e364f4e70e5749e083e14e7fa7416ff0 P d870164263bb44b6b780a557daea3656: Time spent starting tablet: real 0.078s user 0.003s sys 0.001s
I20260130 02:57:58.934652 19840 tablet_bootstrap.cc:654] T c3fce8cc99fb451ebd532fa37390e862 P 78c2c46c7520492db35d51c244fb4f74: Neither blocks nor log segments found. Creating new log.
I20260130 02:57:58.935884 19625 tablet_bootstrap.cc:492] T 1bbc55bbf2254d8bb6b721d7e8f69d85 P d870164263bb44b6b780a557daea3656: Bootstrap starting.
I20260130 02:57:58.964478 18872 master_service.cc:954] ReplaceTablet: received request to replace tablet d4f2b54930cc4f5c825562e3e0a79f4d from {username='slave'} at 127.0.0.1:50284
I20260130 02:57:58.976805 19861 raft_consensus.cc:493] T ceda64471a92476f9ba183d2beab30d1 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:57:58.977317 19861 raft_consensus.cc:515] T ceda64471a92476f9ba183d2beab30d1 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:57:58.979298 19861 leader_election.cc:290] T ceda64471a92476f9ba183d2beab30d1 P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767), d870164263bb44b6b780a557daea3656 (127.18.100.67:39571)
I20260130 02:57:58.980948 19377 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ceda64471a92476f9ba183d2beab30d1" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74" is_pre_election: true
I20260130 02:57:58.987550 18867 catalog_manager.cc:5746] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571) does not have the latest schema for tablet 37470ce4356f4542b621e21698551209 (table default.Table_007e969f70d345a7bcb3dfbaabbc025d [id=77d4fa8705a3419cb0a84d3ca1bb4ce3]). Expected version 1 got 0
I20260130 02:57:58.989315 18867 catalog_manager.cc:5746] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571) does not have the latest schema for tablet ceda64471a92476f9ba183d2beab30d1 (table default.Table_90ad2e09fa9b4952a1564cee70ab8c1c [id=4f59d133b0cd4773bb012145927a92a4]). Expected version 1 got 0
I20260130 02:57:58.996568 19758 ts_tablet_manager.cc:1929] T 93809949599a449ca09b13e9b8bbdef6 P 78c2c46c7520492db35d51c244fb4f74: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260130 02:57:58.997047 19758 log.cc:1199] T 93809949599a449ca09b13e9b8bbdef6 P 78c2c46c7520492db35d51c244fb4f74: Deleting WAL directory at /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-1/wal/wals/93809949599a449ca09b13e9b8bbdef6
I20260130 02:57:58.998477 19758 ts_tablet_manager.cc:1950] T 93809949599a449ca09b13e9b8bbdef6 P 78c2c46c7520492db35d51c244fb4f74: Deleting consensus metadata
I20260130 02:57:58.902020 19754 log.cc:1199] T 3b8cdac4bf4b46efa26236fc930eb9a6 P 371a36d219164974abdf848841528152: Deleting WAL directory at /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-0/wal/wals/3b8cdac4bf4b46efa26236fc930eb9a6
I20260130 02:57:58.999460 19754 ts_tablet_manager.cc:1950] T 3b8cdac4bf4b46efa26236fc930eb9a6 P 371a36d219164974abdf848841528152: Deleting consensus metadata
I20260130 02:57:59.001576 19754 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42870 (request call id 52) took 1473 ms. Trace:
I20260130 02:57:59.002468 19754 rpcz_store.cc:276] 0130 02:57:57.528106 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:57:57.580551 (+ 52445us) service_pool.cc:225] Handling call
0130 02:57:58.554873 (+974322us) ts_tablet_manager.cc:1151] Deleting tablet 3b8cdac4bf4b46efa26236fc930eb9a6
0130 02:57:58.555043 (+ 170us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0130 02:57:58.751475 (+196432us) tablet_metadata.cc:641] Metadata flushed
0130 02:57:58.901321 (+149846us) tablet_metadata.cc:641] Metadata flushed
0130 02:57:59.001491 (+100170us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":972938}
I20260130 02:57:58.998558 18867 catalog_manager.cc:5746] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571) does not have the latest schema for tablet 2112b0402de5464cb1caf2c2abb67d78 (table default.Table_da3d581b15a94840898777f74611994e [id=b7e241ced53e4160ac6fa1f65ef3674e]). Expected version 1 got 0
I20260130 02:57:59.014060 19840 tablet_bootstrap.cc:492] T c3fce8cc99fb451ebd532fa37390e862 P 78c2c46c7520492db35d51c244fb4f74: No bootstrap required, opened a new log
I20260130 02:57:59.015645 19840 ts_tablet_manager.cc:1403] T c3fce8cc99fb451ebd532fa37390e862 P 78c2c46c7520492db35d51c244fb4f74: Time spent bootstrapping tablet: real 0.113s user 0.009s sys 0.016s
I20260130 02:57:59.012444 19224 raft_consensus.cc:3060] T ceda64471a92476f9ba183d2beab30d1 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:57:59.016973 18867 catalog_manager.cc:5746] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571) does not have the latest schema for tablet 878f9346d3044d8d8b15c61f86869212 (table default.Table_a31136243e6241a19ae437271d1cf283 [id=2009496525c94d7f8ee6ce59694b652d]). Expected version 1 got 0
I20260130 02:57:59.018100 19840 raft_consensus.cc:359] T c3fce8cc99fb451ebd532fa37390e862 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:57:59.018779 19840 raft_consensus.cc:385] T c3fce8cc99fb451ebd532fa37390e862 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:57:59.019045 19840 raft_consensus.cc:740] T c3fce8cc99fb451ebd532fa37390e862 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78c2c46c7520492db35d51c244fb4f74, State: Initialized, Role: FOLLOWER
I20260130 02:57:59.019577 19526 raft_consensus.cc:1275] T ceda64471a92476f9ba183d2beab30d1 P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Refusing update from remote peer 78c2c46c7520492db35d51c244fb4f74: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260130 02:57:59.019750 19507 tablet_service.cc:1505] Processing CreateTablet for tablet 1ca34772ef484c1f809a99d126e28eff (DEFAULT_TABLE table=default.Table_9cb9a9f9e5d145a6a42d7ccbc42aced9 [id=5f1f179fadc247da9382dd7796dcf2c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:57:59.021065 19507 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1ca34772ef484c1f809a99d126e28eff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:57:59.020866 19787 consensus_queue.cc:1048] T ceda64471a92476f9ba183d2beab30d1 P 78c2c46c7520492db35d51c244fb4f74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260130 02:57:59.019919 19840 consensus_queue.cc:260] T c3fce8cc99fb451ebd532fa37390e862 P 78c2c46c7520492db35d51c244fb4f74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:57:59.023353 19224 raft_consensus.cc:1275] T ceda64471a92476f9ba183d2beab30d1 P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Refusing update from remote peer 78c2c46c7520492db35d51c244fb4f74: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260130 02:57:59.014842 19200 tablet_service.cc:1505] Processing CreateTablet for tablet a743a82540014c85aeab0e9d0994b0dc (DEFAULT_TABLE table=default.Table_71ecfab2cca249b49070a8340586ce5e [id=19941bcff5454783adb47df8dd77f352]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:57:59.025723 19200 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a743a82540014c85aeab0e9d0994b0dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:57:59.027000 19754 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42870 (request call id 59) took 1159 ms. Trace:
I20260130 02:57:59.030124 19787 consensus_queue.cc:1048] T ceda64471a92476f9ba183d2beab30d1 P 78c2c46c7520492db35d51c244fb4f74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260130 02:57:59.035372 19840 ts_tablet_manager.cc:1434] T c3fce8cc99fb451ebd532fa37390e862 P 78c2c46c7520492db35d51c244fb4f74: Time spent starting tablet: real 0.019s user 0.004s sys 0.001s
I20260130 02:57:59.017397 19807 tablet_bootstrap.cc:492] T 1bbc55bbf2254d8bb6b721d7e8f69d85 P 371a36d219164974abdf848841528152: No bootstrap required, opened a new log
I20260130 02:57:59.037761 19807 ts_tablet_manager.cc:1403] T 1bbc55bbf2254d8bb6b721d7e8f69d85 P 371a36d219164974abdf848841528152: Time spent bootstrapping tablet: real 0.303s user 0.076s sys 0.083s
I20260130 02:57:59.040912 19840 tablet_bootstrap.cc:492] T 7a902d0d571a42bb82ebffd48f7eca3f P 78c2c46c7520492db35d51c244fb4f74: Bootstrap starting.
I20260130 02:57:59.041847 19527 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ceda64471a92476f9ba183d2beab30d1" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
I20260130 02:57:59.027333 19754 rpcz_store.cc:276] 0130 02:57:57.867214 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:57:57.867518 (+ 304us) service_pool.cc:225] Handling call
0130 02:57:59.026669 (+1159151us) ts_tablet_manager.cc:1151] Deleting tablet 7a902d0d571a42bb82ebffd48f7eca3f
0130 02:57:59.026709 (+ 40us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0130 02:57:59.026946 (+ 237us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1158582}
I20260130 02:57:59.042641 19807 raft_consensus.cc:359] T 1bbc55bbf2254d8bb6b721d7e8f69d85 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:57:59.043251 19807 raft_consensus.cc:385] T 1bbc55bbf2254d8bb6b721d7e8f69d85 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:57:59.043385 19754 tablet_replica.cc:333] T 93809949599a449ca09b13e9b8bbdef6 P 371a36d219164974abdf848841528152: stopping tablet replica
I20260130 02:57:59.043784 19159 leader_election.cc:304] T ceda64471a92476f9ba183d2beab30d1 P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 371a36d219164974abdf848841528152; no voters: 78c2c46c7520492db35d51c244fb4f74, d870164263bb44b6b780a557daea3656
I20260130 02:57:59.044304 19754 raft_consensus.cc:2243] T 93809949599a449ca09b13e9b8bbdef6 P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260130 02:57:59.044806 19754 raft_consensus.cc:2272] T 93809949599a449ca09b13e9b8bbdef6 P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260130 02:57:59.043524 19807 raft_consensus.cc:740] T 1bbc55bbf2254d8bb6b721d7e8f69d85 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 371a36d219164974abdf848841528152, State: Initialized, Role: FOLLOWER
I20260130 02:57:59.046788 19840 tablet_bootstrap.cc:654] T 7a902d0d571a42bb82ebffd48f7eca3f P 78c2c46c7520492db35d51c244fb4f74: Neither blocks nor log segments found. Creating new log.
I20260130 02:57:59.046691 19807 consensus_queue.cc:260] T 1bbc55bbf2254d8bb6b721d7e8f69d85 P 371a36d219164974abdf848841528152 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:57:59.052397 19807 ts_tablet_manager.cc:1434] T 1bbc55bbf2254d8bb6b721d7e8f69d85 P 371a36d219164974abdf848841528152: Time spent starting tablet: real 0.013s user 0.006s sys 0.000s
I20260130 02:57:59.053237 19807 tablet_bootstrap.cc:492] T ffa1142fc7274aecbdbee1a0802b95da P 371a36d219164974abdf848841528152: Bootstrap starting.
I20260130 02:57:59.058316 19807 tablet_bootstrap.cc:654] T ffa1142fc7274aecbdbee1a0802b95da P 371a36d219164974abdf848841528152: Neither blocks nor log segments found. Creating new log.
I20260130 02:57:59.026502 18861 catalog_manager.cc:4985] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021): tablet 3b8cdac4bf4b46efa26236fc930eb9a6 (table default.Table_9428a7751159470793188f91cdcba4f3 [id=eaed0f6841e94faa858449fbb74bd5c4]) successfully deleted
W20260130 02:57:59.061234 18861 catalog_manager.cc:4965] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021): delete failed for tablet 7a902d0d571a42bb82ebffd48f7eca3f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7a902d0d571a42bb82ebffd48f7eca3f already in progress: creating tablet
I20260130 02:57:59.064728 18869 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50284:
name: "default.Table_94b23422c82d419d97dc097a3efb92fc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260130 02:57:59.066298 18869 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_94b23422c82d419d97dc097a3efb92fc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260130 02:57:59.077900 19813 tablet.cc:1722] T 878f9346d3044d8d8b15c61f86869212 P 371a36d219164974abdf848841528152: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20260130 02:57:59.082602 19840 tablet_bootstrap.cc:492] T 7a902d0d571a42bb82ebffd48f7eca3f P 78c2c46c7520492db35d51c244fb4f74: No bootstrap required, opened a new log
I20260130 02:57:59.083074 19840 ts_tablet_manager.cc:1403] T 7a902d0d571a42bb82ebffd48f7eca3f P 78c2c46c7520492db35d51c244fb4f74: Time spent bootstrapping tablet: real 0.042s user 0.007s sys 0.005s
I20260130 02:57:59.085831 19202 tablet_service.cc:1505] Processing CreateTablet for tablet 955b9547f49f434699fe1d86b260638c (DEFAULT_TABLE table=default.Table_f246882197f44c69b72ca421cbdf201f [id=f778784dc88a4e4890113916098ad20f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:57:59.087177 19202 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 955b9547f49f434699fe1d86b260638c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:57:59.087556 19874 tablet.cc:1722] T ceda64471a92476f9ba183d2beab30d1 P 78c2c46c7520492db35d51c244fb4f74: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260130 02:57:59.088971 19840 raft_consensus.cc:359] T 7a902d0d571a42bb82ebffd48f7eca3f P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:57:59.091907 19840 raft_consensus.cc:385] T 7a902d0d571a42bb82ebffd48f7eca3f P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:57:59.092201 19840 raft_consensus.cc:740] T 7a902d0d571a42bb82ebffd48f7eca3f P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78c2c46c7520492db35d51c244fb4f74, State: Initialized, Role: FOLLOWER
I20260130 02:57:59.092938 19840 consensus_queue.cc:260] T 7a902d0d571a42bb82ebffd48f7eca3f P 78c2c46c7520492db35d51c244fb4f74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:57:59.099103 19663 raft_consensus.cc:493] T 1bbc55bbf2254d8bb6b721d7e8f69d85 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:57:59.099524 19663 raft_consensus.cc:515] T 1bbc55bbf2254d8bb6b721d7e8f69d85 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:57:59.099735 19199 tablet_service.cc:1505] Processing CreateTablet for tablet bd43c5d148514836869a8ac648c8ad11 (DEFAULT_TABLE table=default.Table_a9476edf8a5445c883f8d6ce786554a2 [id=7e29f588b41b491e8a9d1ad61e50b033]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:57:59.100826 19199 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bd43c5d148514836869a8ac648c8ad11. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:57:59.101219 19663 leader_election.cc:290] T 1bbc55bbf2254d8bb6b721d7e8f69d85 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d870164263bb44b6b780a557daea3656 (127.18.100.67:39571), 371a36d219164974abdf848841528152 (127.18.100.65:42021)
I20260130 02:57:59.103013 19880 tablet.cc:1722] T ceda64471a92476f9ba183d2beab30d1 P d870164263bb44b6b780a557daea3656: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260130 02:57:59.103472 19221 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1bbc55bbf2254d8bb6b721d7e8f69d85" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152" is_pre_election: true
I20260130 02:57:59.104110 19221 raft_consensus.cc:2468] T 1bbc55bbf2254d8bb6b721d7e8f69d85 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78c2c46c7520492db35d51c244fb4f74 in term 0.
I20260130 02:57:59.104964 19525 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1bbc55bbf2254d8bb6b721d7e8f69d85" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
I20260130 02:57:59.105032 19314 leader_election.cc:304] T 1bbc55bbf2254d8bb6b721d7e8f69d85 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 371a36d219164974abdf848841528152, 78c2c46c7520492db35d51c244fb4f74; no voters:
I20260130 02:57:59.106642 19663 raft_consensus.cc:2804] T 1bbc55bbf2254d8bb6b721d7e8f69d85 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260130 02:57:59.106998 19663 raft_consensus.cc:493] T 1bbc55bbf2254d8bb6b721d7e8f69d85 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260130 02:57:59.107338 19663 raft_consensus.cc:3060] T 1bbc55bbf2254d8bb6b721d7e8f69d85 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Advancing to term 1
W20260130 02:57:59.107062 19313 leader_election.cc:343] T 1bbc55bbf2254d8bb6b721d7e8f69d85 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:57:59.056882 18860 catalog_manager.cc:4985] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767): tablet 93809949599a449ca09b13e9b8bbdef6 (table default.Table_1c77605822574b10a05074a093ad30a3 [id=aeba5f5a21564264a8cd975244774ab8]) successfully deleted
W20260130 02:57:59.114709 18860 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.18.100.66:40767, user_credentials={real_user=slave}} blocked reactor thread for 57847.9us
W20260130 02:57:59.115288 18860 catalog_manager.cc:4965] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767): delete failed for tablet d701152960b14396a6ac1bd10480d2b6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d701152960b14396a6ac1bd10480d2b6 already in progress: creating tablet
W20260130 02:57:59.115876 18860 catalog_manager.cc:4965] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767): delete failed for tablet 5b2976108584478a98fd90fe6594af66 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5b2976108584478a98fd90fe6594af66 already in progress: creating tablet
I20260130 02:57:59.116397 19201 tablet_service.cc:1505] Processing CreateTablet for tablet b7b596f944be4daf953b82b65d5cec2a (DEFAULT_TABLE table=default.Table_22d10332a83f4700ac21892bab67f46b [id=da3a9cb1f6d5419abc1afc93659b5324]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:57:59.117447 19201 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b7b596f944be4daf953b82b65d5cec2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:57:59.130597 19874 tablet.cc:1722] T 878f9346d3044d8d8b15c61f86869212 P 78c2c46c7520492db35d51c244fb4f74: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20260130 02:57:59.134440 19850 raft_consensus.cc:493] T ffa1142fc7274aecbdbee1a0802b95da P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:57:59.134940 19850 raft_consensus.cc:515] T ffa1142fc7274aecbdbee1a0802b95da P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:57:59.136745 19850 leader_election.cc:290] T ffa1142fc7274aecbdbee1a0802b95da P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d870164263bb44b6b780a557daea3656 (127.18.100.67:39571), 371a36d219164974abdf848841528152 (127.18.100.65:42021)
I20260130 02:57:59.138492 19221 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ffa1142fc7274aecbdbee1a0802b95da" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152" is_pre_election: true
W20260130 02:57:59.140004 19314 leader_election.cc:343] T ffa1142fc7274aecbdbee1a0802b95da P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:57:59.150007 19880 tablet.cc:1722] T 878f9346d3044d8d8b15c61f86869212 P d870164263bb44b6b780a557daea3656: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20260130 02:57:59.152109 19527 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ffa1142fc7274aecbdbee1a0802b95da" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
I20260130 02:57:59.152850 19527 raft_consensus.cc:2468] T ffa1142fc7274aecbdbee1a0802b95da P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78c2c46c7520492db35d51c244fb4f74 in term 0.
I20260130 02:57:59.153936 19313 leader_election.cc:304] T ffa1142fc7274aecbdbee1a0802b95da P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 78c2c46c7520492db35d51c244fb4f74, d870164263bb44b6b780a557daea3656; no voters: 371a36d219164974abdf848841528152
I20260130 02:57:59.114234 19663 raft_consensus.cc:515] T 1bbc55bbf2254d8bb6b721d7e8f69d85 P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:57:59.155576 19850 raft_consensus.cc:2804] T ffa1142fc7274aecbdbee1a0802b95da P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260130 02:57:59.155977 19850 raft_consensus.cc:493] T ffa1142fc7274aecbdbee1a0802b95da P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260130 02:57:59.156337 19850 raft_consensus.cc:3060] T ffa1142fc7274aecbdbee1a0802b95da P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:57:59.157462 19221 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1bbc55bbf2254d8bb6b721d7e8f69d85" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152"
I20260130 02:57:59.158300 19221 raft_consensus.cc:3060] T 1bbc55bbf2254d8bb6b721d7e8f69d85 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:57:59.164634 19527 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1bbc55bbf2254d8bb6b721d7e8f69d85" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656"
I20260130 02:57:59.166200 19221 raft_consensus.cc:2468] T 1bbc55bbf2254d8bb6b721d7e8f69d85 P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78c2c46c7520492db35d51c244fb4f74 in term 1.
W20260130 02:57:59.166853 19313 leader_election.cc:343] T 1bbc55bbf2254d8bb6b721d7e8f69d85 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:57:59.167356 19314 leader_election.cc:304] T 1bbc55bbf2254d8bb6b721d7e8f69d85 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 371a36d219164974abdf848841528152, 78c2c46c7520492db35d51c244fb4f74; no voters: d870164263bb44b6b780a557daea3656
I20260130 02:57:59.168171 19856 raft_consensus.cc:2804] T 1bbc55bbf2254d8bb6b721d7e8f69d85 P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Leader election won for term 1
I20260130 02:57:59.168627 19856 raft_consensus.cc:697] T 1bbc55bbf2254d8bb6b721d7e8f69d85 P 78c2c46c7520492db35d51c244fb4f74 [term 1 LEADER]: Becoming Leader. State: Replica: 78c2c46c7520492db35d51c244fb4f74, State: Running, Role: LEADER
I20260130 02:57:59.169173 19850 raft_consensus.cc:515] T ffa1142fc7274aecbdbee1a0802b95da P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:57:59.173743 19861 raft_consensus.cc:2749] T ceda64471a92476f9ba183d2beab30d1 P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260130 02:57:59.186395 19813 tablet.cc:1722] T ceda64471a92476f9ba183d2beab30d1 P 371a36d219164974abdf848841528152: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260130 02:57:59.170814 19625 tablet_bootstrap.cc:700] T 1bbc55bbf2254d8bb6b721d7e8f69d85 P d870164263bb44b6b780a557daea3656: Time spent opening tablet: real 0.119s user 0.000s sys 0.002s
I20260130 02:57:59.192111 19625 tablet_bootstrap.cc:654] T 1bbc55bbf2254d8bb6b721d7e8f69d85 P d870164263bb44b6b780a557daea3656: Neither blocks nor log segments found. Creating new log.
I20260130 02:57:59.169454 19856 consensus_queue.cc:237] T 1bbc55bbf2254d8bb6b721d7e8f69d85 P 78c2c46c7520492db35d51c244fb4f74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:57:59.177631 19377 raft_consensus.cc:1275] T 4035b8a330844dd28af9f3244bf8eb9a P 78c2c46c7520492db35d51c244fb4f74 [term 2 FOLLOWER]: Refusing update from remote peer 371a36d219164974abdf848841528152: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260130 02:57:59.191912 19663 leader_election.cc:290] T 1bbc55bbf2254d8bb6b721d7e8f69d85 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 election: Requested vote from peers d870164263bb44b6b780a557daea3656 (127.18.100.67:39571), 371a36d219164974abdf848841528152 (127.18.100.65:42021)
I20260130 02:57:59.203054 19861 consensus_queue.cc:1048] T 4035b8a330844dd28af9f3244bf8eb9a P 371a36d219164974abdf848841528152 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260130 02:57:59.207278 19221 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ffa1142fc7274aecbdbee1a0802b95da" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152"
W20260130 02:57:59.208529 19314 leader_election.cc:343] T ffa1142fc7274aecbdbee1a0802b95da P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:57:59.210590 19527 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ffa1142fc7274aecbdbee1a0802b95da" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656"
I20260130 02:57:59.211203 19527 raft_consensus.cc:3060] T ffa1142fc7274aecbdbee1a0802b95da P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:57:59.221632 19527 raft_consensus.cc:2468] T ffa1142fc7274aecbdbee1a0802b95da P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78c2c46c7520492db35d51c244fb4f74 in term 1.
I20260130 02:57:59.224681 19313 leader_election.cc:304] T ffa1142fc7274aecbdbee1a0802b95da P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 78c2c46c7520492db35d51c244fb4f74, d870164263bb44b6b780a557daea3656; no voters: 371a36d219164974abdf848841528152
I20260130 02:57:59.225673 19856 raft_consensus.cc:2804] T ffa1142fc7274aecbdbee1a0802b95da P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Leader election won for term 1
I20260130 02:57:59.226135 19856 raft_consensus.cc:697] T ffa1142fc7274aecbdbee1a0802b95da P 78c2c46c7520492db35d51c244fb4f74 [term 1 LEADER]: Becoming Leader. State: Replica: 78c2c46c7520492db35d51c244fb4f74, State: Running, Role: LEADER
I20260130 02:57:59.226918 19856 consensus_queue.cc:237] T ffa1142fc7274aecbdbee1a0802b95da P 78c2c46c7520492db35d51c244fb4f74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:57:59.232170 19360 tablet_service.cc:1505] Processing CreateTablet for tablet a743a82540014c85aeab0e9d0994b0dc (DEFAULT_TABLE table=default.Table_71ecfab2cca249b49070a8340586ce5e [id=19941bcff5454783adb47df8dd77f352]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:57:59.233175 19360 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a743a82540014c85aeab0e9d0994b0dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:57:59.240904 19354 tablet_service.cc:1505] Processing CreateTablet for tablet 955b9547f49f434699fe1d86b260638c (DEFAULT_TABLE table=default.Table_f246882197f44c69b72ca421cbdf201f [id=f778784dc88a4e4890113916098ad20f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:57:59.242101 19354 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 955b9547f49f434699fe1d86b260638c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:57:59.263001 19754 ts_tablet_manager.cc:1916] T 93809949599a449ca09b13e9b8bbdef6 P 371a36d219164974abdf848841528152: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260130 02:57:59.266948 19527 raft_consensus.cc:1275] T 4035b8a330844dd28af9f3244bf8eb9a P d870164263bb44b6b780a557daea3656 [term 2 FOLLOWER]: Refusing update from remote peer 371a36d219164974abdf848841528152: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260130 02:57:59.272890 19504 tablet_service.cc:1505] Processing CreateTablet for tablet 955b9547f49f434699fe1d86b260638c (DEFAULT_TABLE table=default.Table_f246882197f44c69b72ca421cbdf201f [id=f778784dc88a4e4890113916098ad20f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:57:59.274040 19508 tablet_service.cc:1505] Processing CreateTablet for tablet bd43c5d148514836869a8ac648c8ad11 (DEFAULT_TABLE table=default.Table_a9476edf8a5445c883f8d6ce786554a2 [id=7e29f588b41b491e8a9d1ad61e50b033]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:57:59.294697 19508 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bd43c5d148514836869a8ac648c8ad11. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:57:59.277563 19505 tablet_service.cc:1505] Processing CreateTablet for tablet b7b596f944be4daf953b82b65d5cec2a (DEFAULT_TABLE table=default.Table_22d10332a83f4700ac21892bab67f46b [id=da3a9cb1f6d5419abc1afc93659b5324]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:57:59.302834 19807 tablet_bootstrap.cc:492] T ffa1142fc7274aecbdbee1a0802b95da P 371a36d219164974abdf848841528152: No bootstrap required, opened a new log
I20260130 02:57:59.303321 19807 ts_tablet_manager.cc:1403] T ffa1142fc7274aecbdbee1a0802b95da P 371a36d219164974abdf848841528152: Time spent bootstrapping tablet: real 0.250s user 0.053s sys 0.076s
I20260130 02:57:59.305491 19357 tablet_service.cc:1505] Processing CreateTablet for tablet bd43c5d148514836869a8ac648c8ad11 (DEFAULT_TABLE table=default.Table_a9476edf8a5445c883f8d6ce786554a2 [id=7e29f588b41b491e8a9d1ad61e50b033]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:57:59.305522 19356 tablet_service.cc:1505] Processing CreateTablet for tablet b7b596f944be4daf953b82b65d5cec2a (DEFAULT_TABLE table=default.Table_22d10332a83f4700ac21892bab67f46b [id=da3a9cb1f6d5419abc1afc93659b5324]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:57:59.305898 18872 catalog_manager.cc:5941] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4: Sending DeleteTablet for 3 replicas of tablet d4f2b54930cc4f5c825562e3e0a79f4d
I20260130 02:57:59.305694 19807 raft_consensus.cc:359] T ffa1142fc7274aecbdbee1a0802b95da P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:57:59.306255 19807 raft_consensus.cc:385] T ffa1142fc7274aecbdbee1a0802b95da P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:57:59.306531 19357 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bd43c5d148514836869a8ac648c8ad11. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:57:59.306484 19807 raft_consensus.cc:740] T ffa1142fc7274aecbdbee1a0802b95da P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 371a36d219164974abdf848841528152, State: Initialized, Role: FOLLOWER
I20260130 02:57:59.307041 19356 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b7b596f944be4daf953b82b65d5cec2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:57:59.307163 18872 catalog_manager.cc:6724] ReplaceTablet: tablet d4f2b54930cc4f5c825562e3e0a79f4d deleted and replaced by tablet 54be22352e404ff9b798a075693f7f21
I20260130 02:57:59.307088 19807 consensus_queue.cc:260] T ffa1142fc7274aecbdbee1a0802b95da P 371a36d219164974abdf848841528152 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:57:59.309322 19203 tablet_service.cc:1552] Processing DeleteTablet for tablet d4f2b54930cc4f5c825562e3e0a79f4d with delete_type TABLET_DATA_DELETED (replaced by tablet 54be22352e404ff9b798a075693f7f21) from {username='slave'} at 127.0.0.1:42870
I20260130 02:57:59.310140 19807 ts_tablet_manager.cc:1434] T ffa1142fc7274aecbdbee1a0802b95da P 371a36d219164974abdf848841528152: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20260130 02:57:59.311019 19807 tablet_bootstrap.cc:492] T e364f4e70e5749e083e14e7fa7416ff0 P 371a36d219164974abdf848841528152: Bootstrap starting.
I20260130 02:57:59.314915 19840 ts_tablet_manager.cc:1434] T 7a902d0d571a42bb82ebffd48f7eca3f P 78c2c46c7520492db35d51c244fb4f74: Time spent starting tablet: real 0.232s user 0.005s sys 0.001s
I20260130 02:57:59.315933 19353 tablet_service.cc:1552] Processing DeleteTablet for tablet d4f2b54930cc4f5c825562e3e0a79f4d with delete_type TABLET_DATA_DELETED (replaced by tablet 54be22352e404ff9b798a075693f7f21) from {username='slave'} at 127.0.0.1:60110
I20260130 02:57:59.317469 19758 tablet_replica.cc:333] T d4f2b54930cc4f5c825562e3e0a79f4d P 78c2c46c7520492db35d51c244fb4f74: stopping tablet replica
I20260130 02:57:59.321446 18874 catalog_manager.cc:5746] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021) does not have the latest schema for tablet ceda64471a92476f9ba183d2beab30d1 (table default.Table_90ad2e09fa9b4952a1564cee70ab8c1c [id=4f59d133b0cd4773bb012145927a92a4]). Expected version 1 got 0
I20260130 02:57:59.322115 19807 tablet_bootstrap.cc:654] T e364f4e70e5749e083e14e7fa7416ff0 P 371a36d219164974abdf848841528152: Neither blocks nor log segments found. Creating new log.
I20260130 02:57:59.278636 19850 leader_election.cc:290] T ffa1142fc7274aecbdbee1a0802b95da P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 election: Requested vote from peers d870164263bb44b6b780a557daea3656 (127.18.100.67:39571), 371a36d219164974abdf848841528152 (127.18.100.65:42021)
I20260130 02:57:59.323820 19840 tablet_bootstrap.cc:492] T e364f4e70e5749e083e14e7fa7416ff0 P 78c2c46c7520492db35d51c244fb4f74: Bootstrap starting.
I20260130 02:57:59.324573 18874 catalog_manager.cc:5746] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021) does not have the latest schema for tablet 878f9346d3044d8d8b15c61f86869212 (table default.Table_8edea1fae34d4d18b54b0f801a145225 [id=2009496525c94d7f8ee6ce59694b652d]). Expected version 2 got 1
I20260130 02:57:59.331876 18874 catalog_manager.cc:5746] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021) does not have the latest schema for tablet 37470ce4356f4542b621e21698551209 (table default.Table_007e969f70d345a7bcb3dfbaabbc025d [id=77d4fa8705a3419cb0a84d3ca1bb4ce3]). Expected version 1 got 0
I20260130 02:57:59.333524 18874 catalog_manager.cc:5654] T 073b5941850c4140b1751956dd7cf5ef P 371a36d219164974abdf848841528152 reported cstate change: term changed from 0 to 1, leader changed from <none> to 371a36d219164974abdf848841528152 (127.18.100.65). New cstate: current_term: 1 leader_uuid: "371a36d219164974abdf848841528152" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } health_report { overall_health: UNKNOWN } } }
I20260130 02:57:59.335989 19506 tablet_service.cc:1552] Processing DeleteTablet for tablet d4f2b54930cc4f5c825562e3e0a79f4d with delete_type TABLET_DATA_DELETED (replaced by tablet 54be22352e404ff9b798a075693f7f21) from {username='slave'} at 127.0.0.1:59068
I20260130 02:57:59.336691 19710 tablet_replica.cc:333] T d4f2b54930cc4f5c825562e3e0a79f4d P d870164263bb44b6b780a557daea3656: stopping tablet replica
I20260130 02:57:59.337458 19710 raft_consensus.cc:2243] T d4f2b54930cc4f5c825562e3e0a79f4d P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260130 02:57:59.337987 19710 raft_consensus.cc:2272] T d4f2b54930cc4f5c825562e3e0a79f4d P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260130 02:57:59.337467 18874 catalog_manager.cc:5654] T 2112b0402de5464cb1caf2c2abb67d78 P 371a36d219164974abdf848841528152 reported cstate change: term changed from 0 to 1, leader changed from <none> to 371a36d219164974abdf848841528152 (127.18.100.65). New cstate: current_term: 1 leader_uuid: "371a36d219164974abdf848841528152" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } health_report { overall_health: UNKNOWN } } }
I20260130 02:57:59.340233 19504 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 955b9547f49f434699fe1d86b260638c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:57:59.339880 18874 catalog_manager.cc:5746] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021) does not have the latest schema for tablet 2112b0402de5464cb1caf2c2abb67d78 (table default.Table_da3d581b15a94840898777f74611994e [id=b7e241ced53e4160ac6fa1f65ef3674e]). Expected version 1 got 0
I20260130 02:57:59.343936 18867 catalog_manager.cc:6199] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4: default.Table_4d43592dc5b24ff28710e62ad4d87ce0 [id=27adf356fd004597a54c7a9887dd3ca8] alter complete (version 3)
I20260130 02:57:59.345116 18870 catalog_manager.cc:6199] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4: default.Table_ebc8b8379c274c18b0a06a919dbe0ee0 [id=28156e0843674fce88f5914a3e4195ae] alter complete (version 1)
I20260130 02:57:59.350662 18873 catalog_manager.cc:5941] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4: Sending DeleteTablet for 3 replicas of tablet 4035b8a330844dd28af9f3244bf8eb9a
I20260130 02:57:59.352142 18873 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:50284 (request call id 89) took 1284 ms. Trace:
I20260130 02:57:59.353169 19506 tablet_service.cc:1552] Processing DeleteTablet for tablet 4035b8a330844dd28af9f3244bf8eb9a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:57:58 UTC) from {username='slave'} at 127.0.0.1:59068
I20260130 02:57:59.353327 19505 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b7b596f944be4daf953b82b65d5cec2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:57:59.352458 18873 rpcz_store.cc:276] 0130 02:57:58.067787 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:57:58.067910 (+ 123us) service_pool.cc:225] Handling call
0130 02:57:58.079429 (+ 11519us) catalog_manager.cc:2393] Looking up, locking, and authorizing table
0130 02:57:58.079984 (+ 555us) catalog_manager.cc:2815] Modifying in-memory table state
0130 02:57:58.080100 (+ 116us) catalog_manager.cc:2824] Locking tablets
0130 02:57:58.749486 (+669386us) catalog_manager.cc:2838] Removing table and tablets from system table
0130 02:57:58.797276 (+ 47790us) catalog_manager.cc:2856] Removing table from by-name map
0130 02:57:58.797494 (+ 218us) catalog_manager.cc:2872] Committing in-memory state
0130 02:57:59.350554 (+553060us) catalog_manager.cc:2876] Aborting table tasks
0130 02:57:59.352082 (+ 1528us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0130 02:57:58.751422 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0130 02:57:58.751794 (+ 372us) write_op.cc:432] Acquiring schema lock in shared mode
0130 02:57:58.751817 (+ 23us) write_op.cc:435] Acquired schema lock
0130 02:57:58.751829 (+ 12us) tablet.cc:660] Decoding operations
0130 02:57:58.752120 (+ 291us) write_op.cc:620] Acquiring the partition lock for write op
0130 02:57:58.752158 (+ 38us) write_op.cc:641] Partition lock acquired for write op
0130 02:57:58.752180 (+ 22us) tablet.cc:683] Acquiring locks for 2 operations
0130 02:57:58.752421 (+ 241us) tablet.cc:699] Row locks acquired
0130 02:57:58.752435 (+ 14us) write_op.cc:260] PREPARE: finished
0130 02:57:58.752549 (+ 114us) write_op.cc:270] Start()
0130 02:57:58.752624 (+ 75us) write_op.cc:276] Timestamp: P: 1769741878752467 usec, L: 0
0130 02:57:58.752639 (+ 15us) op_driver.cc:348] REPLICATION: starting
0130 02:57:58.753063 (+ 424us) log.cc:844] Serialized 765 byte log entry
0130 02:57:58.795261 (+ 42198us) op_driver.cc:464] REPLICATION: finished
0130 02:57:58.795717 (+ 456us) write_op.cc:301] APPLY: starting
0130 02:57:58.795772 (+ 55us) tablet.cc:1365] starting BulkCheckPresence
0130 02:57:58.795874 (+ 102us) tablet.cc:1368] finished BulkCheckPresence
0130 02:57:58.795957 (+ 83us) tablet.cc:1370] starting ApplyRowOperation cycle
0130 02:57:58.796174 (+ 217us) tablet.cc:1381] finished ApplyRowOperation cycle
0130 02:57:58.796302 (+ 128us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0130 02:57:58.796343 (+ 41us) write_op.cc:312] APPLY: finished
0130 02:57:58.796525 (+ 182us) log.cc:844] Serialized 29 byte log entry
0130 02:57:58.796666 (+ 141us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:57:58.796730 (+ 64us) write_op.cc:454] Released schema lock
0130 02:57:58.796761 (+ 31us) write_op.cc:341] FINISH: Updating metrics
0130 02:57:58.796976 (+ 215us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:57:58.797000 (+ 24us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":301,"apply.run_cpu_time_us":1226,"apply.run_wall_time_us":1226,"num_ops":2,"prepare.queue_time_us":109,"prepare.run_cpu_time_us":1881,"prepare.run_wall_time_us":2016,"raft.queue_time_us":75,"raft.run_cpu_time_us":10,"raft.run_wall_time_us":11,"replication_time_us":42528}]]}
I20260130 02:57:59.356848 19710 ts_tablet_manager.cc:1916] T d4f2b54930cc4f5c825562e3e0a79f4d P d870164263bb44b6b780a557daea3656: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260130 02:57:59.358367 19201 tablet_service.cc:1552] Processing DeleteTablet for tablet 4035b8a330844dd28af9f3244bf8eb9a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:57:58 UTC) from {username='slave'} at 127.0.0.1:42870
I20260130 02:57:59.359155 18871 catalog_manager.cc:6199] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4: default.Table_b63ca4e4060f48e893b0099fb6433f04 [id=720decfb55c24f34983ef636f06e3169] alter complete (version 3)
I20260130 02:57:59.369782 18866 catalog_manager.cc:6199] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4: default.Table_14c76bd65820453fb5cade67cf66e752 [id=d128767e303145a885eec3c5f714a27f] alter complete (version 1)
I20260130 02:57:59.382699 19506 tablet_service.cc:1552] Processing DeleteTablet for tablet 3b8cdac4bf4b46efa26236fc930eb9a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:57:54 UTC) from {username='slave'} at 127.0.0.1:59068
I20260130 02:57:59.383396 19503 tablet_service.cc:1552] Processing DeleteTablet for tablet 01db06fcfffe4b259ac4240c0df38047 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:57:56 UTC) from {username='slave'} at 127.0.0.1:59068
W20260130 02:57:59.392338 18866 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.18.100.66:59691 (request call id 17) took 2058 ms (client timeout 1000 ms). Trace:
I20260130 02:57:59.395661 19503 tablet_service.cc:1552] Processing DeleteTablet for tablet 93809949599a449ca09b13e9b8bbdef6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:57:57 UTC) from {username='slave'} at 127.0.0.1:59068
I20260130 02:57:59.396268 19754 ts_tablet_manager.cc:1929] T 93809949599a449ca09b13e9b8bbdef6 P 371a36d219164974abdf848841528152: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260130 02:57:59.279352 19507 tablet_service.cc:1505] Processing CreateTablet for tablet a743a82540014c85aeab0e9d0994b0dc (DEFAULT_TABLE table=default.Table_71ecfab2cca249b49070a8340586ce5e [id=19941bcff5454783adb47df8dd77f352]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:57:59.400938 19507 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a743a82540014c85aeab0e9d0994b0dc. 1 dirs total, 0 dirs full, 0 dirs failed
W20260130 02:57:59.403851 18870 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.18.100.67:47213 (request call id 15) took 1924 ms (client timeout 1000 ms). Trace:
I20260130 02:57:59.407377 19503 tablet_service.cc:1552] Processing DeleteTablet for tablet 30d8136b5ea742bca20ee46b5634d04e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:57:56 UTC) from {username='slave'} at 127.0.0.1:59068
I20260130 02:57:59.408768 19358 tablet_service.cc:1552] Processing DeleteTablet for tablet 4035b8a330844dd28af9f3244bf8eb9a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:57:58 UTC) from {username='slave'} at 127.0.0.1:60110
W20260130 02:57:59.404188 18870 rpcz_store.cc:269] 0130 02:57:57.479311 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:57:57.479445 (+ 134us) service_pool.cc:225] Handling call
0130 02:57:59.403778 (+1924333us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0130 02:57:58.115223 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0130 02:57:58.115613 (+ 390us) write_op.cc:432] Acquiring schema lock in shared mode
0130 02:57:58.115635 (+ 22us) write_op.cc:435] Acquired schema lock
0130 02:57:58.115647 (+ 12us) tablet.cc:660] Decoding operations
0130 02:57:58.115889 (+ 242us) write_op.cc:620] Acquiring the partition lock for write op
0130 02:57:58.115924 (+ 35us) write_op.cc:641] Partition lock acquired for write op
0130 02:57:58.115945 (+ 21us) tablet.cc:683] Acquiring locks for 1 operations
0130 02:57:58.116073 (+ 128us) tablet.cc:699] Row locks acquired
0130 02:57:58.116085 (+ 12us) write_op.cc:260] PREPARE: finished
0130 02:57:58.116186 (+ 101us) write_op.cc:270] Start()
0130 02:57:58.117692 (+ 1506us) write_op.cc:276] Timestamp: P: 1769741878116118 usec, L: 0
0130 02:57:58.117710 (+ 18us) op_driver.cc:348] REPLICATION: starting
0130 02:57:58.118227 (+ 517us) log.cc:844] Serialized 355 byte log entry
0130 02:57:58.419763 (+301536us) op_driver.cc:464] REPLICATION: finished
0130 02:57:58.422111 (+ 2348us) write_op.cc:301] APPLY: starting
0130 02:57:58.422161 (+ 50us) tablet.cc:1365] starting BulkCheckPresence
0130 02:57:58.422249 (+ 88us) tablet.cc:1368] finished BulkCheckPresence
0130 02:57:58.422332 (+ 83us) tablet.cc:1370] starting ApplyRowOperation cycle
0130 02:57:58.422490 (+ 158us) tablet.cc:1381] finished ApplyRowOperation cycle
0130 02:57:58.442877 (+ 20387us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0130 02:57:58.442925 (+ 48us) write_op.cc:312] APPLY: finished
0130 02:57:58.443103 (+ 178us) log.cc:844] Serialized 22 byte log entry
0130 02:57:58.443244 (+ 141us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:57:58.443298 (+ 54us) write_op.cc:454] Released schema lock
0130 02:57:58.443335 (+ 37us) write_op.cc:341] FINISH: Updating metrics
0130 02:57:58.443565 (+ 230us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:57:58.443589 (+ 24us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":13,"child_traces":[["op",{"apply.queue_time_us":2226,"apply.run_cpu_time_us":1214,"apply.run_wall_time_us":21419,"num_ops":1,"prepare.queue_time_us":104,"prepare.run_cpu_time_us":1831,"prepare.run_wall_time_us":4274,"replication_time_us":301974}]]}
I20260130 02:57:59.410174 18870 master_service.cc:954] ReplaceTablet: received request to replace tablet 4035b8a330844dd28af9f3244bf8eb9a from {username='slave'} at 127.0.0.1:50284
I20260130 02:57:59.410282 19355 tablet_service.cc:1552] Processing DeleteTablet for tablet 93809949599a449ca09b13e9b8bbdef6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:57:57 UTC) from {username='slave'} at 127.0.0.1:60110
I20260130 02:57:59.412271 19754 log.cc:1199] T 93809949599a449ca09b13e9b8bbdef6 P 371a36d219164974abdf848841528152: Deleting WAL directory at /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-0/wal/wals/93809949599a449ca09b13e9b8bbdef6
I20260130 02:57:59.398876 19635 consensus_queue.cc:1048] T 4035b8a330844dd28af9f3244bf8eb9a P 371a36d219164974abdf848841528152 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20260130 02:57:59.399554 19200 tablet_service.cc:1552] Processing DeleteTablet for tablet 79e28c3a566c4521ac2453bbaa3567ed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:57:53 UTC) from {username='slave'} at 127.0.0.1:42870
I20260130 02:57:59.411592 19202 tablet_service.cc:1552] Processing DeleteTablet for tablet 30d8136b5ea742bca20ee46b5634d04e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:57:56 UTC) from {username='slave'} at 127.0.0.1:42870
I20260130 02:57:59.412645 19199 tablet_service.cc:1552] Processing DeleteTablet for tablet 3b8cdac4bf4b46efa26236fc930eb9a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:57:54 UTC) from {username='slave'} at 127.0.0.1:42870
I20260130 02:57:59.408275 19201 tablet_service.cc:1552] Processing DeleteTablet for tablet 01db06fcfffe4b259ac4240c0df38047 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:57:56 UTC) from {username='slave'} at 127.0.0.1:42870
I20260130 02:57:59.414685 19754 ts_tablet_manager.cc:1950] T 93809949599a449ca09b13e9b8bbdef6 P 371a36d219164974abdf848841528152: Deleting consensus metadata
I20260130 02:57:59.415584 19352 tablet_service.cc:1552] Processing DeleteTablet for tablet 3b8cdac4bf4b46efa26236fc930eb9a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:57:54 UTC) from {username='slave'} at 127.0.0.1:60110
I20260130 02:57:59.416229 19358 tablet_service.cc:1552] Processing DeleteTablet for tablet 30d8136b5ea742bca20ee46b5634d04e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:57:56 UTC) from {username='slave'} at 127.0.0.1:60110
I20260130 02:57:59.423353 19625 tablet_bootstrap.cc:492] T 1bbc55bbf2254d8bb6b721d7e8f69d85 P d870164263bb44b6b780a557daea3656: No bootstrap required, opened a new log
I20260130 02:57:59.423792 19625 ts_tablet_manager.cc:1403] T 1bbc55bbf2254d8bb6b721d7e8f69d85 P d870164263bb44b6b780a557daea3656: Time spent bootstrapping tablet: real 0.488s user 0.003s sys 0.007s
I20260130 02:57:59.431349 19754 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42870 (request call id 62) took 1339 ms. Trace:
I20260130 02:57:59.436869 19754 rpcz_store.cc:276] 0130 02:57:58.091686 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:57:58.091842 (+ 156us) service_pool.cc:225] Handling call
0130 02:57:59.043190 (+951348us) ts_tablet_manager.cc:1151] Deleting tablet 93809949599a449ca09b13e9b8bbdef6
0130 02:57:59.043227 (+ 37us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0130 02:57:59.346222 (+302995us) tablet_metadata.cc:641] Metadata flushed
0130 02:57:59.396038 (+ 49816us) tablet_metadata.cc:641] Metadata flushed
0130 02:57:59.431280 (+ 35242us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":946148}
I20260130 02:57:59.440466 19754 tablet_replica.cc:333] T d4f2b54930cc4f5c825562e3e0a79f4d P 371a36d219164974abdf848841528152: stopping tablet replica
I20260130 02:57:59.443207 19754 raft_consensus.cc:2243] T d4f2b54930cc4f5c825562e3e0a79f4d P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260130 02:57:59.445525 19754 raft_consensus.cc:2272] T d4f2b54930cc4f5c825562e3e0a79f4d P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260130 02:57:59.415911 18868 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.18.100.65:51021 (request call id 21) took 1511 ms (client timeout 1000 ms). Trace:
W20260130 02:57:59.420100 18871 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.18.100.66:59691 (request call id 18) took 1012 ms (client timeout 1000 ms). Trace:
I20260130 02:57:59.456504 19625 raft_consensus.cc:359] T 1bbc55bbf2254d8bb6b721d7e8f69d85 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:57:59.457149 19625 raft_consensus.cc:385] T 1bbc55bbf2254d8bb6b721d7e8f69d85 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:57:59.457396 19625 raft_consensus.cc:740] T 1bbc55bbf2254d8bb6b721d7e8f69d85 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d870164263bb44b6b780a557daea3656, State: Initialized, Role: FOLLOWER
I20260130 02:57:59.457683 19754 ts_tablet_manager.cc:1916] T d4f2b54930cc4f5c825562e3e0a79f4d P 371a36d219164974abdf848841528152: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260130 02:57:59.458035 19625 consensus_queue.cc:260] T 1bbc55bbf2254d8bb6b721d7e8f69d85 P d870164263bb44b6b780a557daea3656 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:57:59.438350 18861 catalog_manager.cc:4985] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021): tablet 93809949599a449ca09b13e9b8bbdef6 (table default.Table_1c77605822574b10a05074a093ad30a3 [id=aeba5f5a21564264a8cd975244774ab8]) successfully deleted
W20260130 02:57:59.462999 18861 catalog_manager.cc:4965] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021): delete failed for tablet d701152960b14396a6ac1bd10480d2b6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d701152960b14396a6ac1bd10480d2b6 already in progress: creating tablet
W20260130 02:57:59.463817 18861 catalog_manager.cc:4965] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021): delete failed for tablet 5b2976108584478a98fd90fe6594af66 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5b2976108584478a98fd90fe6594af66 already in progress: creating tablet
I20260130 02:57:59.465919 19758 raft_consensus.cc:2243] T d4f2b54930cc4f5c825562e3e0a79f4d P 78c2c46c7520492db35d51c244fb4f74 [term 1 LEADER]: Raft consensus shutting down.
I20260130 02:57:59.466861 19758 raft_consensus.cc:2272] T d4f2b54930cc4f5c825562e3e0a79f4d P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260130 02:57:59.478610 19758 ts_tablet_manager.cc:1916] T d4f2b54930cc4f5c825562e3e0a79f4d P 78c2c46c7520492db35d51c244fb4f74: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260130 02:57:59.494333 19807 tablet_bootstrap.cc:492] T e364f4e70e5749e083e14e7fa7416ff0 P 371a36d219164974abdf848841528152: No bootstrap required, opened a new log
I20260130 02:57:59.494844 19807 ts_tablet_manager.cc:1403] T e364f4e70e5749e083e14e7fa7416ff0 P 371a36d219164974abdf848841528152: Time spent bootstrapping tablet: real 0.184s user 0.028s sys 0.057s
I20260130 02:57:59.495677 19353 tablet_service.cc:1552] Processing DeleteTablet for tablet 01db06fcfffe4b259ac4240c0df38047 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:57:56 UTC) from {username='slave'} at 127.0.0.1:60110
I20260130 02:57:59.496279 19663 raft_consensus.cc:493] T c3fce8cc99fb451ebd532fa37390e862 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:57:59.496686 19663 raft_consensus.cc:515] T c3fce8cc99fb451ebd532fa37390e862 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:57:59.497098 19807 raft_consensus.cc:359] T e364f4e70e5749e083e14e7fa7416ff0 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:57:59.497673 19807 raft_consensus.cc:385] T e364f4e70e5749e083e14e7fa7416ff0 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:57:59.497911 19807 raft_consensus.cc:740] T e364f4e70e5749e083e14e7fa7416ff0 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 371a36d219164974abdf848841528152, State: Initialized, Role: FOLLOWER
I20260130 02:57:59.500896 19224 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c3fce8cc99fb451ebd532fa37390e862" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152" is_pre_election: true
I20260130 02:57:59.501659 19526 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c3fce8cc99fb451ebd532fa37390e862" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
I20260130 02:57:59.502290 19526 raft_consensus.cc:2468] T c3fce8cc99fb451ebd532fa37390e862 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78c2c46c7520492db35d51c244fb4f74 in term 0.
I20260130 02:57:59.503695 19807 consensus_queue.cc:260] T e364f4e70e5749e083e14e7fa7416ff0 P 371a36d219164974abdf848841528152 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:57:59.511381 19807 ts_tablet_manager.cc:1434] T e364f4e70e5749e083e14e7fa7416ff0 P 371a36d219164974abdf848841528152: Time spent starting tablet: real 0.016s user 0.004s sys 0.001s
I20260130 02:57:59.509490 19625 ts_tablet_manager.cc:1434] T 1bbc55bbf2254d8bb6b721d7e8f69d85 P d870164263bb44b6b780a557daea3656: Time spent starting tablet: real 0.085s user 0.006s sys 0.000s
I20260130 02:57:59.498430 19663 leader_election.cc:290] T c3fce8cc99fb451ebd532fa37390e862 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d870164263bb44b6b780a557daea3656 (127.18.100.67:39571), 371a36d219164974abdf848841528152 (127.18.100.65:42021)
I20260130 02:57:59.517066 19807 tablet_bootstrap.cc:492] T c3fce8cc99fb451ebd532fa37390e862 P 371a36d219164974abdf848841528152: Bootstrap starting.
W20260130 02:57:59.461014 18871 rpcz_store.cc:269] 0130 02:57:58.407834 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:57:58.409840 (+ 2006us) service_pool.cc:225] Handling call
0130 02:57:59.420021 (+1010181us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0130 02:57:58.904799 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0130 02:57:58.905169 (+ 370us) write_op.cc:432] Acquiring schema lock in shared mode
0130 02:57:58.905190 (+ 21us) write_op.cc:435] Acquired schema lock
0130 02:57:58.905203 (+ 13us) tablet.cc:660] Decoding operations
0130 02:57:58.905526 (+ 323us) write_op.cc:620] Acquiring the partition lock for write op
0130 02:57:58.905561 (+ 35us) write_op.cc:641] Partition lock acquired for write op
0130 02:57:58.905583 (+ 22us) tablet.cc:683] Acquiring locks for 2 operations
0130 02:57:58.905858 (+ 275us) tablet.cc:699] Row locks acquired
0130 02:57:58.905871 (+ 13us) write_op.cc:260] PREPARE: finished
0130 02:57:58.905975 (+ 104us) write_op.cc:270] Start()
0130 02:57:58.906054 (+ 79us) write_op.cc:276] Timestamp: P: 1769741878905933 usec, L: 0
0130 02:57:58.906076 (+ 22us) op_driver.cc:348] REPLICATION: starting
0130 02:57:58.906550 (+ 474us) log.cc:844] Serialized 913 byte log entry
0130 02:57:58.956987 (+ 50437us) op_driver.cc:464] REPLICATION: finished
0130 02:57:58.960608 (+ 3621us) write_op.cc:301] APPLY: starting
0130 02:57:58.960654 (+ 46us) tablet.cc:1365] starting BulkCheckPresence
0130 02:57:58.960742 (+ 88us) tablet.cc:1368] finished BulkCheckPresence
0130 02:57:58.960845 (+ 103us) tablet.cc:1370] starting ApplyRowOperation cycle
0130 02:57:58.961031 (+ 186us) tablet.cc:1381] finished ApplyRowOperation cycle
0130 02:57:58.961152 (+ 121us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0130 02:57:58.961192 (+ 40us) write_op.cc:312] APPLY: finished
0130 02:57:58.961345 (+ 153us) log.cc:844] Serialized 29 byte log entry
0130 02:57:58.961453 (+ 108us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:57:58.972333 (+ 10880us) write_op.cc:454] Released schema lock
0130 02:57:58.972381 (+ 48us) write_op.cc:341] FINISH: Updating metrics
Related trace 'op':
0130 02:57:59.004665 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0130 02:57:59.005020 (+ 355us) write_op.cc:432] Acquiring schema lock in shared mode
0130 02:57:59.005043 (+ 23us) write_op.cc:435] Acquired schema lock
0130 02:57:59.005055 (+ 12us) tablet.cc:660] Decoding operations
0130 02:57:59.005298 (+ 243us) write_op.cc:620] Acquiring the partition lock for write op
0130 02:57:59.005336 (+ 38us) write_op.cc:641] Partition lock acquired for write op
0130 02:57:59.005357 (+ 21us) tablet.cc:683] Acquiring locks for 1 operations
0130 02:57:59.005511 (+ 154us) tablet.cc:699] Row locks acquired
0130 02:57:59.005524 (+ 13us) write_op.cc:260] PREPARE: finished
0130 02:57:59.005624 (+ 100us) write_op.cc:270] Start()
0130 02:57:59.005699 (+ 75us) write_op.cc:276] Timestamp: P: 1769741879005570 usec, L: 0
0130 02:57:59.005713 (+ 14us) op_driver.cc:348] REPLICATION: starting
0130 02:57:59.059032 (+ 53319us) log.cc:844] Serialized 356 byte log entry
0130 02:57:59.059656 (+ 624us) spinlock_profiling.cc:243] Waited 52.8 ms on lock 0x7b5400030554. stack: 00007fb0bd3f7373 00007fb0bd3f7270 00007fb0bcf6c574 0000000000508744 00000000005086d9 00007fb0c44485a1 00007fb0c44daaa8 00007fb0c494c4d1 00007fb0c494bca8 00007fb0c4950941 00007fb0c49508f9 00007fb0c4950889 00007fb0c4950851 00007fb0c494fa7d 00000000005aa2b4 00000000005aa0e9
0130 02:57:59.293093 (+233437us) op_driver.cc:464] REPLICATION: finished
0130 02:57:59.294102 (+ 1009us) write_op.cc:301] APPLY: starting
0130 02:57:59.294156 (+ 54us) tablet.cc:1365] starting BulkCheckPresence
0130 02:57:59.294243 (+ 87us) tablet.cc:1368] finished BulkCheckPresence
0130 02:57:59.294263 (+ 20us) tablet.cc:1370] starting ApplyRowOperation cycle
0130 02:57:59.294419 (+ 156us) tablet.cc:1381] finished ApplyRowOperation cycle
0130 02:57:59.294656 (+ 237us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0130 02:57:59.294700 (+ 44us) write_op.cc:312] APPLY: finished
0130 02:57:59.294882 (+ 182us) log.cc:844] Serialized 23 byte log entry
0130 02:57:59.295023 (+ 141us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:57:59.295085 (+ 62us) write_op.cc:454] Released schema lock
0130 02:57:59.295115 (+ 30us) write_op.cc:341] FINISH: Updating metrics
0130 02:57:59.295335 (+ 220us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:57:59.295359 (+ 24us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":17,"child_traces":[["op",{"apply.queue_time_us":3511,"apply.run_cpu_time_us":1224,"apply.run_wall_time_us":11995,"num_ops":2,"prepare.queue_time_us":100,"prepare.run_cpu_time_us":2046,"prepare.run_wall_time_us":2055,"raft.queue_time_us":1587,"raft.run_cpu_time_us":19,"raft.run_wall_time_us":20,"replication_time_us":50840,"thread_start_us":100716,"threads_started":1}],["op",{"apply.queue_time_us":844,"apply.run_cpu_time_us":1207,"apply.run_wall_time_us":1207,"num_ops":1,"prepare.queue_time_us":898,"prepare.run_cpu_time_us":2779,"prepare.run_wall_time_us":55096,"replication_time_us":287284,"spinlock_wait_cycles":121458432}]]}
W20260130 02:57:59.447041 18868 rpcz_store.cc:269] 0130 02:57:57.904645 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:57:57.906085 (+ 1440us) service_pool.cc:225] Handling call
0130 02:57:59.415848 (+1509763us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0130 02:57:58.094854 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0130 02:57:58.095197 (+ 343us) write_op.cc:432] Acquiring schema lock in shared mode
0130 02:57:58.095218 (+ 21us) write_op.cc:435] Acquired schema lock
0130 02:57:58.095237 (+ 19us) tablet.cc:660] Decoding operations
0130 02:57:58.095639 (+ 402us) write_op.cc:620] Acquiring the partition lock for write op
0130 02:57:58.095671 (+ 32us) write_op.cc:641] Partition lock acquired for write op
0130 02:57:58.095693 (+ 22us) tablet.cc:683] Acquiring locks for 3 operations
0130 02:57:58.095911 (+ 218us) tablet.cc:699] Row locks acquired
0130 02:57:58.095924 (+ 13us) write_op.cc:260] PREPARE: finished
0130 02:57:58.096021 (+ 97us) write_op.cc:270] Start()
0130 02:57:58.096093 (+ 72us) write_op.cc:276] Timestamp: P: 1769741878095957 usec, L: 0
0130 02:57:58.096108 (+ 15us) op_driver.cc:348] REPLICATION: starting
0130 02:57:58.096550 (+ 442us) log.cc:844] Serialized 1292 byte log entry
0130 02:57:58.419526 (+322976us) op_driver.cc:464] REPLICATION: finished
0130 02:57:58.420394 (+ 868us) write_op.cc:301] APPLY: starting
0130 02:57:58.420450 (+ 56us) tablet.cc:1365] starting BulkCheckPresence
0130 02:57:58.420581 (+ 131us) tablet.cc:1368] finished BulkCheckPresence
0130 02:57:58.420675 (+ 94us) tablet.cc:1370] starting ApplyRowOperation cycle
0130 02:57:58.420966 (+ 291us) tablet.cc:1381] finished ApplyRowOperation cycle
0130 02:57:58.421102 (+ 136us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=3
0130 02:57:58.421143 (+ 41us) write_op.cc:312] APPLY: finished
0130 02:57:58.421340 (+ 197us) log.cc:844] Serialized 34 byte log entry
0130 02:57:58.421508 (+ 168us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:57:58.421576 (+ 68us) write_op.cc:454] Released schema lock
0130 02:57:58.421608 (+ 32us) write_op.cc:341] FINISH: Updating metrics
0130 02:57:58.421830 (+ 222us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:57:58.421855 (+ 25us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":16,"child_traces":[["op",{"apply.queue_time_us":709,"apply.run_cpu_time_us":1402,"apply.run_wall_time_us":1401,"num_ops":3,"prepare.queue_time_us":916,"prepare.run_cpu_time_us":1885,"prepare.run_wall_time_us":2915,"replication_time_us":323304}]]}
I20260130 02:57:59.526675 18872 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50284:
table { table_name: "default.Table_97c3def679ff49dcb74a992828c65cde" } modify_external_catalogs: true
I20260130 02:57:59.527211 18872 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50284:
table { table_name: "default.Table_97c3def679ff49dcb74a992828c65cde" } modify_external_catalogs: true
I20260130 02:57:59.529243 18869 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50284:
name: "default.Table_1e42d571631143088cf33d4c3aafed28"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260130 02:57:59.530747 18869 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1e42d571631143088cf33d4c3aafed28 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260130 02:57:59.533833 19807 tablet_bootstrap.cc:654] T c3fce8cc99fb451ebd532fa37390e862 P 371a36d219164974abdf848841528152: Neither blocks nor log segments found. Creating new log.
I20260130 02:57:59.534165 18875 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50284:
name: "default.Table_0ad5d205dfea4344a759522f45a55475"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260130 02:57:59.535632 18875 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0ad5d205dfea4344a759522f45a55475 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260130 02:57:59.538638 19625 tablet_bootstrap.cc:492] T 15918c90ffdc43a5a38b2950e9834188 P d870164263bb44b6b780a557daea3656: Bootstrap starting.
I20260130 02:57:59.474722 19710 ts_tablet_manager.cc:1929] T d4f2b54930cc4f5c825562e3e0a79f4d P d870164263bb44b6b780a557daea3656: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260130 02:57:59.540091 19710 log.cc:1199] T d4f2b54930cc4f5c825562e3e0a79f4d P d870164263bb44b6b780a557daea3656: Deleting WAL directory at /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-2/wal/wals/d4f2b54930cc4f5c825562e3e0a79f4d
I20260130 02:57:59.542735 18873 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50284:
name: "default.Table_3046889233c24f0a8e8058e1591d63b8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260130 02:57:59.543951 19635 raft_consensus.cc:493] T 0b1877e5f9244913bcd639f1336df7ee P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260130 02:57:59.544694 18873 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3046889233c24f0a8e8058e1591d63b8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260130 02:57:59.544431 19635 raft_consensus.cc:515] T 0b1877e5f9244913bcd639f1336df7ee P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:57:59.546286 19710 ts_tablet_manager.cc:1950] T d4f2b54930cc4f5c825562e3e0a79f4d P d870164263bb44b6b780a557daea3656: Deleting consensus metadata
I20260130 02:57:59.546303 19635 leader_election.cc:290] T 0b1877e5f9244913bcd639f1336df7ee P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d870164263bb44b6b780a557daea3656 (127.18.100.67:39571), 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767)
I20260130 02:57:59.548097 19527 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0b1877e5f9244913bcd639f1336df7ee" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
I20260130 02:57:59.548743 19527 raft_consensus.cc:2468] T 0b1877e5f9244913bcd639f1336df7ee P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 371a36d219164974abdf848841528152 in term 0.
I20260130 02:57:59.549921 19159 leader_election.cc:304] T 0b1877e5f9244913bcd639f1336df7ee P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 371a36d219164974abdf848841528152, d870164263bb44b6b780a557daea3656; no voters:
I20260130 02:57:59.550700 19635 raft_consensus.cc:2804] T 0b1877e5f9244913bcd639f1336df7ee P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260130 02:57:59.550982 19635 raft_consensus.cc:493] T 0b1877e5f9244913bcd639f1336df7ee P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260130 02:57:59.551226 19635 raft_consensus.cc:3060] T 0b1877e5f9244913bcd639f1336df7ee P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:57:59.559024 19710 tablet_replica.cc:333] T 4035b8a330844dd28af9f3244bf8eb9a P d870164263bb44b6b780a557daea3656: stopping tablet replica
I20260130 02:57:59.559732 19710 raft_consensus.cc:2243] T 4035b8a330844dd28af9f3244bf8eb9a P d870164263bb44b6b780a557daea3656 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260130 02:57:59.560216 19710 raft_consensus.cc:2272] T 4035b8a330844dd28af9f3244bf8eb9a P d870164263bb44b6b780a557daea3656 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260130 02:57:59.562139 19635 raft_consensus.cc:515] T 0b1877e5f9244913bcd639f1336df7ee P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:57:59.562937 19625 tablet_bootstrap.cc:654] T 15918c90ffdc43a5a38b2950e9834188 P d870164263bb44b6b780a557daea3656: Neither blocks nor log segments found. Creating new log.
I20260130 02:57:59.564677 19527 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0b1877e5f9244913bcd639f1336df7ee" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656"
I20260130 02:57:59.565218 19527 raft_consensus.cc:3060] T 0b1877e5f9244913bcd639f1336df7ee P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:57:59.567437 19710 ts_tablet_manager.cc:1916] T 4035b8a330844dd28af9f3244bf8eb9a P d870164263bb44b6b780a557daea3656: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260130 02:57:59.392803 18866 rpcz_store.cc:269] 0130 02:57:57.334043 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:57:57.334178 (+ 135us) service_pool.cc:225] Handling call
0130 02:57:59.392256 (+2058078us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0130 02:57:57.875165 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0130 02:57:57.875516 (+ 351us) write_op.cc:432] Acquiring schema lock in shared mode
0130 02:57:57.875538 (+ 22us) write_op.cc:435] Acquired schema lock
0130 02:57:57.875550 (+ 12us) tablet.cc:660] Decoding operations
0130 02:57:57.875784 (+ 234us) write_op.cc:620] Acquiring the partition lock for write op
0130 02:57:57.875815 (+ 31us) write_op.cc:641] Partition lock acquired for write op
0130 02:57:57.875836 (+ 21us) tablet.cc:683] Acquiring locks for 1 operations
0130 02:57:57.875968 (+ 132us) tablet.cc:699] Row locks acquired
0130 02:57:57.875981 (+ 13us) write_op.cc:260] PREPARE: finished
0130 02:57:57.876075 (+ 94us) write_op.cc:270] Start()
0130 02:57:57.876151 (+ 76us) write_op.cc:276] Timestamp: P: 1769741877876037 usec, L: 0
0130 02:57:57.876167 (+ 16us) op_driver.cc:348] REPLICATION: starting
0130 02:57:57.876595 (+ 428us) log.cc:844] Serialized 532 byte log entry
0130 02:57:57.976492 (+ 99897us) op_driver.cc:464] REPLICATION: finished
0130 02:57:57.978930 (+ 2438us) write_op.cc:301] APPLY: starting
0130 02:57:57.978986 (+ 56us) tablet.cc:1365] starting BulkCheckPresence
0130 02:57:57.979061 (+ 75us) tablet.cc:1368] finished BulkCheckPresence
0130 02:57:57.979143 (+ 82us) tablet.cc:1370] starting ApplyRowOperation cycle
0130 02:57:57.979275 (+ 132us) tablet.cc:1381] finished ApplyRowOperation cycle
0130 02:57:57.979389 (+ 114us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0130 02:57:57.979432 (+ 43us) write_op.cc:312] APPLY: finished
0130 02:57:57.979571 (+ 139us) log.cc:844] Serialized 22 byte log entry
0130 02:57:57.979716 (+ 145us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:57:57.979771 (+ 55us) write_op.cc:454] Released schema lock
0130 02:57:57.979798 (+ 27us) write_op.cc:341] FINISH: Updating metrics
0130 02:57:57.980004 (+ 206us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:57:57.980033 (+ 29us) write_op.cc:454] Released schema lock
Related trace 'op':
0130 02:57:58.127501 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0130 02:57:58.127871 (+ 370us) write_op.cc:432] Acquiring schema lock in shared mode
0130 02:57:58.127892 (+ 21us) write_op.cc:435] Acquired schema lock
0130 02:57:58.127905 (+ 13us) tablet.cc:660] Decoding operations
0130 02:57:58.128135 (+ 230us) write_op.cc:620] Acquiring the partition lock for write op
0130 02:57:58.128168 (+ 33us) write_op.cc:641] Partition lock acquired for write op
0130 02:57:58.128190 (+ 22us) tablet.cc:683] Acquiring locks for 1 operations
0130 02:57:58.128319 (+ 129us) tablet.cc:699] Row locks acquired
0130 02:57:58.128332 (+ 13us) write_op.cc:260] PREPARE: finished
0130 02:57:58.128429 (+ 97us) write_op.cc:270] Start()
0130 02:57:58.128505 (+ 76us) write_op.cc:276] Timestamp: P: 1769741878128361 usec, L: 0
0130 02:57:58.128519 (+ 14us) op_driver.cc:348] REPLICATION: starting
0130 02:57:58.128968 (+ 449us) log.cc:844] Serialized 355 byte log entry
0130 02:57:58.419968 (+291000us) op_driver.cc:464] REPLICATION: finished
0130 02:57:58.422751 (+ 2783us) write_op.cc:301] APPLY: starting
0130 02:57:58.422797 (+ 46us) tablet.cc:1365] starting BulkCheckPresence
0130 02:57:58.422882 (+ 85us) tablet.cc:1368] finished BulkCheckPresence
0130 02:57:58.422992 (+ 110us) tablet.cc:1370] starting ApplyRowOperation cycle
0130 02:57:58.423130 (+ 138us) tablet.cc:1381] finished ApplyRowOperation cycle
0130 02:57:58.423246 (+ 116us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0130 02:57:58.423286 (+ 40us) write_op.cc:312] APPLY: finished
0130 02:57:58.423490 (+ 204us) log.cc:844] Serialized 22 byte log entry
0130 02:57:58.423615 (+ 125us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:57:58.423674 (+ 59us) write_op.cc:454] Released schema lock
0130 02:57:58.423708 (+ 34us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"reported_tablets":17,"child_traces":[["op",{"apply.queue_time_us":2287,"apply.run_cpu_time_us":1049,"apply.run_wall_time_us":1049,"num_ops":1,"prepare.queue_time_us":106,"prepare.run_cpu_time_us":1680,"prepare.run_wall_time_us":2719,"raft.queue_time_us":73,"raft.run_cpu_time_us":15,"raft.run_wall_time_us":16,"replication_time_us":100231}],["op",{"apply.queue_time_us":2639,"apply.run_cpu_time_us":1149,"apply.run_wall_time_us":1149,"num_ops":1,"prepare.queue_time_us":98,"prepare.run_cpu_time_us":1657,"prepare.run_wall_time_us":2501,"replication_time_us":291365,"thread_start_us":16658,"threads_started":1}]]}
I20260130 02:57:59.572244 19527 raft_consensus.cc:2468] T 0b1877e5f9244913bcd639f1336df7ee P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 371a36d219164974abdf848841528152 in term 1.
I20260130 02:57:59.573418 19159 leader_election.cc:304] T 0b1877e5f9244913bcd639f1336df7ee P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 371a36d219164974abdf848841528152, d870164263bb44b6b780a557daea3656; no voters:
I20260130 02:57:59.574189 19861 raft_consensus.cc:2804] T 0b1877e5f9244913bcd639f1336df7ee P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Leader election won for term 1
I20260130 02:57:59.574667 19861 raft_consensus.cc:697] T 0b1877e5f9244913bcd639f1336df7ee P 371a36d219164974abdf848841528152 [term 1 LEADER]: Becoming Leader. State: Replica: 371a36d219164974abdf848841528152, State: Running, Role: LEADER
I20260130 02:57:59.575383 19861 consensus_queue.cc:237] T 0b1877e5f9244913bcd639f1336df7ee P 371a36d219164974abdf848841528152 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:57:59.576452 18866 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50284:
table { table_name: "default.Table_235e240c644b42baaf12b08d1b983e8a" } new_table_name: "default.Table_9e942a1ce3944e9aa028e75656ce066d" modify_external_catalogs: true
I20260130 02:57:59.583691 19635 leader_election.cc:290] T 0b1877e5f9244913bcd639f1336df7ee P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 election: Requested vote from peers d870164263bb44b6b780a557daea3656 (127.18.100.67:39571), 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767)
I20260130 02:57:59.591661 19807 tablet_bootstrap.cc:492] T c3fce8cc99fb451ebd532fa37390e862 P 371a36d219164974abdf848841528152: No bootstrap required, opened a new log
I20260130 02:57:59.593519 19807 ts_tablet_manager.cc:1403] T c3fce8cc99fb451ebd532fa37390e862 P 371a36d219164974abdf848841528152: Time spent bootstrapping tablet: real 0.077s user 0.006s sys 0.005s
I20260130 02:57:59.592085 19754 ts_tablet_manager.cc:1929] T d4f2b54930cc4f5c825562e3e0a79f4d P 371a36d219164974abdf848841528152: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260130 02:57:59.596796 19754 log.cc:1199] T d4f2b54930cc4f5c825562e3e0a79f4d P 371a36d219164974abdf848841528152: Deleting WAL directory at /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-0/wal/wals/d4f2b54930cc4f5c825562e3e0a79f4d
I20260130 02:57:59.599171 19754 ts_tablet_manager.cc:1950] T d4f2b54930cc4f5c825562e3e0a79f4d P 371a36d219164974abdf848841528152: Deleting consensus metadata
I20260130 02:57:59.596457 19807 raft_consensus.cc:359] T c3fce8cc99fb451ebd532fa37390e862 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:57:59.600169 19807 raft_consensus.cc:385] T c3fce8cc99fb451ebd532fa37390e862 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:57:59.600417 19807 raft_consensus.cc:740] T c3fce8cc99fb451ebd532fa37390e862 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 371a36d219164974abdf848841528152, State: Initialized, Role: FOLLOWER
W20260130 02:57:59.600950 18860 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.100.124:34629, user_credentials={real_user=slave}} blocked reactor thread for 80413.9us
I20260130 02:57:59.601229 19807 consensus_queue.cc:260] T c3fce8cc99fb451ebd532fa37390e862 P 371a36d219164974abdf848841528152 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:57:59.605228 19807 ts_tablet_manager.cc:1434] T c3fce8cc99fb451ebd532fa37390e862 P 371a36d219164974abdf848841528152: Time spent starting tablet: real 0.011s user 0.001s sys 0.003s
I20260130 02:57:59.605752 18861 catalog_manager.cc:4985] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021): tablet d4f2b54930cc4f5c825562e3e0a79f4d (table default.Table_50c3bb8b3ede494ba9b9f826a2adba72 [id=17150cb73e374ff9a1e92c43e0c3f2c1]) successfully deleted
I20260130 02:57:59.607030 19807 tablet_bootstrap.cc:492] T d701152960b14396a6ac1bd10480d2b6 P 371a36d219164974abdf848841528152: Bootstrap starting.
I20260130 02:57:59.607666 19754 tablet_replica.cc:333] T 4035b8a330844dd28af9f3244bf8eb9a P 371a36d219164974abdf848841528152: stopping tablet replica
I20260130 02:57:59.608314 19754 raft_consensus.cc:2243] T 4035b8a330844dd28af9f3244bf8eb9a P 371a36d219164974abdf848841528152 [term 2 LEADER]: Raft consensus shutting down.
I20260130 02:57:59.613364 19754 raft_consensus.cc:2272] T 4035b8a330844dd28af9f3244bf8eb9a P 371a36d219164974abdf848841528152 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260130 02:57:59.614751 19313 leader_election.cc:304] T c3fce8cc99fb451ebd532fa37390e862 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78c2c46c7520492db35d51c244fb4f74, d870164263bb44b6b780a557daea3656; no voters:
W20260130 02:57:59.616166 19313 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.100.67:39571, user_credentials={real_user=slave}} blocked reactor thread for 112184us
I20260130 02:57:59.619616 19861 raft_consensus.cc:493] T e364f4e70e5749e083e14e7fa7416ff0 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:57:59.620013 19861 raft_consensus.cc:515] T e364f4e70e5749e083e14e7fa7416ff0 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
W20260130 02:57:59.620299 19314 leader_election.cc:343] T c3fce8cc99fb451ebd532fa37390e862 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:57:59.621788 19861 leader_election.cc:290] T e364f4e70e5749e083e14e7fa7416ff0 P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d870164263bb44b6b780a557daea3656 (127.18.100.67:39571), 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767)
I20260130 02:57:59.621487 18871 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50284:
name: "default.Table_081113b62fb743189a6e04bf85dcc5fa"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260130 02:57:59.623473 19527 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e364f4e70e5749e083e14e7fa7416ff0" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
I20260130 02:57:59.624043 19527 raft_consensus.cc:2468] T e364f4e70e5749e083e14e7fa7416ff0 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 371a36d219164974abdf848841528152 in term 0.
I20260130 02:57:59.624683 19840 tablet_bootstrap.cc:654] T e364f4e70e5749e083e14e7fa7416ff0 P 78c2c46c7520492db35d51c244fb4f74: Neither blocks nor log segments found. Creating new log.
I20260130 02:57:59.625203 19159 leader_election.cc:304] T e364f4e70e5749e083e14e7fa7416ff0 P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 371a36d219164974abdf848841528152, d870164263bb44b6b780a557daea3656; no voters:
I20260130 02:57:59.625941 19861 raft_consensus.cc:2804] T e364f4e70e5749e083e14e7fa7416ff0 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260130 02:57:59.626237 19861 raft_consensus.cc:493] T e364f4e70e5749e083e14e7fa7416ff0 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260130 02:57:59.626536 19861 raft_consensus.cc:3060] T e364f4e70e5749e083e14e7fa7416ff0 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:57:59.626626 19663 raft_consensus.cc:2804] T c3fce8cc99fb451ebd532fa37390e862 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260130 02:57:59.627210 19663 raft_consensus.cc:493] T c3fce8cc99fb451ebd532fa37390e862 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260130 02:57:59.627630 19663 raft_consensus.cc:3060] T c3fce8cc99fb451ebd532fa37390e862 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:57:59.627594 19635 consensus_queue.cc:1048] T 073b5941850c4140b1751956dd7cf5ef P 371a36d219164974abdf848841528152 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260130 02:57:59.627501 19378 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0b1877e5f9244913bcd639f1336df7ee" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74" is_pre_election: true
I20260130 02:57:59.628619 19378 raft_consensus.cc:2468] T 0b1877e5f9244913bcd639f1336df7ee P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 371a36d219164974abdf848841528152 in term 0.
I20260130 02:57:59.627157 19379 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0b1877e5f9244913bcd639f1336df7ee" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74"
I20260130 02:57:59.635763 19379 raft_consensus.cc:3060] T 0b1877e5f9244913bcd639f1336df7ee P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:57:59.635821 19378 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e364f4e70e5749e083e14e7fa7416ff0" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74" is_pre_election: true
W20260130 02:57:59.636180 19314 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.100.65:42021, user_credentials={real_user=slave}} blocked reactor thread for 121318us
W20260130 02:57:59.638377 19159 leader_election.cc:343] T e364f4e70e5749e083e14e7fa7416ff0 P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:57:59.612536 19807 tablet_bootstrap.cc:654] T d701152960b14396a6ac1bd10480d2b6 P 371a36d219164974abdf848841528152: Neither blocks nor log segments found. Creating new log.
I20260130 02:57:59.643627 18860 catalog_manager.cc:4985] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): tablet d4f2b54930cc4f5c825562e3e0a79f4d (table default.Table_50c3bb8b3ede494ba9b9f826a2adba72 [id=17150cb73e374ff9a1e92c43e0c3f2c1]) successfully deleted
I20260130 02:57:59.647565 19754 ts_tablet_manager.cc:1916] T 4035b8a330844dd28af9f3244bf8eb9a P 371a36d219164974abdf848841528152: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260130 02:57:59.648386 19861 raft_consensus.cc:515] T e364f4e70e5749e083e14e7fa7416ff0 P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:57:59.651685 19710 ts_tablet_manager.cc:1929] T 4035b8a330844dd28af9f3244bf8eb9a P d870164263bb44b6b780a557daea3656: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 2.2
I20260130 02:57:59.652082 19710 log.cc:1199] T 4035b8a330844dd28af9f3244bf8eb9a P d870164263bb44b6b780a557daea3656: Deleting WAL directory at /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-2/wal/wals/4035b8a330844dd28af9f3244bf8eb9a
I20260130 02:57:59.652427 19527 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e364f4e70e5749e083e14e7fa7416ff0" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656"
I20260130 02:57:59.652767 19710 ts_tablet_manager.cc:1950] T 4035b8a330844dd28af9f3244bf8eb9a P d870164263bb44b6b780a557daea3656: Deleting consensus metadata
I20260130 02:57:59.652944 19527 raft_consensus.cc:3060] T e364f4e70e5749e083e14e7fa7416ff0 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:57:59.659447 19527 raft_consensus.cc:2468] T e364f4e70e5749e083e14e7fa7416ff0 P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 371a36d219164974abdf848841528152 in term 1.
I20260130 02:57:59.660732 19159 leader_election.cc:304] T e364f4e70e5749e083e14e7fa7416ff0 P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 371a36d219164974abdf848841528152, d870164263bb44b6b780a557daea3656; no voters:
I20260130 02:57:59.661418 19635 raft_consensus.cc:2804] T e364f4e70e5749e083e14e7fa7416ff0 P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Leader election won for term 1
I20260130 02:57:59.661818 19635 raft_consensus.cc:697] T e364f4e70e5749e083e14e7fa7416ff0 P 371a36d219164974abdf848841528152 [term 1 LEADER]: Becoming Leader. State: Replica: 371a36d219164974abdf848841528152, State: Running, Role: LEADER
I20260130 02:57:59.661674 19898 tablet.cc:1722] T ceda64471a92476f9ba183d2beab30d1 P 78c2c46c7520492db35d51c244fb4f74: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20260130 02:57:59.662483 19635 consensus_queue.cc:237] T e364f4e70e5749e083e14e7fa7416ff0 P 371a36d219164974abdf848841528152 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:57:59.667243 18860 catalog_manager.cc:4985] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): tablet 4035b8a330844dd28af9f3244bf8eb9a (table default.Table_e7b6f8ca4ed143888d55e87c1b157efd [id=53806f69131d48fdb3328e9fbf8209a2]) successfully deleted
W20260130 02:57:59.667887 18860 catalog_manager.cc:4952] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): delete failed for tablet 3b8cdac4bf4b46efa26236fc930eb9a6 because the tablet was not found. No further retry: Not found: Tablet not found: 3b8cdac4bf4b46efa26236fc930eb9a6
I20260130 02:57:59.670121 19880 tablet.cc:1722] T ceda64471a92476f9ba183d2beab30d1 P d870164263bb44b6b780a557daea3656: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20260130 02:57:59.673203 18869 master_service.cc:954] ReplaceTablet: received request to replace tablet 878f9346d3044d8d8b15c61f86869212 from {username='slave'} at 127.0.0.1:50284
W20260130 02:57:59.679122 18871 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_081113b62fb743189a6e04bf85dcc5fa in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260130 02:57:59.683234 19861 leader_election.cc:290] T e364f4e70e5749e083e14e7fa7416ff0 P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 election: Requested vote from peers d870164263bb44b6b780a557daea3656 (127.18.100.67:39571), 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767)
I20260130 02:57:59.692255 19807 tablet_bootstrap.cc:492] T d701152960b14396a6ac1bd10480d2b6 P 371a36d219164974abdf848841528152: No bootstrap required, opened a new log
I20260130 02:57:59.694850 18875 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50284:
table { table_name: "default.Table_b87dd1ee48f143538e487968bbea94bb" } new_table_name: "default.Table_333d2efbf068485a871f566d50e563ce" modify_external_catalogs: true
I20260130 02:57:59.699790 19807 ts_tablet_manager.cc:1403] T d701152960b14396a6ac1bd10480d2b6 P 371a36d219164974abdf848841528152: Time spent bootstrapping tablet: real 0.093s user 0.013s sys 0.003s
I20260130 02:57:59.700441 19861 consensus_queue.cc:1048] T 073b5941850c4140b1751956dd7cf5ef P 371a36d219164974abdf848841528152 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260130 02:57:59.701915 19807 raft_consensus.cc:359] T d701152960b14396a6ac1bd10480d2b6 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:57:59.706630 19625 tablet_bootstrap.cc:492] T 15918c90ffdc43a5a38b2950e9834188 P d870164263bb44b6b780a557daea3656: No bootstrap required, opened a new log
I20260130 02:57:59.707011 19625 ts_tablet_manager.cc:1403] T 15918c90ffdc43a5a38b2950e9834188 P d870164263bb44b6b780a557daea3656: Time spent bootstrapping tablet: real 0.169s user 0.008s sys 0.016s
I20260130 02:57:59.709209 19625 raft_consensus.cc:359] T 15918c90ffdc43a5a38b2950e9834188 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:57:59.709916 19625 raft_consensus.cc:385] T 15918c90ffdc43a5a38b2950e9834188 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:57:59.710224 19625 raft_consensus.cc:740] T 15918c90ffdc43a5a38b2950e9834188 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d870164263bb44b6b780a557daea3656, State: Initialized, Role: FOLLOWER
I20260130 02:57:59.711018 19625 consensus_queue.cc:260] T 15918c90ffdc43a5a38b2950e9834188 P d870164263bb44b6b780a557daea3656 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:57:59.702521 19807 raft_consensus.cc:385] T d701152960b14396a6ac1bd10480d2b6 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:57:59.712743 19807 raft_consensus.cc:740] T d701152960b14396a6ac1bd10480d2b6 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 371a36d219164974abdf848841528152, State: Initialized, Role: FOLLOWER
I20260130 02:57:59.715109 19807 consensus_queue.cc:260] T d701152960b14396a6ac1bd10480d2b6 P 371a36d219164974abdf848841528152 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:57:59.727991 19625 ts_tablet_manager.cc:1434] T 15918c90ffdc43a5a38b2950e9834188 P d870164263bb44b6b780a557daea3656: Time spent starting tablet: real 0.021s user 0.006s sys 0.000s
I20260130 02:57:59.728425 19378 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e364f4e70e5749e083e14e7fa7416ff0" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74"
I20260130 02:57:59.728885 19625 tablet_bootstrap.cc:492] T e026dee22493424692d00f14848ab57c P d870164263bb44b6b780a557daea3656: Bootstrap starting.
I20260130 02:57:59.730134 19861 consensus_queue.cc:1048] T 2112b0402de5464cb1caf2c2abb67d78 P 371a36d219164974abdf848841528152 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260130 02:57:59.733368 19379 raft_consensus.cc:2468] T 0b1877e5f9244913bcd639f1336df7ee P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 371a36d219164974abdf848841528152 in term 1.
I20260130 02:57:59.735638 19840 tablet_bootstrap.cc:492] T e364f4e70e5749e083e14e7fa7416ff0 P 78c2c46c7520492db35d51c244fb4f74: No bootstrap required, opened a new log
W20260130 02:57:59.736042 19159 leader_election.cc:343] T e364f4e70e5749e083e14e7fa7416ff0 P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:57:59.736107 19840 ts_tablet_manager.cc:1403] T e364f4e70e5749e083e14e7fa7416ff0 P 78c2c46c7520492db35d51c244fb4f74: Time spent bootstrapping tablet: real 0.413s user 0.011s sys 0.012s
I20260130 02:57:59.742197 19813 tablet.cc:1722] T ceda64471a92476f9ba183d2beab30d1 P 371a36d219164974abdf848841528152: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20260130 02:57:59.755659 19663 raft_consensus.cc:515] T c3fce8cc99fb451ebd532fa37390e862 P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:57:59.757881 19663 leader_election.cc:290] T c3fce8cc99fb451ebd532fa37390e862 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 election: Requested vote from peers d870164263bb44b6b780a557daea3656 (127.18.100.67:39571), 371a36d219164974abdf848841528152 (127.18.100.65:42021)
I20260130 02:57:59.759060 19221 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c3fce8cc99fb451ebd532fa37390e862" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152"
I20260130 02:57:59.759378 19525 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c3fce8cc99fb451ebd532fa37390e862" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656"
I20260130 02:57:59.759634 19221 raft_consensus.cc:3060] T c3fce8cc99fb451ebd532fa37390e862 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:57:59.759908 19525 raft_consensus.cc:3060] T c3fce8cc99fb451ebd532fa37390e862 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:57:59.767289 19221 raft_consensus.cc:2468] T c3fce8cc99fb451ebd532fa37390e862 P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78c2c46c7520492db35d51c244fb4f74 in term 1.
I20260130 02:57:59.738523 19840 raft_consensus.cc:359] T e364f4e70e5749e083e14e7fa7416ff0 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:57:59.768437 19840 raft_consensus.cc:385] T e364f4e70e5749e083e14e7fa7416ff0 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:57:59.768494 19314 leader_election.cc:304] T c3fce8cc99fb451ebd532fa37390e862 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 371a36d219164974abdf848841528152, 78c2c46c7520492db35d51c244fb4f74; no voters:
I20260130 02:57:59.768803 19840 raft_consensus.cc:740] T e364f4e70e5749e083e14e7fa7416ff0 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78c2c46c7520492db35d51c244fb4f74, State: Initialized, Role: FOLLOWER
I20260130 02:57:59.769462 19840 consensus_queue.cc:260] T e364f4e70e5749e083e14e7fa7416ff0 P 78c2c46c7520492db35d51c244fb4f74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:57:59.769907 19663 raft_consensus.cc:2804] T c3fce8cc99fb451ebd532fa37390e862 P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Leader election won for term 1
I20260130 02:57:59.770720 19663 raft_consensus.cc:697] T c3fce8cc99fb451ebd532fa37390e862 P 78c2c46c7520492db35d51c244fb4f74 [term 1 LEADER]: Becoming Leader. State: Replica: 78c2c46c7520492db35d51c244fb4f74, State: Running, Role: LEADER
I20260130 02:57:59.771703 19663 consensus_queue.cc:237] T c3fce8cc99fb451ebd532fa37390e862 P 78c2c46c7520492db35d51c244fb4f74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:57:59.779922 19807 ts_tablet_manager.cc:1434] T d701152960b14396a6ac1bd10480d2b6 P 371a36d219164974abdf848841528152: Time spent starting tablet: real 0.080s user 0.005s sys 0.001s
I20260130 02:57:59.772264 19840 ts_tablet_manager.cc:1434] T e364f4e70e5749e083e14e7fa7416ff0 P 78c2c46c7520492db35d51c244fb4f74: Time spent starting tablet: real 0.036s user 0.006s sys 0.000s
I20260130 02:57:59.780730 19525 raft_consensus.cc:2468] T c3fce8cc99fb451ebd532fa37390e862 P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78c2c46c7520492db35d51c244fb4f74 in term 1.
I20260130 02:57:59.781558 19840 tablet_bootstrap.cc:492] T d701152960b14396a6ac1bd10480d2b6 P 78c2c46c7520492db35d51c244fb4f74: Bootstrap starting.
I20260130 02:57:59.793381 19840 tablet_bootstrap.cc:654] T d701152960b14396a6ac1bd10480d2b6 P 78c2c46c7520492db35d51c244fb4f74: Neither blocks nor log segments found. Creating new log.
I20260130 02:57:59.801450 19807 tablet_bootstrap.cc:492] T 15918c90ffdc43a5a38b2950e9834188 P 371a36d219164974abdf848841528152: Bootstrap starting.
I20260130 02:57:59.803251 19754 ts_tablet_manager.cc:1929] T 4035b8a330844dd28af9f3244bf8eb9a P 371a36d219164974abdf848841528152: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 2.2
I20260130 02:57:59.804535 19754 log.cc:1199] T 4035b8a330844dd28af9f3244bf8eb9a P 371a36d219164974abdf848841528152: Deleting WAL directory at /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-0/wal/wals/4035b8a330844dd28af9f3244bf8eb9a
I20260130 02:57:59.806108 19754 ts_tablet_manager.cc:1950] T 4035b8a330844dd28af9f3244bf8eb9a P 371a36d219164974abdf848841528152: Deleting consensus metadata
W20260130 02:57:59.812212 18860 catalog_manager.cc:4952] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): delete failed for tablet 01db06fcfffe4b259ac4240c0df38047 because the tablet was not found. No further retry: Not found: Tablet not found: 01db06fcfffe4b259ac4240c0df38047
W20260130 02:57:59.813287 18860 catalog_manager.cc:4952] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): delete failed for tablet 93809949599a449ca09b13e9b8bbdef6 because the tablet was not found. No further retry: Not found: Tablet not found: 93809949599a449ca09b13e9b8bbdef6
W20260130 02:57:59.814006 18860 catalog_manager.cc:4952] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): delete failed for tablet 30d8136b5ea742bca20ee46b5634d04e because the tablet was not found. No further retry: Not found: Tablet not found: 30d8136b5ea742bca20ee46b5634d04e
I20260130 02:57:59.815173 19880 tablet.cc:1722] T 37470ce4356f4542b621e21698551209 P d870164263bb44b6b780a557daea3656: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260130 02:57:59.823397 19742 raft_consensus.cc:493] T 15918c90ffdc43a5a38b2950e9834188 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:57:59.823844 19742 raft_consensus.cc:515] T 15918c90ffdc43a5a38b2950e9834188 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:57:59.781514 19758 ts_tablet_manager.cc:1929] T d4f2b54930cc4f5c825562e3e0a79f4d P 78c2c46c7520492db35d51c244fb4f74: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260130 02:57:59.825747 19742 leader_election.cc:290] T 15918c90ffdc43a5a38b2950e9834188 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767)
I20260130 02:57:59.825858 19758 log.cc:1199] T d4f2b54930cc4f5c825562e3e0a79f4d P 78c2c46c7520492db35d51c244fb4f74: Deleting WAL directory at /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-1/wal/wals/d4f2b54930cc4f5c825562e3e0a79f4d
I20260130 02:57:59.826975 19758 ts_tablet_manager.cc:1950] T d4f2b54930cc4f5c825562e3e0a79f4d P 78c2c46c7520492db35d51c244fb4f74: Deleting consensus metadata
I20260130 02:57:59.822992 19898 tablet.cc:1722] T 37470ce4356f4542b621e21698551209 P 78c2c46c7520492db35d51c244fb4f74: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260130 02:57:59.830955 19758 tablet_replica.cc:333] T 4035b8a330844dd28af9f3244bf8eb9a P 78c2c46c7520492db35d51c244fb4f74: stopping tablet replica
I20260130 02:57:59.831732 19758 raft_consensus.cc:2243] T 4035b8a330844dd28af9f3244bf8eb9a P 78c2c46c7520492db35d51c244fb4f74 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260130 02:57:59.832300 19758 raft_consensus.cc:2272] T 4035b8a330844dd28af9f3244bf8eb9a P 78c2c46c7520492db35d51c244fb4f74 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260130 02:57:59.815286 19813 tablet.cc:1722] T 37470ce4356f4542b621e21698551209 P 371a36d219164974abdf848841528152: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260130 02:57:59.835119 19221 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "15918c90ffdc43a5a38b2950e9834188" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152" is_pre_election: true
W20260130 02:57:59.837322 19463 leader_election.cc:343] T 15918c90ffdc43a5a38b2950e9834188 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:57:59.837854 19758 ts_tablet_manager.cc:1916] T 4035b8a330844dd28af9f3244bf8eb9a P 78c2c46c7520492db35d51c244fb4f74: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260130 02:57:59.844444 19807 tablet_bootstrap.cc:654] T 15918c90ffdc43a5a38b2950e9834188 P 371a36d219164974abdf848841528152: Neither blocks nor log segments found. Creating new log.
I20260130 02:57:59.844282 19861 consensus_queue.cc:1048] T 2112b0402de5464cb1caf2c2abb67d78 P 371a36d219164974abdf848841528152 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260130 02:57:59.855142 19377 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "15918c90ffdc43a5a38b2950e9834188" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74" is_pre_election: true
W20260130 02:57:59.856478 19462 leader_election.cc:343] T 15918c90ffdc43a5a38b2950e9834188 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:57:59.856886 19840 tablet_bootstrap.cc:492] T d701152960b14396a6ac1bd10480d2b6 P 78c2c46c7520492db35d51c244fb4f74: No bootstrap required, opened a new log
I20260130 02:57:59.856878 19462 leader_election.cc:304] T 15918c90ffdc43a5a38b2950e9834188 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d870164263bb44b6b780a557daea3656; no voters: 371a36d219164974abdf848841528152, 78c2c46c7520492db35d51c244fb4f74
I20260130 02:57:59.857234 19861 raft_consensus.cc:493] T d701152960b14396a6ac1bd10480d2b6 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:57:59.857499 19742 raft_consensus.cc:2749] T 15918c90ffdc43a5a38b2950e9834188 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260130 02:57:59.857633 19861 raft_consensus.cc:515] T d701152960b14396a6ac1bd10480d2b6 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:57:59.859401 19861 leader_election.cc:290] T d701152960b14396a6ac1bd10480d2b6 P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767), d870164263bb44b6b780a557daea3656 (127.18.100.67:39571)
I20260130 02:57:59.860139 19840 ts_tablet_manager.cc:1403] T d701152960b14396a6ac1bd10480d2b6 P 78c2c46c7520492db35d51c244fb4f74: Time spent bootstrapping tablet: real 0.079s user 0.000s sys 0.011s
I20260130 02:57:59.860095 19377 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d701152960b14396a6ac1bd10480d2b6" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74" is_pre_election: true
W20260130 02:57:59.861282 18860 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet bd43c5d148514836869a8ac648c8ad11 (table default.Table_a9476edf8a5445c883f8d6ce786554a2 [id=7e29f588b41b491e8a9d1ad61e50b033], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet bd43c5d148514836869a8ac648c8ad11 consensus state has no leader
I20260130 02:57:59.861609 19527 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d701152960b14396a6ac1bd10480d2b6" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
I20260130 02:57:59.861805 18860 catalog_manager.cc:4985] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767): tablet d4f2b54930cc4f5c825562e3e0a79f4d (table default.Table_50c3bb8b3ede494ba9b9f826a2adba72 [id=17150cb73e374ff9a1e92c43e0c3f2c1]) successfully deleted
I20260130 02:57:59.862740 19840 raft_consensus.cc:359] T d701152960b14396a6ac1bd10480d2b6 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:57:59.863320 19840 raft_consensus.cc:385] T d701152960b14396a6ac1bd10480d2b6 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20260130 02:57:59.863364 19159 leader_election.cc:343] T d701152960b14396a6ac1bd10480d2b6 P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:57:59.863574 19840 raft_consensus.cc:740] T d701152960b14396a6ac1bd10480d2b6 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78c2c46c7520492db35d51c244fb4f74, State: Initialized, Role: FOLLOWER
W20260130 02:57:59.864248 19159 leader_election.cc:343] T d701152960b14396a6ac1bd10480d2b6 P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:57:59.864650 19159 leader_election.cc:304] T d701152960b14396a6ac1bd10480d2b6 P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 371a36d219164974abdf848841528152; no voters: 78c2c46c7520492db35d51c244fb4f74, d870164263bb44b6b780a557daea3656
I20260130 02:57:59.865293 19861 raft_consensus.cc:2749] T d701152960b14396a6ac1bd10480d2b6 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260130 02:57:59.866165 19840 consensus_queue.cc:260] T d701152960b14396a6ac1bd10480d2b6 P 78c2c46c7520492db35d51c244fb4f74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:57:59.873157 18861 catalog_manager.cc:4985] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021): tablet 4035b8a330844dd28af9f3244bf8eb9a (table default.Table_e7b6f8ca4ed143888d55e87c1b157efd [id=53806f69131d48fdb3328e9fbf8209a2]) successfully deleted
I20260130 02:57:59.873963 19840 ts_tablet_manager.cc:1434] T d701152960b14396a6ac1bd10480d2b6 P 78c2c46c7520492db35d51c244fb4f74: Time spent starting tablet: real 0.013s user 0.005s sys 0.000s
W20260130 02:57:59.874055 18861 catalog_manager.cc:4952] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021): delete failed for tablet 30d8136b5ea742bca20ee46b5634d04e because the tablet was not found. No further retry: Not found: Tablet not found: 30d8136b5ea742bca20ee46b5634d04e
W20260130 02:57:59.874740 18861 catalog_manager.cc:4952] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021): delete failed for tablet 3b8cdac4bf4b46efa26236fc930eb9a6 because the tablet was not found. No further retry: Not found: Tablet not found: 3b8cdac4bf4b46efa26236fc930eb9a6
I20260130 02:57:59.875195 19840 tablet_bootstrap.cc:492] T e026dee22493424692d00f14848ab57c P 78c2c46c7520492db35d51c244fb4f74: Bootstrap starting.
W20260130 02:57:59.875381 18861 catalog_manager.cc:4952] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021): delete failed for tablet 79e28c3a566c4521ac2453bbaa3567ed because the tablet was not found. No further retry: Not found: Tablet not found: 79e28c3a566c4521ac2453bbaa3567ed
W20260130 02:57:59.876019 18861 catalog_manager.cc:4952] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021): delete failed for tablet 01db06fcfffe4b259ac4240c0df38047 because the tablet was not found. No further retry: Not found: Tablet not found: 01db06fcfffe4b259ac4240c0df38047
I20260130 02:57:59.885223 19758 ts_tablet_manager.cc:1929] T 4035b8a330844dd28af9f3244bf8eb9a P 78c2c46c7520492db35d51c244fb4f74: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 2.2
I20260130 02:57:59.894454 19758 log.cc:1199] T 4035b8a330844dd28af9f3244bf8eb9a P 78c2c46c7520492db35d51c244fb4f74: Deleting WAL directory at /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-1/wal/wals/4035b8a330844dd28af9f3244bf8eb9a
I20260130 02:57:59.895375 19758 ts_tablet_manager.cc:1950] T 4035b8a330844dd28af9f3244bf8eb9a P 78c2c46c7520492db35d51c244fb4f74: Deleting consensus metadata
I20260130 02:57:59.894105 19840 tablet_bootstrap.cc:654] T e026dee22493424692d00f14848ab57c P 78c2c46c7520492db35d51c244fb4f74: Neither blocks nor log segments found. Creating new log.
I20260130 02:57:59.897936 18860 catalog_manager.cc:4985] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767): tablet 4035b8a330844dd28af9f3244bf8eb9a (table default.Table_e7b6f8ca4ed143888d55e87c1b157efd [id=53806f69131d48fdb3328e9fbf8209a2]) successfully deleted
W20260130 02:57:59.914908 18860 catalog_manager.cc:4952] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767): delete failed for tablet 93809949599a449ca09b13e9b8bbdef6 because the tablet was not found. No further retry: Not found: Tablet not found: 93809949599a449ca09b13e9b8bbdef6
W20260130 02:57:59.915869 18860 catalog_manager.cc:4952] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767): delete failed for tablet 3b8cdac4bf4b46efa26236fc930eb9a6 because the tablet was not found. No further retry: Not found: Tablet not found: 3b8cdac4bf4b46efa26236fc930eb9a6
I20260130 02:57:59.928670 19840 tablet_bootstrap.cc:492] T e026dee22493424692d00f14848ab57c P 78c2c46c7520492db35d51c244fb4f74: No bootstrap required, opened a new log
I20260130 02:57:59.929080 19840 ts_tablet_manager.cc:1403] T e026dee22493424692d00f14848ab57c P 78c2c46c7520492db35d51c244fb4f74: Time spent bootstrapping tablet: real 0.054s user 0.012s sys 0.000s
I20260130 02:57:59.931352 19840 raft_consensus.cc:359] T e026dee22493424692d00f14848ab57c P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:57:59.931948 19840 raft_consensus.cc:385] T e026dee22493424692d00f14848ab57c P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:57:59.932163 19840 raft_consensus.cc:740] T e026dee22493424692d00f14848ab57c P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78c2c46c7520492db35d51c244fb4f74, State: Initialized, Role: FOLLOWER
I20260130 02:57:59.933743 19840 consensus_queue.cc:260] T e026dee22493424692d00f14848ab57c P 78c2c46c7520492db35d51c244fb4f74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:57:59.936228 19840 ts_tablet_manager.cc:1434] T e026dee22493424692d00f14848ab57c P 78c2c46c7520492db35d51c244fb4f74: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260130 02:57:59.937103 19840 tablet_bootstrap.cc:492] T 15918c90ffdc43a5a38b2950e9834188 P 78c2c46c7520492db35d51c244fb4f74: Bootstrap starting.
W20260130 02:57:59.916461 18860 catalog_manager.cc:4952] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767): delete failed for tablet 30d8136b5ea742bca20ee46b5634d04e because the tablet was not found. No further retry: Not found: Tablet not found: 30d8136b5ea742bca20ee46b5634d04e
I20260130 02:57:59.948994 19840 tablet_bootstrap.cc:654] T 15918c90ffdc43a5a38b2950e9834188 P 78c2c46c7520492db35d51c244fb4f74: Neither blocks nor log segments found. Creating new log.
W20260130 02:57:59.949602 18860 catalog_manager.cc:4952] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767): delete failed for tablet 01db06fcfffe4b259ac4240c0df38047 because the tablet was not found. No further retry: Not found: Tablet not found: 01db06fcfffe4b259ac4240c0df38047
I20260130 02:57:59.952139 19807 tablet_bootstrap.cc:492] T 15918c90ffdc43a5a38b2950e9834188 P 371a36d219164974abdf848841528152: No bootstrap required, opened a new log
I20260130 02:57:59.954670 19807 ts_tablet_manager.cc:1403] T 15918c90ffdc43a5a38b2950e9834188 P 371a36d219164974abdf848841528152: Time spent bootstrapping tablet: real 0.153s user 0.020s sys 0.022s
I20260130 02:57:59.968163 19840 tablet_bootstrap.cc:492] T 15918c90ffdc43a5a38b2950e9834188 P 78c2c46c7520492db35d51c244fb4f74: No bootstrap required, opened a new log
I20260130 02:57:59.968650 19840 ts_tablet_manager.cc:1403] T 15918c90ffdc43a5a38b2950e9834188 P 78c2c46c7520492db35d51c244fb4f74: Time spent bootstrapping tablet: real 0.032s user 0.008s sys 0.003s
I20260130 02:57:59.969949 19807 raft_consensus.cc:359] T 15918c90ffdc43a5a38b2950e9834188 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:57:59.972779 19840 raft_consensus.cc:359] T 15918c90ffdc43a5a38b2950e9834188 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:57:59.974051 18868 catalog_manager.cc:5746] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571) does not have the latest schema for tablet ceda64471a92476f9ba183d2beab30d1 (table default.Table_be9aa687d9744ac6a2aa335772c6cc9a [id=4f59d133b0cd4773bb012145927a92a4]). Expected version 2 got 1
I20260130 02:57:59.970561 19807 raft_consensus.cc:385] T 15918c90ffdc43a5a38b2950e9834188 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:57:59.975129 19807 raft_consensus.cc:740] T 15918c90ffdc43a5a38b2950e9834188 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 371a36d219164974abdf848841528152, State: Initialized, Role: FOLLOWER
I20260130 02:57:59.975699 19840 raft_consensus.cc:385] T 15918c90ffdc43a5a38b2950e9834188 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:57:59.975972 19840 raft_consensus.cc:740] T 15918c90ffdc43a5a38b2950e9834188 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78c2c46c7520492db35d51c244fb4f74, State: Initialized, Role: FOLLOWER
I20260130 02:57:59.975745 19807 consensus_queue.cc:260] T 15918c90ffdc43a5a38b2950e9834188 P 371a36d219164974abdf848841528152 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:57:59.976588 19840 consensus_queue.cc:260] T 15918c90ffdc43a5a38b2950e9834188 P 78c2c46c7520492db35d51c244fb4f74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:57:59.981073 19840 ts_tablet_manager.cc:1434] T 15918c90ffdc43a5a38b2950e9834188 P 78c2c46c7520492db35d51c244fb4f74: Time spent starting tablet: real 0.012s user 0.004s sys 0.002s
I20260130 02:57:59.982019 19840 tablet_bootstrap.cc:492] T 26560f746d244e58a182eb6bb24c8720 P 78c2c46c7520492db35d51c244fb4f74: Bootstrap starting.
I20260130 02:57:59.988729 19807 ts_tablet_manager.cc:1434] T 15918c90ffdc43a5a38b2950e9834188 P 371a36d219164974abdf848841528152: Time spent starting tablet: real 0.034s user 0.005s sys 0.000s
I20260130 02:57:59.991793 19840 tablet_bootstrap.cc:654] T 26560f746d244e58a182eb6bb24c8720 P 78c2c46c7520492db35d51c244fb4f74: Neither blocks nor log segments found. Creating new log.
I20260130 02:57:59.981140 18868 catalog_manager.cc:5746] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571) does not have the latest schema for tablet 2112b0402de5464cb1caf2c2abb67d78 (table default.Table_da3d581b15a94840898777f74611994e [id=b7e241ced53e4160ac6fa1f65ef3674e]). Expected version 1 got 0
I20260130 02:57:59.996603 19807 tablet_bootstrap.cc:492] T 7a902d0d571a42bb82ebffd48f7eca3f P 371a36d219164974abdf848841528152: Bootstrap starting.
I20260130 02:58:00.004177 19663 raft_consensus.cc:493] T 7a902d0d571a42bb82ebffd48f7eca3f P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:00.004483 19807 tablet_bootstrap.cc:654] T 7a902d0d571a42bb82ebffd48f7eca3f P 371a36d219164974abdf848841528152: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:00.004655 19663 raft_consensus.cc:515] T 7a902d0d571a42bb82ebffd48f7eca3f P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:00.007980 19221 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7a902d0d571a42bb82ebffd48f7eca3f" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152" is_pre_election: true
I20260130 02:58:00.009224 19861 raft_consensus.cc:493] T 15918c90ffdc43a5a38b2950e9834188 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:00.009675 19861 raft_consensus.cc:515] T 15918c90ffdc43a5a38b2950e9834188 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
W20260130 02:58:00.010088 19314 leader_election.cc:343] T 7a902d0d571a42bb82ebffd48f7eca3f P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:00.006433 19663 leader_election.cc:290] T 7a902d0d571a42bb82ebffd48f7eca3f P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d870164263bb44b6b780a557daea3656 (127.18.100.67:39571), 371a36d219164974abdf848841528152 (127.18.100.65:42021)
I20260130 02:58:00.011363 19840 tablet_bootstrap.cc:492] T 26560f746d244e58a182eb6bb24c8720 P 78c2c46c7520492db35d51c244fb4f74: No bootstrap required, opened a new log
I20260130 02:58:00.011391 19861 leader_election.cc:290] T 15918c90ffdc43a5a38b2950e9834188 P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767), d870164263bb44b6b780a557daea3656 (127.18.100.67:39571)
I20260130 02:58:00.011739 19840 ts_tablet_manager.cc:1403] T 26560f746d244e58a182eb6bb24c8720 P 78c2c46c7520492db35d51c244fb4f74: Time spent bootstrapping tablet: real 0.030s user 0.005s sys 0.007s
I20260130 02:58:00.012666 19377 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "15918c90ffdc43a5a38b2950e9834188" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74" is_pre_election: true
I20260130 02:58:00.013090 19527 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "15918c90ffdc43a5a38b2950e9834188" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
I20260130 02:58:00.013204 19377 raft_consensus.cc:2468] T 15918c90ffdc43a5a38b2950e9834188 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 371a36d219164974abdf848841528152 in term 0.
I20260130 02:58:00.013664 19527 raft_consensus.cc:2468] T 15918c90ffdc43a5a38b2950e9834188 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 371a36d219164974abdf848841528152 in term 0.
I20260130 02:58:00.013978 19840 raft_consensus.cc:359] T 26560f746d244e58a182eb6bb24c8720 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:00.014855 19159 leader_election.cc:304] T 15918c90ffdc43a5a38b2950e9834188 P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 371a36d219164974abdf848841528152, d870164263bb44b6b780a557daea3656; no voters:
I20260130 02:58:00.014564 19840 raft_consensus.cc:385] T 26560f746d244e58a182eb6bb24c8720 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:00.015440 19840 raft_consensus.cc:740] T 26560f746d244e58a182eb6bb24c8720 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78c2c46c7520492db35d51c244fb4f74, State: Initialized, Role: FOLLOWER
I20260130 02:58:00.016157 19861 raft_consensus.cc:2804] T 15918c90ffdc43a5a38b2950e9834188 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260130 02:58:00.015967 19840 consensus_queue.cc:260] T 26560f746d244e58a182eb6bb24c8720 P 78c2c46c7520492db35d51c244fb4f74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:00.016461 19861 raft_consensus.cc:493] T 15918c90ffdc43a5a38b2950e9834188 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260130 02:58:00.016755 19861 raft_consensus.cc:3060] T 15918c90ffdc43a5a38b2950e9834188 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:00.017408 18868 catalog_manager.cc:5746] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571) does not have the latest schema for tablet 37470ce4356f4542b621e21698551209 (table default.Table_007e969f70d345a7bcb3dfbaabbc025d [id=77d4fa8705a3419cb0a84d3ca1bb4ce3]). Expected version 1 got 0
I20260130 02:58:00.018903 19527 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7a902d0d571a42bb82ebffd48f7eca3f" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
I20260130 02:58:00.019471 19527 raft_consensus.cc:2468] T 7a902d0d571a42bb82ebffd48f7eca3f P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78c2c46c7520492db35d51c244fb4f74 in term 0.
I20260130 02:58:00.020087 19840 ts_tablet_manager.cc:1434] T 26560f746d244e58a182eb6bb24c8720 P 78c2c46c7520492db35d51c244fb4f74: Time spent starting tablet: real 0.008s user 0.005s sys 0.000s
I20260130 02:58:00.020889 19840 tablet_bootstrap.cc:492] T 5b2976108584478a98fd90fe6594af66 P 78c2c46c7520492db35d51c244fb4f74: Bootstrap starting.
I20260130 02:58:00.024173 19861 raft_consensus.cc:515] T 15918c90ffdc43a5a38b2950e9834188 P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:00.030737 19625 tablet_bootstrap.cc:654] T e026dee22493424692d00f14848ab57c P d870164263bb44b6b780a557daea3656: Neither blocks nor log segments found. Creating new log.
W20260130 02:58:00.033221 18860 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.100.125:39711, user_credentials={real_user=slave}} blocked reactor thread for 82956.2us
I20260130 02:58:00.034034 19840 tablet_bootstrap.cc:654] T 5b2976108584478a98fd90fe6594af66 P 78c2c46c7520492db35d51c244fb4f74: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:00.039919 19861 leader_election.cc:290] T 15918c90ffdc43a5a38b2950e9834188 P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 election: Requested vote from peers 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767), d870164263bb44b6b780a557daea3656 (127.18.100.67:39571)
I20260130 02:58:00.040935 19380 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "15918c90ffdc43a5a38b2950e9834188" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74"
I20260130 02:58:00.041553 19380 raft_consensus.cc:3060] T 15918c90ffdc43a5a38b2950e9834188 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:00.046949 19527 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "15918c90ffdc43a5a38b2950e9834188" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656"
I20260130 02:58:00.047644 19527 raft_consensus.cc:3060] T 15918c90ffdc43a5a38b2950e9834188 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:00.054761 19807 tablet_bootstrap.cc:492] T 7a902d0d571a42bb82ebffd48f7eca3f P 371a36d219164974abdf848841528152: No bootstrap required, opened a new log
I20260130 02:58:00.055176 19807 ts_tablet_manager.cc:1403] T 7a902d0d571a42bb82ebffd48f7eca3f P 371a36d219164974abdf848841528152: Time spent bootstrapping tablet: real 0.059s user 0.009s sys 0.000s
I20260130 02:58:00.055647 19840 tablet_bootstrap.cc:492] T 5b2976108584478a98fd90fe6594af66 P 78c2c46c7520492db35d51c244fb4f74: No bootstrap required, opened a new log
I20260130 02:58:00.056057 19840 ts_tablet_manager.cc:1403] T 5b2976108584478a98fd90fe6594af66 P 78c2c46c7520492db35d51c244fb4f74: Time spent bootstrapping tablet: real 0.035s user 0.010s sys 0.001s
I20260130 02:58:00.057425 19807 raft_consensus.cc:359] T 7a902d0d571a42bb82ebffd48f7eca3f P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:00.058039 19807 raft_consensus.cc:385] T 7a902d0d571a42bb82ebffd48f7eca3f P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:00.058297 19807 raft_consensus.cc:740] T 7a902d0d571a42bb82ebffd48f7eca3f P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 371a36d219164974abdf848841528152, State: Initialized, Role: FOLLOWER
I20260130 02:58:00.058784 19840 raft_consensus.cc:359] T 5b2976108584478a98fd90fe6594af66 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:00.059383 19840 raft_consensus.cc:385] T 5b2976108584478a98fd90fe6594af66 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:00.059626 19840 raft_consensus.cc:740] T 5b2976108584478a98fd90fe6594af66 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78c2c46c7520492db35d51c244fb4f74, State: Initialized, Role: FOLLOWER
I20260130 02:58:00.060184 19840 consensus_queue.cc:260] T 5b2976108584478a98fd90fe6594af66 P 78c2c46c7520492db35d51c244fb4f74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:00.062413 19840 ts_tablet_manager.cc:1434] T 5b2976108584478a98fd90fe6594af66 P 78c2c46c7520492db35d51c244fb4f74: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260130 02:58:00.064679 19807 consensus_queue.cc:260] T 7a902d0d571a42bb82ebffd48f7eca3f P 371a36d219164974abdf848841528152 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:00.066711 19840 tablet_bootstrap.cc:492] T c7c621b99deb457998c44ab1644d3939 P 78c2c46c7520492db35d51c244fb4f74: Bootstrap starting.
I20260130 02:58:00.066918 19663 raft_consensus.cc:493] T d701152960b14396a6ac1bd10480d2b6 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:00.067322 19663 raft_consensus.cc:515] T d701152960b14396a6ac1bd10480d2b6 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:00.069087 19663 leader_election.cc:290] T d701152960b14396a6ac1bd10480d2b6 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), d870164263bb44b6b780a557daea3656 (127.18.100.67:39571)
I20260130 02:58:00.069433 19380 raft_consensus.cc:2468] T 15918c90ffdc43a5a38b2950e9834188 P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 371a36d219164974abdf848841528152 in term 1.
I20260130 02:58:00.073647 19221 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d701152960b14396a6ac1bd10480d2b6" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152" is_pre_election: true
I20260130 02:58:00.074223 19221 raft_consensus.cc:2468] T d701152960b14396a6ac1bd10480d2b6 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78c2c46c7520492db35d51c244fb4f74 in term 0.
I20260130 02:58:00.077795 19807 ts_tablet_manager.cc:1434] T 7a902d0d571a42bb82ebffd48f7eca3f P 371a36d219164974abdf848841528152: Time spent starting tablet: real 0.022s user 0.005s sys 0.000s
I20260130 02:58:00.085270 19807 tablet_bootstrap.cc:492] T e026dee22493424692d00f14848ab57c P 371a36d219164974abdf848841528152: Bootstrap starting.
I20260130 02:58:00.089686 19527 raft_consensus.cc:2468] T 15918c90ffdc43a5a38b2950e9834188 P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 371a36d219164974abdf848841528152 in term 1.
I20260130 02:58:00.090292 19807 tablet_bootstrap.cc:654] T e026dee22493424692d00f14848ab57c P 371a36d219164974abdf848841528152: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:00.091387 19159 leader_election.cc:304] T 15918c90ffdc43a5a38b2950e9834188 P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 371a36d219164974abdf848841528152, d870164263bb44b6b780a557daea3656; no voters:
I20260130 02:58:00.071637 19840 tablet_bootstrap.cc:654] T c7c621b99deb457998c44ab1644d3939 P 78c2c46c7520492db35d51c244fb4f74: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:00.092067 19861 raft_consensus.cc:2804] T 15918c90ffdc43a5a38b2950e9834188 P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Leader election won for term 1
I20260130 02:58:00.092485 19861 raft_consensus.cc:697] T 15918c90ffdc43a5a38b2950e9834188 P 371a36d219164974abdf848841528152 [term 1 LEADER]: Becoming Leader. State: Replica: 371a36d219164974abdf848841528152, State: Running, Role: LEADER
I20260130 02:58:00.093211 19861 consensus_queue.cc:237] T 15918c90ffdc43a5a38b2950e9834188 P 371a36d219164974abdf848841528152 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:00.075675 19314 leader_election.cc:304] T d701152960b14396a6ac1bd10480d2b6 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 371a36d219164974abdf848841528152, 78c2c46c7520492db35d51c244fb4f74; no voters:
I20260130 02:58:00.095418 19663 raft_consensus.cc:2804] T d701152960b14396a6ac1bd10480d2b6 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260130 02:58:00.095737 19663 raft_consensus.cc:493] T d701152960b14396a6ac1bd10480d2b6 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260130 02:58:00.096035 19663 raft_consensus.cc:3060] T d701152960b14396a6ac1bd10480d2b6 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Advancing to term 1
W20260130 02:58:00.106644 19313 outbound_call.cc:321] RPC callback for RPC call kudu.master.MasterService.TSHeartbeat -> {remote=127.18.100.125:39711, user_credentials={real_user=slave}} blocked reactor thread for 85562.4us
I20260130 02:58:00.102247 19663 raft_consensus.cc:515] T d701152960b14396a6ac1bd10480d2b6 P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:00.113888 19221 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d701152960b14396a6ac1bd10480d2b6" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152"
I20260130 02:58:00.114441 19221 raft_consensus.cc:3060] T d701152960b14396a6ac1bd10480d2b6 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:00.114634 19663 leader_election.cc:290] T d701152960b14396a6ac1bd10480d2b6 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 election: Requested vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), d870164263bb44b6b780a557daea3656 (127.18.100.67:39571)
I20260130 02:58:00.121258 19221 raft_consensus.cc:2468] T d701152960b14396a6ac1bd10480d2b6 P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78c2c46c7520492db35d51c244fb4f74 in term 1.
I20260130 02:58:00.122563 19527 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d701152960b14396a6ac1bd10480d2b6" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
I20260130 02:58:00.125818 19314 leader_election.cc:304] T d701152960b14396a6ac1bd10480d2b6 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 371a36d219164974abdf848841528152, 78c2c46c7520492db35d51c244fb4f74; no voters:
I20260130 02:58:00.126749 19663 raft_consensus.cc:2804] T d701152960b14396a6ac1bd10480d2b6 P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Leader election won for term 1
I20260130 02:58:00.127439 19663 raft_consensus.cc:697] T d701152960b14396a6ac1bd10480d2b6 P 78c2c46c7520492db35d51c244fb4f74 [term 1 LEADER]: Becoming Leader. State: Replica: 78c2c46c7520492db35d51c244fb4f74, State: Running, Role: LEADER
I20260130 02:58:00.128278 19663 consensus_queue.cc:237] T d701152960b14396a6ac1bd10480d2b6 P 78c2c46c7520492db35d51c244fb4f74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:00.130180 19313 leader_election.cc:304] T 7a902d0d571a42bb82ebffd48f7eca3f P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 78c2c46c7520492db35d51c244fb4f74, d870164263bb44b6b780a557daea3656; no voters: 371a36d219164974abdf848841528152
I20260130 02:58:00.132395 19840 tablet_bootstrap.cc:492] T c7c621b99deb457998c44ab1644d3939 P 78c2c46c7520492db35d51c244fb4f74: No bootstrap required, opened a new log
I20260130 02:58:00.139921 19840 ts_tablet_manager.cc:1403] T c7c621b99deb457998c44ab1644d3939 P 78c2c46c7520492db35d51c244fb4f74: Time spent bootstrapping tablet: real 0.077s user 0.009s sys 0.009s
I20260130 02:58:00.142279 19840 raft_consensus.cc:359] T c7c621b99deb457998c44ab1644d3939 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:00.142892 19840 raft_consensus.cc:385] T c7c621b99deb457998c44ab1644d3939 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:00.144284 19625 tablet_bootstrap.cc:492] T e026dee22493424692d00f14848ab57c P d870164263bb44b6b780a557daea3656: No bootstrap required, opened a new log
I20260130 02:58:00.143133 19840 raft_consensus.cc:740] T c7c621b99deb457998c44ab1644d3939 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78c2c46c7520492db35d51c244fb4f74, State: Initialized, Role: FOLLOWER
I20260130 02:58:00.144611 19807 tablet_bootstrap.cc:492] T e026dee22493424692d00f14848ab57c P 371a36d219164974abdf848841528152: No bootstrap required, opened a new log
I20260130 02:58:00.144737 19625 ts_tablet_manager.cc:1403] T e026dee22493424692d00f14848ab57c P d870164263bb44b6b780a557daea3656: Time spent bootstrapping tablet: real 0.416s user 0.023s sys 0.005s
I20260130 02:58:00.145115 19807 ts_tablet_manager.cc:1403] T e026dee22493424692d00f14848ab57c P 371a36d219164974abdf848841528152: Time spent bootstrapping tablet: real 0.060s user 0.011s sys 0.000s
I20260130 02:58:00.145092 19840 consensus_queue.cc:260] T c7c621b99deb457998c44ab1644d3939 P 78c2c46c7520492db35d51c244fb4f74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:00.147202 19840 ts_tablet_manager.cc:1434] T c7c621b99deb457998c44ab1644d3939 P 78c2c46c7520492db35d51c244fb4f74: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260130 02:58:00.146983 19625 raft_consensus.cc:359] T e026dee22493424692d00f14848ab57c P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:00.147584 19625 raft_consensus.cc:385] T e026dee22493424692d00f14848ab57c P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:00.147477 19807 raft_consensus.cc:359] T e026dee22493424692d00f14848ab57c P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:00.147841 19625 raft_consensus.cc:740] T e026dee22493424692d00f14848ab57c P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d870164263bb44b6b780a557daea3656, State: Initialized, Role: FOLLOWER
I20260130 02:58:00.148062 19807 raft_consensus.cc:385] T e026dee22493424692d00f14848ab57c P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:00.148299 19807 raft_consensus.cc:740] T e026dee22493424692d00f14848ab57c P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 371a36d219164974abdf848841528152, State: Initialized, Role: FOLLOWER
I20260130 02:58:00.149091 19807 consensus_queue.cc:260] T e026dee22493424692d00f14848ab57c P 371a36d219164974abdf848841528152 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:00.149922 19663 raft_consensus.cc:2804] T 7a902d0d571a42bb82ebffd48f7eca3f P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260130 02:58:00.150195 19663 raft_consensus.cc:493] T 7a902d0d571a42bb82ebffd48f7eca3f P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260130 02:58:00.150437 19663 raft_consensus.cc:3060] T 7a902d0d571a42bb82ebffd48f7eca3f P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Advancing to term 1
W20260130 02:58:00.150480 19313 leader_election.cc:343] T d701152960b14396a6ac1bd10480d2b6 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:00.152184 19527 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d701152960b14396a6ac1bd10480d2b6" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656"
I20260130 02:58:00.153782 19840 tablet_bootstrap.cc:492] T 1ca34772ef484c1f809a99d126e28eff P 78c2c46c7520492db35d51c244fb4f74: Bootstrap starting.
I20260130 02:58:00.155062 19807 ts_tablet_manager.cc:1434] T e026dee22493424692d00f14848ab57c P 371a36d219164974abdf848841528152: Time spent starting tablet: real 0.010s user 0.005s sys 0.000s
I20260130 02:58:00.154827 19856 consensus_queue.cc:1048] T ffa1142fc7274aecbdbee1a0802b95da P 78c2c46c7520492db35d51c244fb4f74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260130 02:58:00.157958 19663 raft_consensus.cc:515] T 7a902d0d571a42bb82ebffd48f7eca3f P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:00.160602 19221 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7a902d0d571a42bb82ebffd48f7eca3f" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152"
I20260130 02:58:00.161219 19221 raft_consensus.cc:3060] T 7a902d0d571a42bb82ebffd48f7eca3f P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:00.163998 19663 leader_election.cc:290] T 7a902d0d571a42bb82ebffd48f7eca3f P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 election: Requested vote from peers d870164263bb44b6b780a557daea3656 (127.18.100.67:39571), 371a36d219164974abdf848841528152 (127.18.100.65:42021)
I20260130 02:58:00.164088 18871 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50284:
table { table_name: "default.Table_a9476edf8a5445c883f8d6ce786554a2" } modify_external_catalogs: true
I20260130 02:58:00.164775 18871 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50284:
table { table_name: "default.Table_a9476edf8a5445c883f8d6ce786554a2" } modify_external_catalogs: true
W20260130 02:58:00.167364 18875 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 15918c90ffdc43a5a38b2950e9834188 consensus state has no leader
I20260130 02:58:00.168949 19221 raft_consensus.cc:2468] T 7a902d0d571a42bb82ebffd48f7eca3f P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78c2c46c7520492db35d51c244fb4f74 in term 1.
I20260130 02:58:00.168622 18875 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50284:
name: "default.Table_c04335826bdf4c2c94c4cf6e3bca1a7a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260130 02:58:00.169970 19314 leader_election.cc:304] T 7a902d0d571a42bb82ebffd48f7eca3f P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 371a36d219164974abdf848841528152, 78c2c46c7520492db35d51c244fb4f74; no voters:
W20260130 02:58:00.170291 18875 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c04335826bdf4c2c94c4cf6e3bca1a7a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260130 02:58:00.170894 19899 raft_consensus.cc:2804] T 7a902d0d571a42bb82ebffd48f7eca3f P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Leader election won for term 1
I20260130 02:58:00.171288 19899 raft_consensus.cc:697] T 7a902d0d571a42bb82ebffd48f7eca3f P 78c2c46c7520492db35d51c244fb4f74 [term 1 LEADER]: Becoming Leader. State: Replica: 78c2c46c7520492db35d51c244fb4f74, State: Running, Role: LEADER
I20260130 02:58:00.172014 19899 consensus_queue.cc:237] T 7a902d0d571a42bb82ebffd48f7eca3f P 78c2c46c7520492db35d51c244fb4f74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:00.188720 19807 tablet_bootstrap.cc:492] T 26560f746d244e58a182eb6bb24c8720 P 371a36d219164974abdf848841528152: Bootstrap starting.
I20260130 02:58:00.191756 18869 catalog_manager.cc:5941] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4: Sending DeleteTablet for 3 replicas of tablet 878f9346d3044d8d8b15c61f86869212
I20260130 02:58:00.193066 18869 catalog_manager.cc:6724] ReplaceTablet: tablet 878f9346d3044d8d8b15c61f86869212 deleted and replaced by tablet 3b148691090e4ee581d46f9845a6e49a
I20260130 02:58:00.193902 18869 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50284:
name: "default.Table_1e9e20a1bfec4bcf81f9eb6219c33d59"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260130 02:58:00.195531 19204 tablet_service.cc:1552] Processing DeleteTablet for tablet 878f9346d3044d8d8b15c61f86869212 with delete_type TABLET_DATA_DELETED (replaced by tablet 3b148691090e4ee581d46f9845a6e49a) from {username='slave'} at 127.0.0.1:42870
I20260130 02:58:00.196149 19754 tablet_replica.cc:333] T 878f9346d3044d8d8b15c61f86869212 P 371a36d219164974abdf848841528152: stopping tablet replica
I20260130 02:58:00.196784 19754 raft_consensus.cc:2243] T 878f9346d3044d8d8b15c61f86869212 P 371a36d219164974abdf848841528152 [term 1 LEADER]: Raft consensus shutting down.
W20260130 02:58:00.196929 18869 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1e9e20a1bfec4bcf81f9eb6219c33d59 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260130 02:58:00.197556 19754 raft_consensus.cc:2272] T 878f9346d3044d8d8b15c61f86869212 P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260130 02:58:00.199188 19754 ts_tablet_manager.cc:1916] T 878f9346d3044d8d8b15c61f86869212 P 371a36d219164974abdf848841528152: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260130 02:58:00.206310 19360 tablet_service.cc:1552] Processing DeleteTablet for tablet 878f9346d3044d8d8b15c61f86869212 with delete_type TABLET_DATA_DELETED (replaced by tablet 3b148691090e4ee581d46f9845a6e49a) from {username='slave'} at 127.0.0.1:60110
I20260130 02:58:00.207032 19758 tablet_replica.cc:333] T 878f9346d3044d8d8b15c61f86869212 P 78c2c46c7520492db35d51c244fb4f74: stopping tablet replica
I20260130 02:58:00.161382 19625 consensus_queue.cc:260] T e026dee22493424692d00f14848ab57c P d870164263bb44b6b780a557daea3656 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:00.207670 19758 raft_consensus.cc:2243] T 878f9346d3044d8d8b15c61f86869212 P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260130 02:58:00.208139 19758 raft_consensus.cc:2272] T 878f9346d3044d8d8b15c61f86869212 P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260130 02:58:00.208423 19507 tablet_service.cc:1552] Processing DeleteTablet for tablet 878f9346d3044d8d8b15c61f86869212 with delete_type TABLET_DATA_DELETED (replaced by tablet 3b148691090e4ee581d46f9845a6e49a) from {username='slave'} at 127.0.0.1:59068
I20260130 02:58:00.209659 19526 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7a902d0d571a42bb82ebffd48f7eca3f" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656"
I20260130 02:58:00.209895 19758 ts_tablet_manager.cc:1916] T 878f9346d3044d8d8b15c61f86869212 P 78c2c46c7520492db35d51c244fb4f74: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260130 02:58:00.210251 19526 raft_consensus.cc:3060] T 7a902d0d571a42bb82ebffd48f7eca3f P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:00.217800 19526 raft_consensus.cc:2468] T 7a902d0d571a42bb82ebffd48f7eca3f P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78c2c46c7520492db35d51c244fb4f74 in term 1.
I20260130 02:58:00.196770 19807 tablet_bootstrap.cc:654] T 26560f746d244e58a182eb6bb24c8720 P 371a36d219164974abdf848841528152: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:00.226032 19758 ts_tablet_manager.cc:1929] T 878f9346d3044d8d8b15c61f86869212 P 78c2c46c7520492db35d51c244fb4f74: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20260130 02:58:00.224397 19625 ts_tablet_manager.cc:1434] T e026dee22493424692d00f14848ab57c P d870164263bb44b6b780a557daea3656: Time spent starting tablet: real 0.079s user 0.005s sys 0.000s
I20260130 02:58:00.227594 19754 ts_tablet_manager.cc:1929] T 878f9346d3044d8d8b15c61f86869212 P 371a36d219164974abdf848841528152: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20260130 02:58:00.227943 19754 log.cc:1199] T 878f9346d3044d8d8b15c61f86869212 P 371a36d219164974abdf848841528152: Deleting WAL directory at /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-0/wal/wals/878f9346d3044d8d8b15c61f86869212
I20260130 02:58:00.231460 19947 tablet_replica.cc:333] T 878f9346d3044d8d8b15c61f86869212 P d870164263bb44b6b780a557daea3656: stopping tablet replica
I20260130 02:58:00.228571 19754 ts_tablet_manager.cc:1950] T 878f9346d3044d8d8b15c61f86869212 P 371a36d219164974abdf848841528152: Deleting consensus metadata
I20260130 02:58:00.235478 19947 raft_consensus.cc:2243] T 878f9346d3044d8d8b15c61f86869212 P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260130 02:58:00.239698 19947 raft_consensus.cc:2272] T 878f9346d3044d8d8b15c61f86869212 P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260130 02:58:00.239851 19807 tablet_bootstrap.cc:492] T 26560f746d244e58a182eb6bb24c8720 P 371a36d219164974abdf848841528152: No bootstrap required, opened a new log
I20260130 02:58:00.240238 19807 ts_tablet_manager.cc:1403] T 26560f746d244e58a182eb6bb24c8720 P 371a36d219164974abdf848841528152: Time spent bootstrapping tablet: real 0.052s user 0.010s sys 0.000s
I20260130 02:58:00.228011 19625 tablet_bootstrap.cc:492] T d701152960b14396a6ac1bd10480d2b6 P d870164263bb44b6b780a557daea3656: Bootstrap starting.
I20260130 02:58:00.242698 19861 raft_consensus.cc:493] T ffa1142fc7274aecbdbee1a0802b95da P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:00.242399 19807 raft_consensus.cc:359] T 26560f746d244e58a182eb6bb24c8720 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:00.243142 19807 raft_consensus.cc:385] T 26560f746d244e58a182eb6bb24c8720 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:00.243110 19861 raft_consensus.cc:515] T ffa1142fc7274aecbdbee1a0802b95da P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:00.243569 19807 raft_consensus.cc:740] T 26560f746d244e58a182eb6bb24c8720 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 371a36d219164974abdf848841528152, State: Initialized, Role: FOLLOWER
I20260130 02:58:00.244915 19861 leader_election.cc:290] T ffa1142fc7274aecbdbee1a0802b95da P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d870164263bb44b6b780a557daea3656 (127.18.100.67:39571), 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767)
I20260130 02:58:00.244869 18861 catalog_manager.cc:4985] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021): tablet 878f9346d3044d8d8b15c61f86869212 (table default.Table_8edea1fae34d4d18b54b0f801a145225 [id=2009496525c94d7f8ee6ce59694b652d]) successfully deleted
I20260130 02:58:00.247964 19526 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ffa1142fc7274aecbdbee1a0802b95da" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
I20260130 02:58:00.250684 18875 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50284:
name: "default.Table_1c2b251034c0418cba750ae4d1308b9f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260130 02:58:00.250967 19807 consensus_queue.cc:260] T 26560f746d244e58a182eb6bb24c8720 P 371a36d219164974abdf848841528152 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
W20260130 02:58:00.252640 18875 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1c2b251034c0418cba750ae4d1308b9f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260130 02:58:00.247228 19380 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ffa1142fc7274aecbdbee1a0802b95da" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74" is_pre_election: true
I20260130 02:58:00.256419 19159 leader_election.cc:304] T ffa1142fc7274aecbdbee1a0802b95da P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 371a36d219164974abdf848841528152; no voters: 78c2c46c7520492db35d51c244fb4f74, d870164263bb44b6b780a557daea3656
I20260130 02:58:00.257335 19861 raft_consensus.cc:2749] T ffa1142fc7274aecbdbee1a0802b95da P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260130 02:58:00.257958 19758 log.cc:1199] T 878f9346d3044d8d8b15c61f86869212 P 78c2c46c7520492db35d51c244fb4f74: Deleting WAL directory at /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-1/wal/wals/878f9346d3044d8d8b15c61f86869212
I20260130 02:58:00.258708 19758 ts_tablet_manager.cc:1950] T 878f9346d3044d8d8b15c61f86869212 P 78c2c46c7520492db35d51c244fb4f74: Deleting consensus metadata
W20260130 02:58:00.220084 19313 leader_election.cc:343] T d701152960b14396a6ac1bd10480d2b6 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:00.262406 19807 ts_tablet_manager.cc:1434] T 26560f746d244e58a182eb6bb24c8720 P 371a36d219164974abdf848841528152: Time spent starting tablet: real 0.022s user 0.006s sys 0.000s
I20260130 02:58:00.264353 19947 ts_tablet_manager.cc:1916] T 878f9346d3044d8d8b15c61f86869212 P d870164263bb44b6b780a557daea3656: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260130 02:58:00.265141 19807 tablet_bootstrap.cc:492] T 5b2976108584478a98fd90fe6594af66 P 371a36d219164974abdf848841528152: Bootstrap starting.
I20260130 02:58:00.272922 19625 tablet_bootstrap.cc:654] T d701152960b14396a6ac1bd10480d2b6 P d870164263bb44b6b780a557daea3656: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:00.274323 19742 raft_consensus.cc:493] T e026dee22493424692d00f14848ab57c P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:00.274771 19742 raft_consensus.cc:515] T e026dee22493424692d00f14848ab57c P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:00.276688 19742 leader_election.cc:290] T e026dee22493424692d00f14848ab57c P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767)
I20260130 02:58:00.278208 19221 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e026dee22493424692d00f14848ab57c" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152" is_pre_election: true
I20260130 02:58:00.278860 19221 raft_consensus.cc:2468] T e026dee22493424692d00f14848ab57c P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d870164263bb44b6b780a557daea3656 in term 0.
I20260130 02:58:00.280339 19463 leader_election.cc:304] T e026dee22493424692d00f14848ab57c P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 371a36d219164974abdf848841528152, d870164263bb44b6b780a557daea3656; no voters:
I20260130 02:58:00.281199 19742 raft_consensus.cc:2804] T e026dee22493424692d00f14848ab57c P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260130 02:58:00.281426 19807 tablet_bootstrap.cc:654] T 5b2976108584478a98fd90fe6594af66 P 371a36d219164974abdf848841528152: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:00.281567 19742 raft_consensus.cc:493] T e026dee22493424692d00f14848ab57c P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260130 02:58:00.282006 19742 raft_consensus.cc:3060] T e026dee22493424692d00f14848ab57c P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:00.284292 19840 tablet_bootstrap.cc:654] T 1ca34772ef484c1f809a99d126e28eff P 78c2c46c7520492db35d51c244fb4f74: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:00.285496 18869 master_service.cc:954] ReplaceTablet: received request to replace tablet 26560f746d244e58a182eb6bb24c8720 from {username='slave'} at 127.0.0.1:50284
I20260130 02:58:00.291311 19899 raft_consensus.cc:493] T 5b2976108584478a98fd90fe6594af66 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:00.291733 19899 raft_consensus.cc:515] T 5b2976108584478a98fd90fe6594af66 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:00.294428 19221 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5b2976108584478a98fd90fe6594af66" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152" is_pre_election: true
W20260130 02:58:00.295801 19314 leader_election.cc:343] T 5b2976108584478a98fd90fe6594af66 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:00.296689 19899 leader_election.cc:290] T 5b2976108584478a98fd90fe6594af66 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), d870164263bb44b6b780a557daea3656 (127.18.100.67:39571)
I20260130 02:58:00.299122 19380 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e026dee22493424692d00f14848ab57c" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74" is_pre_election: true
I20260130 02:58:00.299660 19380 raft_consensus.cc:2468] T e026dee22493424692d00f14848ab57c P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d870164263bb44b6b780a557daea3656 in term 0.
I20260130 02:58:00.307528 19526 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5b2976108584478a98fd90fe6594af66" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
I20260130 02:58:00.310593 19947 ts_tablet_manager.cc:1929] T 878f9346d3044d8d8b15c61f86869212 P d870164263bb44b6b780a557daea3656: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20260130 02:58:00.311045 19947 log.cc:1199] T 878f9346d3044d8d8b15c61f86869212 P d870164263bb44b6b780a557daea3656: Deleting WAL directory at /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-2/wal/wals/878f9346d3044d8d8b15c61f86869212
W20260130 02:58:00.311148 19313 leader_election.cc:343] T 5b2976108584478a98fd90fe6594af66 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:00.311506 19313 leader_election.cc:304] T 5b2976108584478a98fd90fe6594af66 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78c2c46c7520492db35d51c244fb4f74; no voters: 371a36d219164974abdf848841528152, d870164263bb44b6b780a557daea3656
I20260130 02:58:00.311692 19526 raft_consensus.cc:3060] T 1bbc55bbf2254d8bb6b721d7e8f69d85 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:00.312026 19947 ts_tablet_manager.cc:1950] T 878f9346d3044d8d8b15c61f86869212 P d870164263bb44b6b780a557daea3656: Deleting consensus metadata
I20260130 02:58:00.312707 19899 raft_consensus.cc:2749] T 5b2976108584478a98fd90fe6594af66 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260130 02:58:00.286259 18860 catalog_manager.cc:4985] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767): tablet 878f9346d3044d8d8b15c61f86869212 (table default.Table_8edea1fae34d4d18b54b0f801a145225 [id=2009496525c94d7f8ee6ce59694b652d]) successfully deleted
I20260130 02:58:00.319648 19899 raft_consensus.cc:493] T e364f4e70e5749e083e14e7fa7416ff0 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:00.320058 19899 raft_consensus.cc:515] T e364f4e70e5749e083e14e7fa7416ff0 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:00.289167 19742 raft_consensus.cc:515] T e026dee22493424692d00f14848ab57c P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:00.322957 19221 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e364f4e70e5749e083e14e7fa7416ff0" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152" is_pre_election: true
I20260130 02:58:00.323501 19224 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e026dee22493424692d00f14848ab57c" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152"
I20260130 02:58:00.323632 19742 leader_election.cc:290] T e026dee22493424692d00f14848ab57c P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 election: Requested vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767)
I20260130 02:58:00.324093 19224 raft_consensus.cc:3060] T e026dee22493424692d00f14848ab57c P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:00.325253 19899 leader_election.cc:290] T e364f4e70e5749e083e14e7fa7416ff0 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d870164263bb44b6b780a557daea3656 (127.18.100.67:39571), 371a36d219164974abdf848841528152 (127.18.100.65:42021)
I20260130 02:58:00.329152 19625 tablet_bootstrap.cc:492] T d701152960b14396a6ac1bd10480d2b6 P d870164263bb44b6b780a557daea3656: No bootstrap required, opened a new log
I20260130 02:58:00.329552 19625 ts_tablet_manager.cc:1403] T d701152960b14396a6ac1bd10480d2b6 P d870164263bb44b6b780a557daea3656: Time spent bootstrapping tablet: real 0.102s user 0.004s sys 0.004s
I20260130 02:58:00.330233 19840 tablet_bootstrap.cc:492] T 1ca34772ef484c1f809a99d126e28eff P 78c2c46c7520492db35d51c244fb4f74: No bootstrap required, opened a new log
I20260130 02:58:00.330688 19840 ts_tablet_manager.cc:1403] T 1ca34772ef484c1f809a99d126e28eff P 78c2c46c7520492db35d51c244fb4f74: Time spent bootstrapping tablet: real 0.177s user 0.006s sys 0.003s
I20260130 02:58:00.330525 19224 raft_consensus.cc:2468] T e026dee22493424692d00f14848ab57c P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d870164263bb44b6b780a557daea3656 in term 1.
I20260130 02:58:00.332228 19463 leader_election.cc:304] T e026dee22493424692d00f14848ab57c P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 371a36d219164974abdf848841528152, d870164263bb44b6b780a557daea3656; no voters:
I20260130 02:58:00.333004 19742 raft_consensus.cc:2804] T e026dee22493424692d00f14848ab57c P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Leader election won for term 1
I20260130 02:58:00.333424 19742 raft_consensus.cc:697] T e026dee22493424692d00f14848ab57c P d870164263bb44b6b780a557daea3656 [term 1 LEADER]: Becoming Leader. State: Replica: d870164263bb44b6b780a557daea3656, State: Running, Role: LEADER
I20260130 02:58:00.334146 19742 consensus_queue.cc:237] T e026dee22493424692d00f14848ab57c P d870164263bb44b6b780a557daea3656 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:00.333137 19840 raft_consensus.cc:359] T 1ca34772ef484c1f809a99d126e28eff P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:00.335474 19840 raft_consensus.cc:385] T 1ca34772ef484c1f809a99d126e28eff P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:00.335759 19840 raft_consensus.cc:740] T 1ca34772ef484c1f809a99d126e28eff P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78c2c46c7520492db35d51c244fb4f74, State: Initialized, Role: FOLLOWER
I20260130 02:58:00.337453 18860 catalog_manager.cc:4985] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): tablet 878f9346d3044d8d8b15c61f86869212 (table default.Table_8edea1fae34d4d18b54b0f801a145225 [id=2009496525c94d7f8ee6ce59694b652d]) successfully deleted
I20260130 02:58:00.338148 19840 consensus_queue.cc:260] T 1ca34772ef484c1f809a99d126e28eff P 78c2c46c7520492db35d51c244fb4f74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:00.341691 19625 raft_consensus.cc:359] T d701152960b14396a6ac1bd10480d2b6 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:00.342293 19625 raft_consensus.cc:385] T d701152960b14396a6ac1bd10480d2b6 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:00.342518 19625 raft_consensus.cc:740] T d701152960b14396a6ac1bd10480d2b6 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d870164263bb44b6b780a557daea3656, State: Initialized, Role: FOLLOWER
I20260130 02:58:00.345132 19807 tablet_bootstrap.cc:492] T 5b2976108584478a98fd90fe6594af66 P 371a36d219164974abdf848841528152: No bootstrap required, opened a new log
I20260130 02:58:00.345834 19527 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e364f4e70e5749e083e14e7fa7416ff0" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
I20260130 02:58:00.346513 19527 raft_consensus.cc:2393] T e364f4e70e5749e083e14e7fa7416ff0 P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 78c2c46c7520492db35d51c244fb4f74 in current term 1: Already voted for candidate 371a36d219164974abdf848841528152 in this term.
I20260130 02:58:00.346668 19807 ts_tablet_manager.cc:1403] T 5b2976108584478a98fd90fe6594af66 P 371a36d219164974abdf848841528152: Time spent bootstrapping tablet: real 0.082s user 0.017s sys 0.005s
I20260130 02:58:00.344871 19625 consensus_queue.cc:260] T d701152960b14396a6ac1bd10480d2b6 P d870164263bb44b6b780a557daea3656 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:00.349217 19807 raft_consensus.cc:359] T 5b2976108584478a98fd90fe6594af66 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:00.351917 19840 ts_tablet_manager.cc:1434] T 1ca34772ef484c1f809a99d126e28eff P 78c2c46c7520492db35d51c244fb4f74: Time spent starting tablet: real 0.021s user 0.005s sys 0.000s
I20260130 02:58:00.353073 19625 ts_tablet_manager.cc:1434] T d701152960b14396a6ac1bd10480d2b6 P d870164263bb44b6b780a557daea3656: Time spent starting tablet: real 0.023s user 0.007s sys 0.000s
I20260130 02:58:00.353408 19807 raft_consensus.cc:385] T 5b2976108584478a98fd90fe6594af66 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:00.353907 19625 tablet_bootstrap.cc:492] T 5b2976108584478a98fd90fe6594af66 P d870164263bb44b6b780a557daea3656: Bootstrap starting.
I20260130 02:58:00.354182 19807 raft_consensus.cc:740] T 5b2976108584478a98fd90fe6594af66 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 371a36d219164974abdf848841528152, State: Initialized, Role: FOLLOWER
I20260130 02:58:00.354846 19840 tablet_bootstrap.cc:492] T b7b596f944be4daf953b82b65d5cec2a P 78c2c46c7520492db35d51c244fb4f74: Bootstrap starting.
I20260130 02:58:00.355417 19807 consensus_queue.cc:260] T 5b2976108584478a98fd90fe6594af66 P 371a36d219164974abdf848841528152 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:00.366075 19625 tablet_bootstrap.cc:654] T 5b2976108584478a98fd90fe6594af66 P d870164263bb44b6b780a557daea3656: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:00.367805 19807 ts_tablet_manager.cc:1434] T 5b2976108584478a98fd90fe6594af66 P 371a36d219164974abdf848841528152: Time spent starting tablet: real 0.021s user 0.005s sys 0.000s
I20260130 02:58:00.370707 19807 tablet_bootstrap.cc:492] T c7c621b99deb457998c44ab1644d3939 P 371a36d219164974abdf848841528152: Bootstrap starting.
I20260130 02:58:00.373136 19899 consensus_queue.cc:1048] T 1bbc55bbf2254d8bb6b721d7e8f69d85 P 78c2c46c7520492db35d51c244fb4f74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260130 02:58:00.374846 19840 tablet_bootstrap.cc:654] T b7b596f944be4daf953b82b65d5cec2a P 78c2c46c7520492db35d51c244fb4f74: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:00.374775 18875 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50284:
name: "default.Table_bb334df0beee420e883c3225d93892c0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260130 02:58:00.376217 18875 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bb334df0beee420e883c3225d93892c0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260130 02:58:00.379201 19807 tablet_bootstrap.cc:654] T c7c621b99deb457998c44ab1644d3939 P 371a36d219164974abdf848841528152: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:00.390003 19313 leader_election.cc:304] T e364f4e70e5749e083e14e7fa7416ff0 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78c2c46c7520492db35d51c244fb4f74; no voters: 371a36d219164974abdf848841528152, d870164263bb44b6b780a557daea3656
I20260130 02:58:00.397066 18875 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50284:
table { table_name: "default.Table_495035e92963454c8690f28fb4557c00" } new_table_name: "default.Table_bc11cb1bce0348ed9ea0912b3ee521b9" modify_external_catalogs: true
I20260130 02:58:00.402662 19625 tablet_bootstrap.cc:492] T 5b2976108584478a98fd90fe6594af66 P d870164263bb44b6b780a557daea3656: No bootstrap required, opened a new log
I20260130 02:58:00.403059 19625 ts_tablet_manager.cc:1403] T 5b2976108584478a98fd90fe6594af66 P d870164263bb44b6b780a557daea3656: Time spent bootstrapping tablet: real 0.049s user 0.012s sys 0.000s
I20260130 02:58:00.405164 19625 raft_consensus.cc:359] T 5b2976108584478a98fd90fe6594af66 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:00.405771 19625 raft_consensus.cc:385] T 5b2976108584478a98fd90fe6594af66 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:00.405995 19625 raft_consensus.cc:740] T 5b2976108584478a98fd90fe6594af66 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d870164263bb44b6b780a557daea3656, State: Initialized, Role: FOLLOWER
I20260130 02:58:00.406740 19625 consensus_queue.cc:260] T 5b2976108584478a98fd90fe6594af66 P d870164263bb44b6b780a557daea3656 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:00.416651 19625 ts_tablet_manager.cc:1434] T 5b2976108584478a98fd90fe6594af66 P d870164263bb44b6b780a557daea3656: Time spent starting tablet: real 0.013s user 0.006s sys 0.000s
I20260130 02:58:00.417749 19625 tablet_bootstrap.cc:492] T c7c621b99deb457998c44ab1644d3939 P d870164263bb44b6b780a557daea3656: Bootstrap starting.
I20260130 02:58:00.421375 19224 raft_consensus.cc:3060] T ffa1142fc7274aecbdbee1a0802b95da P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:00.423018 19625 tablet_bootstrap.cc:654] T c7c621b99deb457998c44ab1644d3939 P d870164263bb44b6b780a557daea3656: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:00.430403 19899 raft_consensus.cc:3060] T e364f4e70e5749e083e14e7fa7416ff0 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:00.430807 19742 raft_consensus.cc:493] T 1bbc55bbf2254d8bb6b721d7e8f69d85 P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:00.431182 19856 consensus_queue.cc:1048] T ffa1142fc7274aecbdbee1a0802b95da P 78c2c46c7520492db35d51c244fb4f74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260130 02:58:00.431246 19742 raft_consensus.cc:515] T 1bbc55bbf2254d8bb6b721d7e8f69d85 P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:00.432993 19742 leader_election.cc:290] T 1bbc55bbf2254d8bb6b721d7e8f69d85 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767)
I20260130 02:58:00.434428 19223 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1bbc55bbf2254d8bb6b721d7e8f69d85" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152" is_pre_election: true
I20260130 02:58:00.435027 19223 raft_consensus.cc:2468] T 1bbc55bbf2254d8bb6b721d7e8f69d85 P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d870164263bb44b6b780a557daea3656 in term 1.
W20260130 02:58:00.436038 19313 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.100.67:39571, user_credentials={real_user=slave}} blocked reactor thread for 46101.5us
I20260130 02:58:00.436203 19463 leader_election.cc:304] T 1bbc55bbf2254d8bb6b721d7e8f69d85 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 371a36d219164974abdf848841528152, d870164263bb44b6b780a557daea3656; no voters:
I20260130 02:58:00.437145 19742 raft_consensus.cc:2804] T 1bbc55bbf2254d8bb6b721d7e8f69d85 P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260130 02:58:00.437484 19742 raft_consensus.cc:493] T 1bbc55bbf2254d8bb6b721d7e8f69d85 P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260130 02:58:00.437803 19742 raft_consensus.cc:3060] T 1bbc55bbf2254d8bb6b721d7e8f69d85 P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Advancing to term 2
I20260130 02:58:00.446457 19742 raft_consensus.cc:515] T 1bbc55bbf2254d8bb6b721d7e8f69d85 P d870164263bb44b6b780a557daea3656 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:00.449291 19224 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1bbc55bbf2254d8bb6b721d7e8f69d85" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152"
I20260130 02:58:00.449936 19224 raft_consensus.cc:3060] T 1bbc55bbf2254d8bb6b721d7e8f69d85 P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Advancing to term 2
I20260130 02:58:00.454105 19380 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e026dee22493424692d00f14848ab57c" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74"
I20260130 02:58:00.454646 19380 raft_consensus.cc:3060] T e026dee22493424692d00f14848ab57c P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:00.460150 19380 raft_consensus.cc:2468] T e026dee22493424692d00f14848ab57c P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d870164263bb44b6b780a557daea3656 in term 1.
I20260130 02:58:00.463431 19742 leader_election.cc:290] T 1bbc55bbf2254d8bb6b721d7e8f69d85 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 2 election: Requested vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767)
I20260130 02:58:00.465179 19899 raft_consensus.cc:2749] T e364f4e70e5749e083e14e7fa7416ff0 P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260130 02:58:00.466302 19224 raft_consensus.cc:2468] T 1bbc55bbf2254d8bb6b721d7e8f69d85 P 371a36d219164974abdf848841528152 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d870164263bb44b6b780a557daea3656 in term 2.
I20260130 02:58:00.468242 19463 leader_election.cc:304] T 1bbc55bbf2254d8bb6b721d7e8f69d85 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 371a36d219164974abdf848841528152, d870164263bb44b6b780a557daea3656; no voters:
I20260130 02:58:00.468390 19380 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1bbc55bbf2254d8bb6b721d7e8f69d85" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74" is_pre_election: true
I20260130 02:58:00.469174 19742 raft_consensus.cc:2804] T 1bbc55bbf2254d8bb6b721d7e8f69d85 P d870164263bb44b6b780a557daea3656 [term 2 FOLLOWER]: Leader election won for term 2
I20260130 02:58:00.469594 19742 raft_consensus.cc:697] T 1bbc55bbf2254d8bb6b721d7e8f69d85 P d870164263bb44b6b780a557daea3656 [term 2 LEADER]: Becoming Leader. State: Replica: d870164263bb44b6b780a557daea3656, State: Running, Role: LEADER
I20260130 02:58:00.470357 19380 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1bbc55bbf2254d8bb6b721d7e8f69d85" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74"
I20260130 02:58:00.472141 19742 consensus_queue.cc:237] T 1bbc55bbf2254d8bb6b721d7e8f69d85 P d870164263bb44b6b780a557daea3656 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:00.481856 19527 raft_consensus.cc:1240] T 1bbc55bbf2254d8bb6b721d7e8f69d85 P d870164263bb44b6b780a557daea3656 [term 2 LEADER]: Rejecting Update request from peer 78c2c46c7520492db35d51c244fb4f74 for earlier term 1. Current term is 2. Ops: [1.1-1.1]
I20260130 02:58:00.485556 19224 raft_consensus.cc:1240] T 1bbc55bbf2254d8bb6b721d7e8f69d85 P 371a36d219164974abdf848841528152 [term 2 FOLLOWER]: Rejecting Update request from peer 78c2c46c7520492db35d51c244fb4f74 for earlier term 1. Current term is 2. Ops: []
I20260130 02:58:00.489107 19663 consensus_queue.cc:1059] T 1bbc55bbf2254d8bb6b721d7e8f69d85 P 78c2c46c7520492db35d51c244fb4f74 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 }, Status: INVALID_TERM, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260130 02:58:00.490113 19856 raft_consensus.cc:3055] T 1bbc55bbf2254d8bb6b721d7e8f69d85 P 78c2c46c7520492db35d51c244fb4f74 [term 1 LEADER]: Stepping down as leader of term 1
I20260130 02:58:00.490433 19856 raft_consensus.cc:740] T 1bbc55bbf2254d8bb6b721d7e8f69d85 P 78c2c46c7520492db35d51c244fb4f74 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 78c2c46c7520492db35d51c244fb4f74, State: Running, Role: LEADER
I20260130 02:58:00.491143 19856 consensus_queue.cc:260] T 1bbc55bbf2254d8bb6b721d7e8f69d85 P 78c2c46c7520492db35d51c244fb4f74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:00.491989 19856 raft_consensus.cc:3060] T 1bbc55bbf2254d8bb6b721d7e8f69d85 P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Advancing to term 2
I20260130 02:58:00.511943 19840 tablet_bootstrap.cc:492] T b7b596f944be4daf953b82b65d5cec2a P 78c2c46c7520492db35d51c244fb4f74: No bootstrap required, opened a new log
I20260130 02:58:00.512377 19840 ts_tablet_manager.cc:1403] T b7b596f944be4daf953b82b65d5cec2a P 78c2c46c7520492db35d51c244fb4f74: Time spent bootstrapping tablet: real 0.158s user 0.012s sys 0.012s
I20260130 02:58:00.537627 19625 tablet_bootstrap.cc:492] T c7c621b99deb457998c44ab1644d3939 P d870164263bb44b6b780a557daea3656: No bootstrap required, opened a new log
I20260130 02:58:00.538059 19625 ts_tablet_manager.cc:1403] T c7c621b99deb457998c44ab1644d3939 P d870164263bb44b6b780a557daea3656: Time spent bootstrapping tablet: real 0.120s user 0.022s sys 0.014s
I20260130 02:58:00.546175 19807 tablet_bootstrap.cc:492] T c7c621b99deb457998c44ab1644d3939 P 371a36d219164974abdf848841528152: No bootstrap required, opened a new log
I20260130 02:58:00.514417 19840 raft_consensus.cc:359] T b7b596f944be4daf953b82b65d5cec2a P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:00.547113 19840 raft_consensus.cc:385] T b7b596f944be4daf953b82b65d5cec2a P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:00.547024 19807 ts_tablet_manager.cc:1403] T c7c621b99deb457998c44ab1644d3939 P 371a36d219164974abdf848841528152: Time spent bootstrapping tablet: real 0.177s user 0.018s sys 0.004s
I20260130 02:58:00.547372 19840 raft_consensus.cc:740] T b7b596f944be4daf953b82b65d5cec2a P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78c2c46c7520492db35d51c244fb4f74, State: Initialized, Role: FOLLOWER
I20260130 02:58:00.549552 19807 raft_consensus.cc:359] T c7c621b99deb457998c44ab1644d3939 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:00.550316 19807 raft_consensus.cc:385] T c7c621b99deb457998c44ab1644d3939 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:00.550204 19840 consensus_queue.cc:260] T b7b596f944be4daf953b82b65d5cec2a P 78c2c46c7520492db35d51c244fb4f74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:00.553046 19840 ts_tablet_manager.cc:1434] T b7b596f944be4daf953b82b65d5cec2a P 78c2c46c7520492db35d51c244fb4f74: Time spent starting tablet: real 0.040s user 0.006s sys 0.000s
I20260130 02:58:00.553856 19840 tablet_bootstrap.cc:492] T bd43c5d148514836869a8ac648c8ad11 P 78c2c46c7520492db35d51c244fb4f74: Bootstrap starting.
I20260130 02:58:00.563994 19625 raft_consensus.cc:359] T c7c621b99deb457998c44ab1644d3939 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:00.568581 19625 raft_consensus.cc:385] T c7c621b99deb457998c44ab1644d3939 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:00.568595 19807 raft_consensus.cc:740] T c7c621b99deb457998c44ab1644d3939 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 371a36d219164974abdf848841528152, State: Initialized, Role: FOLLOWER
I20260130 02:58:00.568866 19625 raft_consensus.cc:740] T c7c621b99deb457998c44ab1644d3939 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d870164263bb44b6b780a557daea3656, State: Initialized, Role: FOLLOWER
I20260130 02:58:00.577777 19807 consensus_queue.cc:260] T c7c621b99deb457998c44ab1644d3939 P 371a36d219164974abdf848841528152 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:00.579779 19807 ts_tablet_manager.cc:1434] T c7c621b99deb457998c44ab1644d3939 P 371a36d219164974abdf848841528152: Time spent starting tablet: real 0.032s user 0.004s sys 0.000s
I20260130 02:58:00.575134 19625 consensus_queue.cc:260] T c7c621b99deb457998c44ab1644d3939 P d870164263bb44b6b780a557daea3656 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:00.586510 19625 ts_tablet_manager.cc:1434] T c7c621b99deb457998c44ab1644d3939 P d870164263bb44b6b780a557daea3656: Time spent starting tablet: real 0.048s user 0.006s sys 0.001s
I20260130 02:58:00.587391 19625 tablet_bootstrap.cc:492] T 26560f746d244e58a182eb6bb24c8720 P d870164263bb44b6b780a557daea3656: Bootstrap starting.
I20260130 02:58:00.588132 19840 tablet_bootstrap.cc:654] T bd43c5d148514836869a8ac648c8ad11 P 78c2c46c7520492db35d51c244fb4f74: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:00.597994 19625 tablet_bootstrap.cc:654] T 26560f746d244e58a182eb6bb24c8720 P d870164263bb44b6b780a557daea3656: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:00.599225 18870 catalog_manager.cc:5654] T e364f4e70e5749e083e14e7fa7416ff0 P 371a36d219164974abdf848841528152 reported cstate change: term changed from 0 to 1, leader changed from <none> to 371a36d219164974abdf848841528152 (127.18.100.65). New cstate: current_term: 1 leader_uuid: "371a36d219164974abdf848841528152" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } health_report { overall_health: UNKNOWN } } }
I20260130 02:58:00.602233 18869 catalog_manager.cc:5941] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4: Sending DeleteTablet for 3 replicas of tablet 26560f746d244e58a182eb6bb24c8720
I20260130 02:58:00.602897 18870 catalog_manager.cc:5746] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021) does not have the latest schema for tablet 2112b0402de5464cb1caf2c2abb67d78 (table default.Table_da3d581b15a94840898777f74611994e [id=b7e241ced53e4160ac6fa1f65ef3674e]). Expected version 1 got 0
I20260130 02:58:00.603895 19204 tablet_service.cc:1552] Processing DeleteTablet for tablet 26560f746d244e58a182eb6bb24c8720 with delete_type TABLET_DATA_DELETED (replaced by tablet 341041cb0be2456e88403367c0814704) from {username='slave'} at 127.0.0.1:42870
I20260130 02:58:00.605089 18869 catalog_manager.cc:6724] ReplaceTablet: tablet 26560f746d244e58a182eb6bb24c8720 deleted and replaced by tablet 341041cb0be2456e88403367c0814704
I20260130 02:58:00.608134 18871 catalog_manager.cc:5941] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4: Sending DeleteTablet for 3 replicas of tablet bd43c5d148514836869a8ac648c8ad11
I20260130 02:58:00.609360 18871 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:50284 (request call id 114) took 1007 ms. Trace:
I20260130 02:58:00.609620 19204 tablet_service.cc:1552] Processing DeleteTablet for tablet bd43c5d148514836869a8ac648c8ad11 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:58:00 UTC) from {username='slave'} at 127.0.0.1:42870
I20260130 02:58:00.609670 18871 rpcz_store.cc:276] 0130 02:57:59.602353 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:58:00.163640 (+561287us) service_pool.cc:225] Handling call
0130 02:58:00.165211 (+ 1571us) catalog_manager.cc:2393] Looking up, locking, and authorizing table
0130 02:58:00.176078 (+ 10867us) catalog_manager.cc:2815] Modifying in-memory table state
0130 02:58:00.176245 (+ 167us) catalog_manager.cc:2824] Locking tablets
0130 02:58:00.565512 (+389267us) catalog_manager.cc:2838] Removing table and tablets from system table
0130 02:58:00.607791 (+ 42279us) catalog_manager.cc:2856] Removing table from by-name map
0130 02:58:00.607954 (+ 163us) catalog_manager.cc:2872] Committing in-memory state
0130 02:58:00.608058 (+ 104us) catalog_manager.cc:2876] Aborting table tasks
0130 02:58:00.609307 (+ 1249us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0130 02:58:00.567313 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0130 02:58:00.567665 (+ 352us) write_op.cc:432] Acquiring schema lock in shared mode
0130 02:58:00.567687 (+ 22us) write_op.cc:435] Acquired schema lock
0130 02:58:00.567698 (+ 11us) tablet.cc:660] Decoding operations
0130 02:58:00.567980 (+ 282us) write_op.cc:620] Acquiring the partition lock for write op
0130 02:58:00.568014 (+ 34us) write_op.cc:641] Partition lock acquired for write op
0130 02:58:00.568034 (+ 20us) tablet.cc:683] Acquiring locks for 2 operations
0130 02:58:00.568273 (+ 239us) tablet.cc:699] Row locks acquired
0130 02:58:00.568285 (+ 12us) write_op.cc:260] PREPARE: finished
0130 02:58:00.568376 (+ 91us) write_op.cc:270] Start()
0130 02:58:00.568446 (+ 70us) write_op.cc:276] Timestamp: P: 1769741880568319 usec, L: 0
0130 02:58:00.568459 (+ 13us) op_driver.cc:348] REPLICATION: starting
0130 02:58:00.576956 (+ 8497us) log.cc:844] Serialized 762 byte log entry
0130 02:58:00.605211 (+ 28255us) op_driver.cc:464] REPLICATION: finished
0130 02:58:00.605716 (+ 505us) write_op.cc:301] APPLY: starting
0130 02:58:00.605770 (+ 54us) tablet.cc:1365] starting BulkCheckPresence
0130 02:58:00.605866 (+ 96us) tablet.cc:1368] finished BulkCheckPresence
0130 02:58:00.605952 (+ 86us) tablet.cc:1370] starting ApplyRowOperation cycle
0130 02:58:00.606170 (+ 218us) tablet.cc:1381] finished ApplyRowOperation cycle
0130 02:58:00.606309 (+ 139us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0130 02:58:00.606349 (+ 40us) write_op.cc:312] APPLY: finished
0130 02:58:00.606518 (+ 169us) log.cc:844] Serialized 29 byte log entry
0130 02:58:00.606686 (+ 168us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:58:00.606754 (+ 68us) write_op.cc:454] Released schema lock
0130 02:58:00.606786 (+ 32us) write_op.cc:341] FINISH: Updating metrics
0130 02:58:00.607494 (+ 708us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:58:00.607517 (+ 23us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":283,"apply.run_cpu_time_us":1249,"apply.run_wall_time_us":1744,"num_ops":2,"prepare.queue_time_us":93,"prepare.run_cpu_time_us":1903,"prepare.run_wall_time_us":22472,"replication_time_us":36659}]]}
I20260130 02:58:00.613430 18871 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50284:
name: "default.Table_fec49c37fdd045e7801325ace326f367"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260130 02:58:00.616490 19625 tablet_bootstrap.cc:492] T 26560f746d244e58a182eb6bb24c8720 P d870164263bb44b6b780a557daea3656: No bootstrap required, opened a new log
I20260130 02:58:00.616863 19625 ts_tablet_manager.cc:1403] T 26560f746d244e58a182eb6bb24c8720 P d870164263bb44b6b780a557daea3656: Time spent bootstrapping tablet: real 0.030s user 0.005s sys 0.003s
I20260130 02:58:00.617653 19861 consensus_queue.cc:1048] T 0b1877e5f9244913bcd639f1336df7ee P 371a36d219164974abdf848841528152 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260130 02:58:00.620549 19625 raft_consensus.cc:359] T 26560f746d244e58a182eb6bb24c8720 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:00.621147 19625 raft_consensus.cc:385] T 26560f746d244e58a182eb6bb24c8720 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:00.621382 19625 raft_consensus.cc:740] T 26560f746d244e58a182eb6bb24c8720 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d870164263bb44b6b780a557daea3656, State: Initialized, Role: FOLLOWER
I20260130 02:58:00.622253 19625 consensus_queue.cc:260] T 26560f746d244e58a182eb6bb24c8720 P d870164263bb44b6b780a557daea3656 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:00.625458 19899 raft_consensus.cc:493] T b7b596f944be4daf953b82b65d5cec2a P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:00.625910 19899 raft_consensus.cc:515] T b7b596f944be4daf953b82b65d5cec2a P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:00.626996 19625 ts_tablet_manager.cc:1434] T 26560f746d244e58a182eb6bb24c8720 P d870164263bb44b6b780a557daea3656: Time spent starting tablet: real 0.010s user 0.005s sys 0.002s
I20260130 02:58:00.630049 19625 tablet_bootstrap.cc:492] T 1ca34772ef484c1f809a99d126e28eff P d870164263bb44b6b780a557daea3656: Bootstrap starting.
W20260130 02:58:00.632428 18871 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fec49c37fdd045e7801325ace326f367 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260130 02:58:00.636818 19507 tablet_service.cc:1552] Processing DeleteTablet for tablet 26560f746d244e58a182eb6bb24c8720 with delete_type TABLET_DATA_DELETED (replaced by tablet 341041cb0be2456e88403367c0814704) from {username='slave'} at 127.0.0.1:59068
I20260130 02:58:00.637527 19947 tablet_replica.cc:333] T 26560f746d244e58a182eb6bb24c8720 P d870164263bb44b6b780a557daea3656: stopping tablet replica
I20260130 02:58:00.638418 19947 raft_consensus.cc:2243] T 26560f746d244e58a182eb6bb24c8720 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260130 02:58:00.638928 19947 raft_consensus.cc:2272] T 26560f746d244e58a182eb6bb24c8720 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260130 02:58:00.640125 19507 tablet_service.cc:1552] Processing DeleteTablet for tablet bd43c5d148514836869a8ac648c8ad11 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:58:00 UTC) from {username='slave'} at 127.0.0.1:59068
I20260130 02:58:00.640115 19861 consensus_queue.cc:1048] T e364f4e70e5749e083e14e7fa7416ff0 P 371a36d219164974abdf848841528152 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260130 02:58:00.642818 19947 ts_tablet_manager.cc:1916] T 26560f746d244e58a182eb6bb24c8720 P d870164263bb44b6b780a557daea3656: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260130 02:58:00.644450 19360 tablet_service.cc:1552] Processing DeleteTablet for tablet 26560f746d244e58a182eb6bb24c8720 with delete_type TABLET_DATA_DELETED (replaced by tablet 341041cb0be2456e88403367c0814704) from {username='slave'} at 127.0.0.1:60110
I20260130 02:58:00.645206 19354 tablet_service.cc:1552] Processing DeleteTablet for tablet bd43c5d148514836869a8ac648c8ad11 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:58:00 UTC) from {username='slave'} at 127.0.0.1:60110
I20260130 02:58:00.646035 19758 tablet_replica.cc:333] T 26560f746d244e58a182eb6bb24c8720 P 78c2c46c7520492db35d51c244fb4f74: stopping tablet replica
I20260130 02:58:00.646862 19758 raft_consensus.cc:2243] T 26560f746d244e58a182eb6bb24c8720 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260130 02:58:00.647289 19758 raft_consensus.cc:2272] T 26560f746d244e58a182eb6bb24c8720 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260130 02:58:00.648984 19758 ts_tablet_manager.cc:1916] T 26560f746d244e58a182eb6bb24c8720 P 78c2c46c7520492db35d51c244fb4f74: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260130 02:58:00.659106 19201 tablet_service.cc:1505] Processing CreateTablet for tablet b3f3177306914e3c8b253db54ee30fb6 (DEFAULT_TABLE table=default.Table_87dbbd65d92043b0b91c801517e56dfe [id=adced31928ba4204b79e3b43f8b737ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:00.660343 19204 tablet_service.cc:1505] Processing CreateTablet for tablet d3ba2df0b15c44f882070570f91b254d (DEFAULT_TABLE table=default.Table_92836e7fcf3a48d8bac2160c2ff7c482 [id=e3ae4b66fa0048668cb1e4d9da7d16fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:00.661152 19507 tablet_service.cc:1505] Processing CreateTablet for tablet d3ba2df0b15c44f882070570f91b254d (DEFAULT_TABLE table=default.Table_92836e7fcf3a48d8bac2160c2ff7c482 [id=e3ae4b66fa0048668cb1e4d9da7d16fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:00.661749 19354 tablet_service.cc:1505] Processing CreateTablet for tablet d3ba2df0b15c44f882070570f91b254d (DEFAULT_TABLE table=default.Table_92836e7fcf3a48d8bac2160c2ff7c482 [id=e3ae4b66fa0048668cb1e4d9da7d16fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:00.662206 19507 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d3ba2df0b15c44f882070570f91b254d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:00.663661 19354 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d3ba2df0b15c44f882070570f91b254d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:00.665167 19505 tablet_service.cc:1505] Processing CreateTablet for tablet b3f3177306914e3c8b253db54ee30fb6 (DEFAULT_TABLE table=default.Table_87dbbd65d92043b0b91c801517e56dfe [id=adced31928ba4204b79e3b43f8b737ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:00.666271 19505 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b3f3177306914e3c8b253db54ee30fb6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:00.665973 19861 consensus_queue.cc:1048] T 0b1877e5f9244913bcd639f1336df7ee P 371a36d219164974abdf848841528152 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260130 02:58:00.668002 19525 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b7b596f944be4daf953b82b65d5cec2a" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
W20260130 02:58:00.669317 19313 leader_election.cc:343] T b7b596f944be4daf953b82b65d5cec2a P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:00.672024 19899 leader_election.cc:290] T b7b596f944be4daf953b82b65d5cec2a P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), d870164263bb44b6b780a557daea3656 (127.18.100.67:39571)
I20260130 02:58:00.681757 19360 tablet_service.cc:1505] Processing CreateTablet for tablet b3f3177306914e3c8b253db54ee30fb6 (DEFAULT_TABLE table=default.Table_87dbbd65d92043b0b91c801517e56dfe [id=adced31928ba4204b79e3b43f8b737ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:00.682842 19360 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b3f3177306914e3c8b253db54ee30fb6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:00.690969 19742 raft_consensus.cc:493] T 5b2976108584478a98fd90fe6594af66 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:00.691447 19742 raft_consensus.cc:515] T 5b2976108584478a98fd90fe6594af66 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:00.693325 19221 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b7b596f944be4daf953b82b65d5cec2a" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152" is_pre_election: true
I20260130 02:58:00.693464 19742 leader_election.cc:290] T 5b2976108584478a98fd90fe6594af66 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767)
I20260130 02:58:00.694890 19377 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5b2976108584478a98fd90fe6594af66" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74" is_pre_election: true
I20260130 02:58:00.695526 19377 raft_consensus.cc:2468] T 5b2976108584478a98fd90fe6594af66 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d870164263bb44b6b780a557daea3656 in term 0.
I20260130 02:58:00.695719 19201 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b3f3177306914e3c8b253db54ee30fb6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:00.696825 19462 leader_election.cc:304] T 5b2976108584478a98fd90fe6594af66 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78c2c46c7520492db35d51c244fb4f74, d870164263bb44b6b780a557daea3656; no voters:
I20260130 02:58:00.697911 19742 raft_consensus.cc:2804] T 5b2976108584478a98fd90fe6594af66 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260130 02:58:00.698261 19742 raft_consensus.cc:493] T 5b2976108584478a98fd90fe6594af66 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260130 02:58:00.698755 19754 tablet_replica.cc:333] T 26560f746d244e58a182eb6bb24c8720 P 371a36d219164974abdf848841528152: stopping tablet replica
I20260130 02:58:00.698709 19742 raft_consensus.cc:3060] T 5b2976108584478a98fd90fe6594af66 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:00.699486 19754 raft_consensus.cc:2243] T 26560f746d244e58a182eb6bb24c8720 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260130 02:58:00.699913 19754 raft_consensus.cc:2272] T 26560f746d244e58a182eb6bb24c8720 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260130 02:58:00.701012 19204 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d3ba2df0b15c44f882070570f91b254d. 1 dirs total, 0 dirs full, 0 dirs failed
W20260130 02:58:00.707724 19314 leader_election.cc:343] T b7b596f944be4daf953b82b65d5cec2a P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:00.708166 19314 leader_election.cc:304] T b7b596f944be4daf953b82b65d5cec2a P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78c2c46c7520492db35d51c244fb4f74; no voters: 371a36d219164974abdf848841528152, d870164263bb44b6b780a557daea3656
I20260130 02:58:00.663004 18870 catalog_manager.cc:5654] T 0b1877e5f9244913bcd639f1336df7ee P 371a36d219164974abdf848841528152 reported cstate change: term changed from 0 to 1, leader changed from <none> to 371a36d219164974abdf848841528152 (127.18.100.65). New cstate: current_term: 1 leader_uuid: "371a36d219164974abdf848841528152" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } health_report { overall_health: UNKNOWN } } }
I20260130 02:58:00.709419 19224 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5b2976108584478a98fd90fe6594af66" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152" is_pre_election: true
I20260130 02:58:00.710003 19224 raft_consensus.cc:2468] T 5b2976108584478a98fd90fe6594af66 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d870164263bb44b6b780a557daea3656 in term 0.
I20260130 02:58:00.712152 19754 ts_tablet_manager.cc:1916] T 26560f746d244e58a182eb6bb24c8720 P 371a36d219164974abdf848841528152: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260130 02:58:00.715497 19625 tablet_bootstrap.cc:654] T 1ca34772ef484c1f809a99d126e28eff P d870164263bb44b6b780a557daea3656: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:00.717788 19742 raft_consensus.cc:515] T 5b2976108584478a98fd90fe6594af66 P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:00.721081 19377 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5b2976108584478a98fd90fe6594af66" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74"
I20260130 02:58:00.721786 19377 raft_consensus.cc:3060] T 5b2976108584478a98fd90fe6594af66 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:00.723141 19203 tablet_service.cc:1505] Processing CreateTablet for tablet ec28ff12927d450ead16d26cf016138d (DEFAULT_TABLE table=default.Table_0480357296ac4a27812d802ee4566351 [id=0da6197ab32e4645aa8d8787662ee5e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:00.723583 18871 master_service.cc:954] ReplaceTablet: received request to replace tablet 073b5941850c4140b1751956dd7cf5ef from {username='slave'} at 127.0.0.1:50284
I20260130 02:58:00.724195 19203 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ec28ff12927d450ead16d26cf016138d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:00.725950 19663 raft_consensus.cc:2749] T b7b596f944be4daf953b82b65d5cec2a P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260130 02:58:00.728674 19377 raft_consensus.cc:2468] T 5b2976108584478a98fd90fe6594af66 P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d870164263bb44b6b780a557daea3656 in term 1.
I20260130 02:58:00.730228 19224 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5b2976108584478a98fd90fe6594af66" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152"
I20260130 02:58:00.730355 19462 leader_election.cc:304] T 5b2976108584478a98fd90fe6594af66 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78c2c46c7520492db35d51c244fb4f74, d870164263bb44b6b780a557daea3656; no voters:
I20260130 02:58:00.730769 19224 raft_consensus.cc:3060] T 5b2976108584478a98fd90fe6594af66 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:00.731420 19663 consensus_queue.cc:1048] T c3fce8cc99fb451ebd532fa37390e862 P 78c2c46c7520492db35d51c244fb4f74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260130 02:58:00.734566 19948 raft_consensus.cc:2804] T 5b2976108584478a98fd90fe6594af66 P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Leader election won for term 1
I20260130 02:58:00.743011 19742 leader_election.cc:290] T 5b2976108584478a98fd90fe6594af66 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 election: Requested vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767)
I20260130 02:58:00.748185 19947 ts_tablet_manager.cc:1929] T 26560f746d244e58a182eb6bb24c8720 P d870164263bb44b6b780a557daea3656: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260130 02:58:00.757156 19625 tablet_bootstrap.cc:492] T 1ca34772ef484c1f809a99d126e28eff P d870164263bb44b6b780a557daea3656: No bootstrap required, opened a new log
I20260130 02:58:00.757709 19625 ts_tablet_manager.cc:1403] T 1ca34772ef484c1f809a99d126e28eff P d870164263bb44b6b780a557daea3656: Time spent bootstrapping tablet: real 0.130s user 0.006s sys 0.006s
I20260130 02:58:00.755663 19947 log.cc:1199] T 26560f746d244e58a182eb6bb24c8720 P d870164263bb44b6b780a557daea3656: Deleting WAL directory at /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-2/wal/wals/26560f746d244e58a182eb6bb24c8720
I20260130 02:58:00.760538 19947 ts_tablet_manager.cc:1950] T 26560f746d244e58a182eb6bb24c8720 P d870164263bb44b6b780a557daea3656: Deleting consensus metadata
I20260130 02:58:00.761633 19224 raft_consensus.cc:2468] T 5b2976108584478a98fd90fe6594af66 P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d870164263bb44b6b780a557daea3656 in term 1.
I20260130 02:58:00.761557 19625 raft_consensus.cc:359] T 1ca34772ef484c1f809a99d126e28eff P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:00.763257 19625 raft_consensus.cc:385] T 1ca34772ef484c1f809a99d126e28eff P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:00.763556 19625 raft_consensus.cc:740] T 1ca34772ef484c1f809a99d126e28eff P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d870164263bb44b6b780a557daea3656, State: Initialized, Role: FOLLOWER
I20260130 02:58:00.765650 19625 consensus_queue.cc:260] T 1ca34772ef484c1f809a99d126e28eff P d870164263bb44b6b780a557daea3656 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:00.769470 18860 catalog_manager.cc:4985] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): tablet 26560f746d244e58a182eb6bb24c8720 (table default.Table_b2d2da7aae8b430aba32bac2944aa931 [id=8d76f4328edc491291491c78ba205d30]) successfully deleted
I20260130 02:58:00.777799 19625 ts_tablet_manager.cc:1434] T 1ca34772ef484c1f809a99d126e28eff P d870164263bb44b6b780a557daea3656: Time spent starting tablet: real 0.020s user 0.005s sys 0.001s
I20260130 02:58:00.778788 19625 tablet_bootstrap.cc:492] T bd43c5d148514836869a8ac648c8ad11 P d870164263bb44b6b780a557daea3656: Bootstrap starting.
I20260130 02:58:00.788100 19625 tablet_bootstrap.cc:654] T bd43c5d148514836869a8ac648c8ad11 P d870164263bb44b6b780a557daea3656: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:00.778735 19807 tablet_bootstrap.cc:492] T b7b596f944be4daf953b82b65d5cec2a P 371a36d219164974abdf848841528152: Bootstrap starting.
I20260130 02:58:00.797533 19505 tablet_service.cc:1505] Processing CreateTablet for tablet ec28ff12927d450ead16d26cf016138d (DEFAULT_TABLE table=default.Table_0480357296ac4a27812d802ee4566351 [id=0da6197ab32e4645aa8d8787662ee5e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:00.798794 19505 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ec28ff12927d450ead16d26cf016138d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:00.845609 19625 tablet_bootstrap.cc:492] T bd43c5d148514836869a8ac648c8ad11 P d870164263bb44b6b780a557daea3656: No bootstrap required, opened a new log
I20260130 02:58:00.846035 19625 ts_tablet_manager.cc:1403] T bd43c5d148514836869a8ac648c8ad11 P d870164263bb44b6b780a557daea3656: Time spent bootstrapping tablet: real 0.067s user 0.011s sys 0.002s
I20260130 02:58:00.848290 19625 raft_consensus.cc:359] T bd43c5d148514836869a8ac648c8ad11 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:00.848892 19625 raft_consensus.cc:385] T bd43c5d148514836869a8ac648c8ad11 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:00.852468 19625 raft_consensus.cc:740] T bd43c5d148514836869a8ac648c8ad11 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d870164263bb44b6b780a557daea3656, State: Initialized, Role: FOLLOWER
I20260130 02:58:00.854425 19625 consensus_queue.cc:260] T bd43c5d148514836869a8ac648c8ad11 P d870164263bb44b6b780a557daea3656 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:00.865090 19625 ts_tablet_manager.cc:1434] T bd43c5d148514836869a8ac648c8ad11 P d870164263bb44b6b780a557daea3656: Time spent starting tablet: real 0.019s user 0.007s sys 0.000s
I20260130 02:58:00.866809 19625 tablet_bootstrap.cc:492] T 955b9547f49f434699fe1d86b260638c P d870164263bb44b6b780a557daea3656: Bootstrap starting.
I20260130 02:58:00.871901 19856 consensus_queue.cc:1048] T c3fce8cc99fb451ebd532fa37390e862 P 78c2c46c7520492db35d51c244fb4f74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260130 02:58:00.877177 19625 tablet_bootstrap.cc:654] T 955b9547f49f434699fe1d86b260638c P d870164263bb44b6b780a557daea3656: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:00.892526 19625 tablet_bootstrap.cc:492] T 955b9547f49f434699fe1d86b260638c P d870164263bb44b6b780a557daea3656: No bootstrap required, opened a new log
I20260130 02:58:00.892923 19625 ts_tablet_manager.cc:1403] T 955b9547f49f434699fe1d86b260638c P d870164263bb44b6b780a557daea3656: Time spent bootstrapping tablet: real 0.026s user 0.001s sys 0.008s
I20260130 02:58:00.895792 19625 raft_consensus.cc:359] T 955b9547f49f434699fe1d86b260638c P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:00.896413 19625 raft_consensus.cc:385] T 955b9547f49f434699fe1d86b260638c P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:00.896647 19625 raft_consensus.cc:740] T 955b9547f49f434699fe1d86b260638c P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d870164263bb44b6b780a557daea3656, State: Initialized, Role: FOLLOWER
I20260130 02:58:00.897195 19625 consensus_queue.cc:260] T 955b9547f49f434699fe1d86b260638c P d870164263bb44b6b780a557daea3656 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:00.899243 19625 ts_tablet_manager.cc:1434] T 955b9547f49f434699fe1d86b260638c P d870164263bb44b6b780a557daea3656: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260130 02:58:00.899802 19948 raft_consensus.cc:697] T 5b2976108584478a98fd90fe6594af66 P d870164263bb44b6b780a557daea3656 [term 1 LEADER]: Becoming Leader. State: Replica: d870164263bb44b6b780a557daea3656, State: Running, Role: LEADER
I20260130 02:58:00.901099 19625 tablet_bootstrap.cc:492] T b7b596f944be4daf953b82b65d5cec2a P d870164263bb44b6b780a557daea3656: Bootstrap starting.
I20260130 02:58:00.901100 19948 consensus_queue.cc:237] T 5b2976108584478a98fd90fe6594af66 P d870164263bb44b6b780a557daea3656 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:00.908210 19625 tablet_bootstrap.cc:654] T b7b596f944be4daf953b82b65d5cec2a P d870164263bb44b6b780a557daea3656: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:00.915335 19899 raft_consensus.cc:493] T c7c621b99deb457998c44ab1644d3939 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:00.915865 19899 raft_consensus.cc:515] T c7c621b99deb457998c44ab1644d3939 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
W20260130 02:58:00.917910 19159 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.100.67:39571, user_credentials={real_user=slave}} blocked reactor thread for 91586.2us
I20260130 02:58:00.918922 19221 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c7c621b99deb457998c44ab1644d3939" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152" is_pre_election: true
I20260130 02:58:00.919524 19221 raft_consensus.cc:2468] T c7c621b99deb457998c44ab1644d3939 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78c2c46c7520492db35d51c244fb4f74 in term 0.
I20260130 02:58:00.921919 19314 leader_election.cc:304] T c7c621b99deb457998c44ab1644d3939 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 371a36d219164974abdf848841528152, 78c2c46c7520492db35d51c244fb4f74; no voters:
I20260130 02:58:00.922631 19899 leader_election.cc:290] T c7c621b99deb457998c44ab1644d3939 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), d870164263bb44b6b780a557daea3656 (127.18.100.67:39571)
I20260130 02:58:00.923439 19899 raft_consensus.cc:2804] T c7c621b99deb457998c44ab1644d3939 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260130 02:58:00.923837 19899 raft_consensus.cc:493] T c7c621b99deb457998c44ab1644d3939 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260130 02:58:00.924242 19899 raft_consensus.cc:3060] T c7c621b99deb457998c44ab1644d3939 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:00.934414 19525 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c7c621b99deb457998c44ab1644d3939" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
I20260130 02:58:00.936055 19899 raft_consensus.cc:515] T c7c621b99deb457998c44ab1644d3939 P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
W20260130 02:58:00.938010 18860 catalog_manager.cc:4965] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): delete failed for tablet bd43c5d148514836869a8ac648c8ad11 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bd43c5d148514836869a8ac648c8ad11 already in progress: creating tablet
W20260130 02:58:00.939311 18860 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet 15918c90ffdc43a5a38b2950e9834188 (table default.Table_333d2efbf068485a871f566d50e563ce [id=8b7db40afb894ae99625a756c75ef1de], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 15918c90ffdc43a5a38b2950e9834188 consensus state has no leader
I20260130 02:58:00.936672 19840 tablet_bootstrap.cc:492] T bd43c5d148514836869a8ac648c8ad11 P 78c2c46c7520492db35d51c244fb4f74: No bootstrap required, opened a new log
I20260130 02:58:00.941639 19840 ts_tablet_manager.cc:1403] T bd43c5d148514836869a8ac648c8ad11 P 78c2c46c7520492db35d51c244fb4f74: Time spent bootstrapping tablet: real 0.388s user 0.117s sys 0.081s
I20260130 02:58:00.944535 19840 raft_consensus.cc:359] T bd43c5d148514836869a8ac648c8ad11 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:00.944828 19899 leader_election.cc:290] T c7c621b99deb457998c44ab1644d3939 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 election: Requested vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), d870164263bb44b6b780a557daea3656 (127.18.100.67:39571)
I20260130 02:58:00.945154 19840 raft_consensus.cc:385] T bd43c5d148514836869a8ac648c8ad11 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:00.945394 19840 raft_consensus.cc:740] T bd43c5d148514836869a8ac648c8ad11 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78c2c46c7520492db35d51c244fb4f74, State: Initialized, Role: FOLLOWER
I20260130 02:58:00.946022 19758 ts_tablet_manager.cc:1929] T 26560f746d244e58a182eb6bb24c8720 P 78c2c46c7520492db35d51c244fb4f74: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260130 02:58:00.946756 19224 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c7c621b99deb457998c44ab1644d3939" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152"
I20260130 02:58:00.947348 19224 raft_consensus.cc:3060] T c7c621b99deb457998c44ab1644d3939 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:00.965978 19200 tablet_service.cc:1505] Processing CreateTablet for tablet 5ca9919f80954e74a4d91c94b74de0e4 (DEFAULT_TABLE table=default.Table_9d0cd9756483406081c8bdb18565a42f [id=5f6488df8ba94ef293b181328cc3b680]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:00.967209 19200 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5ca9919f80954e74a4d91c94b74de0e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:00.971999 19199 tablet_service.cc:1505] Processing CreateTablet for tablet 481bd4cca6ee4c178873bf12392e3441 (DEFAULT_TABLE table=default.Table_e3b63d7d5d874900aac850654d98d62e [id=bc8cc55916fa4a41b78019f59e6fbe39]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:00.973122 19199 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 481bd4cca6ee4c178873bf12392e3441. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:00.976816 18873 catalog_manager.cc:5746] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767) does not have the latest schema for tablet 2112b0402de5464cb1caf2c2abb67d78 (table default.Table_da3d581b15a94840898777f74611994e [id=b7e241ced53e4160ac6fa1f65ef3674e]). Expected version 1 got 0
I20260130 02:58:00.977662 18873 catalog_manager.cc:5654] T 1bbc55bbf2254d8bb6b721d7e8f69d85 P 78c2c46c7520492db35d51c244fb4f74 reported cstate change: term changed from 0 to 1, leader changed from <none> to 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66). New cstate: current_term: 1 leader_uuid: "78c2c46c7520492db35d51c244fb4f74" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } health_report { overall_health: HEALTHY } } }
I20260130 02:58:00.980770 18873 catalog_manager.cc:5654] T ffa1142fc7274aecbdbee1a0802b95da P 78c2c46c7520492db35d51c244fb4f74 reported cstate change: term changed from 0 to 1, leader changed from <none> to 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66). New cstate: current_term: 1 leader_uuid: "78c2c46c7520492db35d51c244fb4f74" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } health_report { overall_health: HEALTHY } } }
I20260130 02:58:00.985145 19807 tablet_bootstrap.cc:654] T b7b596f944be4daf953b82b65d5cec2a P 371a36d219164974abdf848841528152: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:00.941102 19625 tablet_bootstrap.cc:492] T b7b596f944be4daf953b82b65d5cec2a P d870164263bb44b6b780a557daea3656: No bootstrap required, opened a new log
I20260130 02:58:00.956055 19526 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c7c621b99deb457998c44ab1644d3939" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656"
I20260130 02:58:00.997551 19625 ts_tablet_manager.cc:1403] T b7b596f944be4daf953b82b65d5cec2a P d870164263bb44b6b780a557daea3656: Time spent bootstrapping tablet: real 0.097s user 0.005s sys 0.012s
I20260130 02:58:00.989596 19202 tablet_service.cc:1505] Processing CreateTablet for tablet 7b80b22bd44c4233837c951aa97e6996 (DEFAULT_TABLE table=default.Table_7d1c27ab3515406ba5b5333c90df093e [id=b6fabeeae79941dc8094d0f2b51704f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:00.960307 19505 tablet_service.cc:1505] Processing CreateTablet for tablet 2ca0df71836145748cada4e4559a5656 (DEFAULT_TABLE table=default.Table_d764cbaecfee476ba5e734dd1e2f50fd [id=3f9236f0631a4e4c94ca7456f0478a45]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:01.005782 18873 catalog_manager.cc:5746] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767) does not have the latest schema for tablet 37470ce4356f4542b621e21698551209 (table default.Table_007e969f70d345a7bcb3dfbaabbc025d [id=77d4fa8705a3419cb0a84d3ca1bb4ce3]). Expected version 1 got 0
I20260130 02:58:00.993641 19754 ts_tablet_manager.cc:1929] T 26560f746d244e58a182eb6bb24c8720 P 371a36d219164974abdf848841528152: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260130 02:58:01.023296 19754 log.cc:1199] T 26560f746d244e58a182eb6bb24c8720 P 371a36d219164974abdf848841528152: Deleting WAL directory at /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-0/wal/wals/26560f746d244e58a182eb6bb24c8720
I20260130 02:58:01.023972 19754 ts_tablet_manager.cc:1950] T 26560f746d244e58a182eb6bb24c8720 P 371a36d219164974abdf848841528152: Deleting consensus metadata
I20260130 02:58:01.026921 19204 tablet_service.cc:1505] Processing CreateTablet for tablet 7d76046ee6c8420fa4964decbe8209c1 (DEFAULT_TABLE table=default.Table_e37f8d618ff244a6af90cc83a9035328 [id=4c70005d1005479fa6c3a6cb6f5ce658]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:00.946019 19840 consensus_queue.cc:260] T bd43c5d148514836869a8ac648c8ad11 P 78c2c46c7520492db35d51c244fb4f74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:00.960388 19360 tablet_service.cc:1505] Processing CreateTablet for tablet ec28ff12927d450ead16d26cf016138d (DEFAULT_TABLE table=default.Table_0480357296ac4a27812d802ee4566351 [id=0da6197ab32e4645aa8d8787662ee5e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:01.037189 19360 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ec28ff12927d450ead16d26cf016138d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:01.037988 19840 ts_tablet_manager.cc:1434] T bd43c5d148514836869a8ac648c8ad11 P 78c2c46c7520492db35d51c244fb4f74: Time spent starting tablet: real 0.096s user 0.005s sys 0.000s
I20260130 02:58:01.038829 19840 tablet_bootstrap.cc:492] T 955b9547f49f434699fe1d86b260638c P 78c2c46c7520492db35d51c244fb4f74: Bootstrap starting.
I20260130 02:58:00.972268 19504 tablet_service.cc:1505] Processing CreateTablet for tablet 5ca9919f80954e74a4d91c94b74de0e4 (DEFAULT_TABLE table=default.Table_9d0cd9756483406081c8bdb18565a42f [id=5f6488df8ba94ef293b181328cc3b680]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:00.973237 19508 tablet_service.cc:1505] Processing CreateTablet for tablet 481bd4cca6ee4c178873bf12392e3441 (DEFAULT_TABLE table=default.Table_e3b63d7d5d874900aac850654d98d62e [id=bc8cc55916fa4a41b78019f59e6fbe39]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:00.975625 19507 tablet_service.cc:1505] Processing CreateTablet for tablet 7d76046ee6c8420fa4964decbe8209c1 (DEFAULT_TABLE table=default.Table_e37f8d618ff244a6af90cc83a9035328 [id=4c70005d1005479fa6c3a6cb6f5ce658]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:01.000320 19625 raft_consensus.cc:359] T b7b596f944be4daf953b82b65d5cec2a P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:01.003718 19503 tablet_service.cc:1505] Processing CreateTablet for tablet 7b80b22bd44c4233837c951aa97e6996 (DEFAULT_TABLE table=default.Table_7d1c27ab3515406ba5b5333c90df093e [id=b6fabeeae79941dc8094d0f2b51704f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:01.039959 19625 raft_consensus.cc:385] T b7b596f944be4daf953b82b65d5cec2a P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:00.961583 19356 tablet_service.cc:1505] Processing CreateTablet for tablet 2ca0df71836145748cada4e4559a5656 (DEFAULT_TABLE table=default.Table_d764cbaecfee476ba5e734dd1e2f50fd [id=3f9236f0631a4e4c94ca7456f0478a45]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:01.040244 19625 raft_consensus.cc:740] T b7b596f944be4daf953b82b65d5cec2a P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d870164263bb44b6b780a557daea3656, State: Initialized, Role: FOLLOWER
I20260130 02:58:01.041114 19356 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2ca0df71836145748cada4e4559a5656. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:01.040845 19625 consensus_queue.cc:260] T b7b596f944be4daf953b82b65d5cec2a P d870164263bb44b6b780a557daea3656 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:00.997502 19525 raft_consensus.cc:2468] T c7c621b99deb457998c44ab1644d3939 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78c2c46c7520492db35d51c244fb4f74 in term 0.
I20260130 02:58:01.042840 19625 ts_tablet_manager.cc:1434] T b7b596f944be4daf953b82b65d5cec2a P d870164263bb44b6b780a557daea3656: Time spent starting tablet: real 0.045s user 0.002s sys 0.004s
I20260130 02:58:01.044744 19840 tablet_bootstrap.cc:654] T 955b9547f49f434699fe1d86b260638c P 78c2c46c7520492db35d51c244fb4f74: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:00.979329 19357 tablet_service.cc:1505] Processing CreateTablet for tablet 7d76046ee6c8420fa4964decbe8209c1 (DEFAULT_TABLE table=default.Table_e37f8d618ff244a6af90cc83a9035328 [id=4c70005d1005479fa6c3a6cb6f5ce658]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:01.046265 19357 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7d76046ee6c8420fa4964decbe8209c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:00.980732 19353 tablet_service.cc:1505] Processing CreateTablet for tablet 5ca9919f80954e74a4d91c94b74de0e4 (DEFAULT_TABLE table=default.Table_9d0cd9756483406081c8bdb18565a42f [id=5f6488df8ba94ef293b181328cc3b680]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:01.051306 19353 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5ca9919f80954e74a4d91c94b74de0e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:01.053894 19504 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5ca9919f80954e74a4d91c94b74de0e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:00.987885 19358 tablet_service.cc:1505] Processing CreateTablet for tablet 481bd4cca6ee4c178873bf12392e3441 (DEFAULT_TABLE table=default.Table_e3b63d7d5d874900aac850654d98d62e [id=bc8cc55916fa4a41b78019f59e6fbe39]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:01.008598 19352 tablet_service.cc:1505] Processing CreateTablet for tablet 7b80b22bd44c4233837c951aa97e6996 (DEFAULT_TABLE table=default.Table_7d1c27ab3515406ba5b5333c90df093e [id=b6fabeeae79941dc8094d0f2b51704f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:01.027468 19355 tablet_service.cc:1505] Processing CreateTablet for tablet e47b22aafe5247c4b62c4193c24f427c (DEFAULT_TABLE table=default.Table_d11b53ac041740d8818edf7d2861dc6d [id=edeb74c78d234ee6a5a031c8347d3884]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:01.055295 19358 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 481bd4cca6ee4c178873bf12392e3441. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:01.055794 19355 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e47b22aafe5247c4b62c4193c24f427c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:01.059042 19352 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7b80b22bd44c4233837c951aa97e6996. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:01.059211 19508 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 481bd4cca6ee4c178873bf12392e3441. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:01.063071 19507 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7d76046ee6c8420fa4964decbe8209c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:01.067215 19503 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7b80b22bd44c4233837c951aa97e6996. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:01.067531 19625 tablet_bootstrap.cc:492] T a743a82540014c85aeab0e9d0994b0dc P d870164263bb44b6b780a557daea3656: Bootstrap starting.
I20260130 02:58:01.071040 19505 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2ca0df71836145748cada4e4559a5656. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:01.072773 19625 tablet_bootstrap.cc:654] T a743a82540014c85aeab0e9d0994b0dc P d870164263bb44b6b780a557daea3656: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:01.022624 19948 raft_consensus.cc:493] T d701152960b14396a6ac1bd10480d2b6 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260130 02:58:01.087210 19872 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
I20260130 02:58:01.018937 19502 tablet_service.cc:1505] Processing CreateTablet for tablet e47b22aafe5247c4b62c4193c24f427c (DEFAULT_TABLE table=default.Table_d11b53ac041740d8818edf7d2861dc6d [id=edeb74c78d234ee6a5a031c8347d3884]), partition=RANGE (key) PARTITION UNBOUNDED
W20260130 02:58:01.089160 19872 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
I20260130 02:58:01.089475 19502 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e47b22aafe5247c4b62c4193c24f427c. 1 dirs total, 0 dirs full, 0 dirs failed
W20260130 02:58:01.089855 19872 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260130 02:58:01.091957 19625 tablet_bootstrap.cc:492] T a743a82540014c85aeab0e9d0994b0dc P d870164263bb44b6b780a557daea3656: No bootstrap required, opened a new log
I20260130 02:58:01.092370 19625 ts_tablet_manager.cc:1403] T a743a82540014c85aeab0e9d0994b0dc P d870164263bb44b6b780a557daea3656: Time spent bootstrapping tablet: real 0.025s user 0.007s sys 0.003s
I20260130 02:58:01.094501 19625 raft_consensus.cc:359] T a743a82540014c85aeab0e9d0994b0dc P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:01.101441 19625 raft_consensus.cc:385] T a743a82540014c85aeab0e9d0994b0dc P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:01.101960 19625 raft_consensus.cc:740] T a743a82540014c85aeab0e9d0994b0dc P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d870164263bb44b6b780a557daea3656, State: Initialized, Role: FOLLOWER
I20260130 02:58:01.108068 19625 consensus_queue.cc:260] T a743a82540014c85aeab0e9d0994b0dc P d870164263bb44b6b780a557daea3656 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:01.110059 19625 ts_tablet_manager.cc:1434] T a743a82540014c85aeab0e9d0994b0dc P d870164263bb44b6b780a557daea3656: Time spent starting tablet: real 0.017s user 0.005s sys 0.000s
I20260130 02:58:01.111646 19625 tablet_bootstrap.cc:492] T d3ba2df0b15c44f882070570f91b254d P d870164263bb44b6b780a557daea3656: Bootstrap starting.
I20260130 02:58:01.089039 19948 raft_consensus.cc:515] T d701152960b14396a6ac1bd10480d2b6 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:01.122334 19840 tablet_bootstrap.cc:492] T 955b9547f49f434699fe1d86b260638c P 78c2c46c7520492db35d51c244fb4f74: No bootstrap required, opened a new log
I20260130 02:58:01.122838 19840 ts_tablet_manager.cc:1403] T 955b9547f49f434699fe1d86b260638c P 78c2c46c7520492db35d51c244fb4f74: Time spent bootstrapping tablet: real 0.084s user 0.011s sys 0.000s
I20260130 02:58:01.125001 19840 raft_consensus.cc:359] T 955b9547f49f434699fe1d86b260638c P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:01.126302 19948 leader_election.cc:290] T d701152960b14396a6ac1bd10480d2b6 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767)
I20260130 02:58:01.113900 18869 catalog_manager.cc:5654] T 1bbc55bbf2254d8bb6b721d7e8f69d85 P d870164263bb44b6b780a557daea3656 reported cstate change: term changed from 1 to 2, leader changed from 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66) to d870164263bb44b6b780a557daea3656 (127.18.100.67). New cstate: current_term: 2 leader_uuid: "d870164263bb44b6b780a557daea3656" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } health_report { overall_health: UNKNOWN } } }
W20260130 02:58:01.133656 19159 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.100.67:39571, user_credentials={real_user=slave}} blocked reactor thread for 129505us
I20260130 02:58:01.141673 18869 catalog_manager.cc:5746] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571) does not have the latest schema for tablet 2112b0402de5464cb1caf2c2abb67d78 (table default.Table_da3d581b15a94840898777f74611994e [id=b7e241ced53e4160ac6fa1f65ef3674e]). Expected version 1 got 0
I20260130 02:58:01.142824 18869 catalog_manager.cc:5746] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571) does not have the latest schema for tablet 15918c90ffdc43a5a38b2950e9834188 (table default.Table_333d2efbf068485a871f566d50e563ce [id=8b7db40afb894ae99625a756c75ef1de]). Expected version 1 got 0
I20260130 02:58:01.146800 19202 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7b80b22bd44c4233837c951aa97e6996. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:01.152748 18869 catalog_manager.cc:5654] T e026dee22493424692d00f14848ab57c P d870164263bb44b6b780a557daea3656 reported cstate change: term changed from 0 to 1, leader changed from <none> to d870164263bb44b6b780a557daea3656 (127.18.100.67). New cstate: current_term: 1 leader_uuid: "d870164263bb44b6b780a557daea3656" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } health_report { overall_health: HEALTHY } } }
I20260130 02:58:01.155862 19204 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7d76046ee6c8420fa4964decbe8209c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:01.143688 19979 consensus_queue.cc:1048] T e364f4e70e5749e083e14e7fa7416ff0 P 371a36d219164974abdf848841528152 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.016s
I20260130 02:58:01.159674 18866 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:50284 (request call id 110) took 1840 ms. Trace:
I20260130 02:58:01.164206 18861 catalog_manager.cc:4985] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021): tablet 26560f746d244e58a182eb6bb24c8720 (table default.Table_b2d2da7aae8b430aba32bac2944aa931 [id=8d76f4328edc491291491c78ba205d30]) successfully deleted
W20260130 02:58:01.164866 18861 catalog_manager.cc:4965] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021): delete failed for tablet bd43c5d148514836869a8ac648c8ad11 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bd43c5d148514836869a8ac648c8ad11 already in progress: creating tablet
I20260130 02:58:01.161885 19203 tablet_service.cc:1505] Processing CreateTablet for tablet e47b22aafe5247c4b62c4193c24f427c (DEFAULT_TABLE table=default.Table_d11b53ac041740d8818edf7d2861dc6d [id=edeb74c78d234ee6a5a031c8347d3884]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:01.159991 18866 rpcz_store.cc:276] 0130 02:57:59.319112 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:57:59.573769 (+254657us) service_pool.cc:225] Handling call
0130 02:57:59.577120 (+ 3351us) catalog_manager.cc:2393] Looking up, locking, and authorizing table
0130 02:57:59.578900 (+ 1780us) catalog_manager.cc:3708] Apply alter schema
0130 02:57:59.579603 (+ 703us) catalog_manager.cc:3740] Acquired catalog manager lock
0130 02:57:59.579828 (+ 225us) catalog_manager.cc:3898] Updating metadata on disk
0130 02:58:00.084863 (+505035us) catalog_manager.cc:3929] Committing alterations to in-memory state
0130 02:58:01.159507 (+1074644us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0130 02:57:59.663338 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0130 02:57:59.665110 (+ 1772us) write_op.cc:432] Acquiring schema lock in shared mode
0130 02:57:59.665135 (+ 25us) write_op.cc:435] Acquired schema lock
0130 02:57:59.665148 (+ 13us) tablet.cc:660] Decoding operations
0130 02:57:59.665393 (+ 245us) write_op.cc:620] Acquiring the partition lock for write op
0130 02:57:59.665428 (+ 35us) write_op.cc:641] Partition lock acquired for write op
0130 02:57:59.665449 (+ 21us) tablet.cc:683] Acquiring locks for 1 operations
0130 02:57:59.665619 (+ 170us) tablet.cc:699] Row locks acquired
0130 02:57:59.665632 (+ 13us) write_op.cc:260] PREPARE: finished
0130 02:57:59.665741 (+ 109us) write_op.cc:270] Start()
0130 02:57:59.665820 (+ 79us) write_op.cc:276] Timestamp: P: 1769741879665665 usec, L: 0
0130 02:57:59.665843 (+ 23us) op_driver.cc:348] REPLICATION: starting
0130 02:57:59.666346 (+ 503us) log.cc:844] Serialized 405 byte log entry
0130 02:58:00.057745 (+391399us) op_driver.cc:464] REPLICATION: finished
0130 02:58:00.083052 (+ 25307us) write_op.cc:301] APPLY: starting
0130 02:58:00.083121 (+ 69us) tablet.cc:1365] starting BulkCheckPresence
0130 02:58:00.083211 (+ 90us) tablet.cc:1368] finished BulkCheckPresence
0130 02:58:00.083315 (+ 104us) tablet.cc:1370] starting ApplyRowOperation cycle
0130 02:58:00.083463 (+ 148us) tablet.cc:1381] finished ApplyRowOperation cycle
0130 02:58:00.083597 (+ 134us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0130 02:58:00.083637 (+ 40us) write_op.cc:312] APPLY: finished
0130 02:58:00.083920 (+ 283us) log.cc:844] Serialized 23 byte log entry
0130 02:58:00.084496 (+ 576us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:58:00.084566 (+ 70us) write_op.cc:454] Released schema lock
0130 02:58:00.084608 (+ 42us) write_op.cc:341] FINISH: Updating metrics
0130 02:58:00.085146 (+ 538us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:58:00.085194 (+ 48us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":25129,"apply.run_cpu_time_us":1286,"apply.run_wall_time_us":2030,"mutex_wait_us":26818,"num_ops":1,"prepare.queue_time_us":82302,"prepare.run_cpu_time_us":2006,"prepare.run_wall_time_us":35096,"raft.queue_time_us":26932,"raft.run_cpu_time_us":16,"raft.run_wall_time_us":18,"replication_time_us":391782}]]}
I20260130 02:58:01.180217 19203 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e47b22aafe5247c4b62c4193c24f427c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:01.188084 18872 catalog_manager.cc:5941] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4: Sending DeleteTablet for 3 replicas of tablet a46702bef0e6442d85e4ca49a5f7cfbb
I20260130 02:58:01.189369 18872 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:50284 (request call id 105) took 1915 ms. Trace:
I20260130 02:58:01.197784 19625 tablet_bootstrap.cc:654] T d3ba2df0b15c44f882070570f91b254d P d870164263bb44b6b780a557daea3656: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:01.189654 18872 rpcz_store.cc:276] 0130 02:57:59.273964 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:57:59.307591 (+ 33627us) service_pool.cc:225] Handling call
0130 02:57:59.526534 (+218943us) spinlock_profiling.cc:243] Waited 218 ms on lock 0x7b5400030554. stack: 00007fb0bd3f7373 00007fb0bd3f7270 00007fb0bcf6c574 0000000000508744 00000000005086d9 00007fb0c44490c0 00007fb0c44e7bba 00007fb0c61e6eac 00007fb0c62e2966 00007fb0c3f0a024 00007fb0c3f09fb2 00007fb0c3f09ee1 00007fb0c3f09e5c 00007fb0c3f09112 00007fb0c21dd34c 00007fb0c21dc786
0130 02:57:59.527605 (+ 1071us) catalog_manager.cc:2393] Looking up, locking, and authorizing table
0130 02:57:59.528064 (+ 459us) catalog_manager.cc:2815] Modifying in-memory table state
0130 02:57:59.528202 (+ 138us) catalog_manager.cc:2824] Locking tablets
0130 02:58:00.035216 (+507014us) catalog_manager.cc:2838] Removing table and tablets from system table
0130 02:58:00.190030 (+154814us) catalog_manager.cc:2856] Removing table from by-name map
0130 02:58:00.190247 (+ 217us) catalog_manager.cc:2872] Committing in-memory state
0130 02:58:01.187901 (+997654us) catalog_manager.cc:2876] Aborting table tasks
0130 02:58:01.189313 (+ 1412us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0130 02:58:00.070424 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0130 02:58:00.070880 (+ 456us) write_op.cc:432] Acquiring schema lock in shared mode
0130 02:58:00.070905 (+ 25us) write_op.cc:435] Acquired schema lock
0130 02:58:00.070918 (+ 13us) tablet.cc:660] Decoding operations
0130 02:58:00.071256 (+ 338us) write_op.cc:620] Acquiring the partition lock for write op
0130 02:58:00.071294 (+ 38us) write_op.cc:641] Partition lock acquired for write op
0130 02:58:00.071318 (+ 24us) tablet.cc:683] Acquiring locks for 2 operations
0130 02:58:00.071617 (+ 299us) tablet.cc:699] Row locks acquired
0130 02:58:00.071631 (+ 14us) write_op.cc:260] PREPARE: finished
0130 02:58:00.071752 (+ 121us) write_op.cc:270] Start()
0130 02:58:00.071836 (+ 84us) write_op.cc:276] Timestamp: P: 1769741880071629 usec, L: 0
0130 02:58:00.071852 (+ 16us) op_driver.cc:348] REPLICATION: starting
0130 02:58:00.072327 (+ 475us) log.cc:844] Serialized 765 byte log entry
0130 02:58:00.186176 (+113849us) op_driver.cc:464] REPLICATION: finished
0130 02:58:00.188703 (+ 2527us) write_op.cc:301] APPLY: starting
0130 02:58:00.188754 (+ 51us) tablet.cc:1365] starting BulkCheckPresence
0130 02:58:00.188862 (+ 108us) tablet.cc:1368] finished BulkCheckPresence
0130 02:58:00.189058 (+ 196us) tablet.cc:1370] starting ApplyRowOperation cycle
0130 02:58:00.189285 (+ 227us) tablet.cc:1381] finished ApplyRowOperation cycle
0130 02:58:00.189414 (+ 129us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0130 02:58:00.189455 (+ 41us) write_op.cc:312] APPLY: finished
0130 02:58:00.189610 (+ 155us) log.cc:844] Serialized 29 byte log entry
0130 02:58:00.189747 (+ 137us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:58:00.189808 (+ 61us) write_op.cc:454] Released schema lock
0130 02:58:00.189845 (+ 37us) write_op.cc:341] FINISH: Updating metrics
0130 02:58:00.190423 (+ 578us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:58:00.190448 (+ 25us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":501897600,"child_traces":[["op",{"apply.queue_time_us":2404,"apply.run_cpu_time_us":1241,"apply.run_wall_time_us":1682,"num_ops":2,"prepare.queue_time_us":28272,"prepare.run_cpu_time_us":2141,"prepare.run_wall_time_us":17484,"replication_time_us":114259}]]}
I20260130 02:58:01.201648 18872 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50284:
name: "default.Table_69267d31df644f4396a44130fa3d499d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260130 02:58:01.203266 18872 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_69267d31df644f4396a44130fa3d499d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260130 02:58:01.205433 18867 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.18.100.67:47213 (request call id 16) took 2712 ms (client timeout 1000 ms). Trace:
W20260130 02:58:01.206384 18867 rpcz_store.cc:269] 0130 02:57:58.493314 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:57:58.511030 (+ 17716us) service_pool.cc:225] Handling call
0130 02:58:01.205273 (+2694243us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0130 02:57:59.269435 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0130 02:57:59.269873 (+ 438us) write_op.cc:432] Acquiring schema lock in shared mode
0130 02:57:59.269898 (+ 25us) write_op.cc:435] Acquired schema lock
0130 02:57:59.269910 (+ 12us) tablet.cc:660] Decoding operations
0130 02:57:59.270167 (+ 257us) write_op.cc:620] Acquiring the partition lock for write op
0130 02:57:59.270203 (+ 36us) write_op.cc:641] Partition lock acquired for write op
0130 02:57:59.270226 (+ 23us) tablet.cc:683] Acquiring locks for 1 operations
0130 02:57:59.270360 (+ 134us) tablet.cc:699] Row locks acquired
0130 02:57:59.270373 (+ 13us) write_op.cc:260] PREPARE: finished
0130 02:57:59.270484 (+ 111us) write_op.cc:270] Start()
0130 02:57:59.270565 (+ 81us) write_op.cc:276] Timestamp: P: 1769741879270398 usec, L: 0
0130 02:57:59.270614 (+ 49us) op_driver.cc:348] REPLICATION: starting
0130 02:57:59.271111 (+ 497us) log.cc:844] Serialized 356 byte log entry
0130 02:57:59.293684 (+ 22573us) op_driver.cc:464] REPLICATION: finished
0130 02:57:59.299353 (+ 5669us) write_op.cc:301] APPLY: starting
0130 02:57:59.299405 (+ 52us) tablet.cc:1365] starting BulkCheckPresence
0130 02:57:59.299495 (+ 90us) tablet.cc:1368] finished BulkCheckPresence
0130 02:57:59.299577 (+ 82us) tablet.cc:1370] starting ApplyRowOperation cycle
0130 02:57:59.299699 (+ 122us) tablet.cc:1381] finished ApplyRowOperation cycle
0130 02:57:59.299812 (+ 113us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0130 02:57:59.299852 (+ 40us) write_op.cc:312] APPLY: finished
0130 02:57:59.299999 (+ 147us) log.cc:844] Serialized 23 byte log entry
0130 02:57:59.300106 (+ 107us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:57:59.300153 (+ 47us) write_op.cc:454] Released schema lock
0130 02:57:59.300181 (+ 28us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"reported_tablets":18,"child_traces":[["op",{"apply.queue_time_us":5554,"apply.run_cpu_time_us":992,"apply.run_wall_time_us":992,"num_ops":1,"prepare.queue_time_us":80831,"prepare.run_cpu_time_us":1889,"prepare.run_wall_time_us":1888,"replication_time_us":23005,"thread_start_us":191329,"threads_started":1}]]}
I20260130 02:58:01.211408 18867 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50284:
name: "default.Table_2c737f14bf854cfca58d4a9f83fb2e9e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260130 02:58:01.126718 19758 log.cc:1199] T 26560f746d244e58a182eb6bb24c8720 P 78c2c46c7520492db35d51c244fb4f74: Deleting WAL directory at /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-1/wal/wals/26560f746d244e58a182eb6bb24c8720
I20260130 02:58:01.212102 19840 raft_consensus.cc:385] T 955b9547f49f434699fe1d86b260638c P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:01.212383 19840 raft_consensus.cc:740] T 955b9547f49f434699fe1d86b260638c P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78c2c46c7520492db35d51c244fb4f74, State: Initialized, Role: FOLLOWER
I20260130 02:58:01.212606 19758 ts_tablet_manager.cc:1950] T 26560f746d244e58a182eb6bb24c8720 P 78c2c46c7520492db35d51c244fb4f74: Deleting consensus metadata
I20260130 02:58:01.214933 19758 tablet_replica.cc:333] T bd43c5d148514836869a8ac648c8ad11 P 78c2c46c7520492db35d51c244fb4f74: stopping tablet replica
I20260130 02:58:01.215624 19758 raft_consensus.cc:2243] T bd43c5d148514836869a8ac648c8ad11 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260130 02:58:01.216035 19758 raft_consensus.cc:2272] T bd43c5d148514836869a8ac648c8ad11 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260130 02:58:01.168886 19899 consensus_queue.cc:1048] T 7a902d0d571a42bb82ebffd48f7eca3f P 78c2c46c7520492db35d51c244fb4f74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260130 02:58:01.219966 19758 ts_tablet_manager.cc:1916] T bd43c5d148514836869a8ac648c8ad11 P 78c2c46c7520492db35d51c244fb4f74: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260130 02:58:01.229722 18867 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2c737f14bf854cfca58d4a9f83fb2e9e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260130 02:58:01.235431 19899 raft_consensus.cc:493] T 1ca34772ef484c1f809a99d126e28eff P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260130 02:58:01.235800 19872 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260130 02:58:01.236181 19872 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
I20260130 02:58:01.235919 19899 raft_consensus.cc:515] T 1ca34772ef484c1f809a99d126e28eff P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
W20260130 02:58:01.236510 19872 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260130 02:58:01.236824 19872 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260130 02:58:01.237725 19899 leader_election.cc:290] T 1ca34772ef484c1f809a99d126e28eff P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), d870164263bb44b6b780a557daea3656 (127.18.100.67:39571)
I20260130 02:58:01.238270 19360 tablet_service.cc:1552] Processing DeleteTablet for tablet a46702bef0e6442d85e4ca49a5f7cfbb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:57:59 UTC) from {username='slave'} at 127.0.0.1:60110
I20260130 02:58:01.246384 19204 tablet_service.cc:1552] Processing DeleteTablet for tablet a46702bef0e6442d85e4ca49a5f7cfbb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:57:59 UTC) from {username='slave'} at 127.0.0.1:42870
I20260130 02:58:01.220480 19840 consensus_queue.cc:260] T 955b9547f49f434699fe1d86b260638c P 78c2c46c7520492db35d51c244fb4f74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:01.255206 19754 tablet_replica.cc:333] T a46702bef0e6442d85e4ca49a5f7cfbb P 371a36d219164974abdf848841528152: stopping tablet replica
I20260130 02:58:01.255945 19754 raft_consensus.cc:2243] T a46702bef0e6442d85e4ca49a5f7cfbb P 371a36d219164974abdf848841528152 [term 1 LEADER]: Raft consensus shutting down.
I20260130 02:58:01.248793 19840 ts_tablet_manager.cc:1434] T 955b9547f49f434699fe1d86b260638c P 78c2c46c7520492db35d51c244fb4f74: Time spent starting tablet: real 0.126s user 0.002s sys 0.001s
I20260130 02:58:01.260035 19754 raft_consensus.cc:2272] T a46702bef0e6442d85e4ca49a5f7cfbb P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260130 02:58:01.260744 19840 tablet_bootstrap.cc:492] T a743a82540014c85aeab0e9d0994b0dc P 78c2c46c7520492db35d51c244fb4f74: Bootstrap starting.
I20260130 02:58:01.268929 19807 tablet_bootstrap.cc:492] T b7b596f944be4daf953b82b65d5cec2a P 371a36d219164974abdf848841528152: No bootstrap required, opened a new log
I20260130 02:58:01.269341 19807 ts_tablet_manager.cc:1403] T b7b596f944be4daf953b82b65d5cec2a P 371a36d219164974abdf848841528152: Time spent bootstrapping tablet: real 0.491s user 0.010s sys 0.007s
I20260130 02:58:01.275914 19225 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1ca34772ef484c1f809a99d126e28eff" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152" is_pre_election: true
I20260130 02:58:01.276258 18860 catalog_manager.cc:4985] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767): tablet 26560f746d244e58a182eb6bb24c8720 (table default.Table_b2d2da7aae8b430aba32bac2944aa931 [id=8d76f4328edc491291491c78ba205d30]) successfully deleted
W20260130 02:58:01.279352 19314 leader_election.cc:343] T 1ca34772ef484c1f809a99d126e28eff P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Illegal state: must be running to vote when last-logged opid is not known
W20260130 02:58:01.288662 18861 catalog_manager.cc:5058] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021): alter failed for tablet 878f9346d3044d8d8b15c61f86869212 (table default.Table_8edea1fae34d4d18b54b0f801a145225 [id=2009496525c94d7f8ee6ce59694b652d]),no further retry: Not found: Tablet not found: 878f9346d3044d8d8b15c61f86869212
I20260130 02:58:01.291626 19754 ts_tablet_manager.cc:1916] T a46702bef0e6442d85e4ca49a5f7cfbb P 371a36d219164974abdf848841528152: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260130 02:58:01.291496 18875 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50284:
name: "default.Table_112ffec66bd143398baeb18ddf231064"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260130 02:58:01.293059 18875 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_112ffec66bd143398baeb18ddf231064 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260130 02:58:01.302955 19872 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.100.84:46033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/master-2/wal
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.18.100.126:38191,127.18.100.125:39711,127.18.100.124:34629
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.100.124:34629
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.18.100.124
--webserver_port=37741
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type FASTDEBUG
built by None at 30 Jan 2026 02:43:21 UTC on 5fd53c4cbb9d
build id 10257
TSAN enabled
I20260130 02:58:01.303241 19758 ts_tablet_manager.cc:1929] T bd43c5d148514836869a8ac648c8ad11 P 78c2c46c7520492db35d51c244fb4f74: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260130 02:58:01.303592 19758 log.cc:1199] T bd43c5d148514836869a8ac648c8ad11 P 78c2c46c7520492db35d51c244fb4f74: Deleting WAL directory at /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-1/wal/wals/bd43c5d148514836869a8ac648c8ad11
I20260130 02:58:01.304184 19758 ts_tablet_manager.cc:1950] T bd43c5d148514836869a8ac648c8ad11 P 78c2c46c7520492db35d51c244fb4f74: Deleting consensus metadata
I20260130 02:58:01.304845 19872 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260130 02:58:01.307056 19872 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20260130 02:58:01.314996 19758 tablet_replica.cc:333] T a46702bef0e6442d85e4ca49a5f7cfbb P 78c2c46c7520492db35d51c244fb4f74: stopping tablet replica
I20260130 02:58:01.315725 19758 raft_consensus.cc:2243] T a46702bef0e6442d85e4ca49a5f7cfbb P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260130 02:58:01.316250 19758 raft_consensus.cc:2272] T a46702bef0e6442d85e4ca49a5f7cfbb P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260130 02:58:01.318041 19758 ts_tablet_manager.cc:1916] T a46702bef0e6442d85e4ca49a5f7cfbb P 78c2c46c7520492db35d51c244fb4f74: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260130 02:58:01.329118 19201 tablet_service.cc:1505] Processing CreateTablet for tablet 2ca0df71836145748cada4e4559a5656 (DEFAULT_TABLE table=default.Table_d764cbaecfee476ba5e734dd1e2f50fd [id=3f9236f0631a4e4c94ca7456f0478a45]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:01.330353 19201 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2ca0df71836145748cada4e4559a5656. 1 dirs total, 0 dirs full, 0 dirs failed
W20260130 02:58:01.334295 20001 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260130 02:58:01.341449 18860 catalog_manager.cc:4985] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767): tablet bd43c5d148514836869a8ac648c8ad11 (table default.Table_a9476edf8a5445c883f8d6ce786554a2 [id=7e29f588b41b491e8a9d1ad61e50b033]) successfully deleted
W20260130 02:58:01.342079 18860 consensus_peers.cc:597] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4 -> Peer 3749baa287ed4f249329a74071781edd (127.18.100.124:34629): Couldn't send request to peer 3749baa287ed4f249329a74071781edd. Status: Network error: Client connection negotiation failed: client connection to 127.18.100.124:34629: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20260130 02:58:01.328722 19224 raft_consensus.cc:2468] T c7c621b99deb457998c44ab1644d3939 P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78c2c46c7520492db35d51c244fb4f74 in term 1.
I20260130 02:58:01.347756 19314 leader_election.cc:304] T c7c621b99deb457998c44ab1644d3939 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 371a36d219164974abdf848841528152, 78c2c46c7520492db35d51c244fb4f74; no voters: d870164263bb44b6b780a557daea3656
I20260130 02:58:01.348892 19663 raft_consensus.cc:2804] T c7c621b99deb457998c44ab1644d3939 P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Leader election won for term 1
I20260130 02:58:01.349295 19663 raft_consensus.cc:697] T c7c621b99deb457998c44ab1644d3939 P 78c2c46c7520492db35d51c244fb4f74 [term 1 LEADER]: Becoming Leader. State: Replica: 78c2c46c7520492db35d51c244fb4f74, State: Running, Role: LEADER
I20260130 02:58:01.350044 19663 consensus_queue.cc:237] T c7c621b99deb457998c44ab1644d3939 P 78c2c46c7520492db35d51c244fb4f74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:01.352221 19807 raft_consensus.cc:359] T b7b596f944be4daf953b82b65d5cec2a P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:01.353752 19807 raft_consensus.cc:385] T b7b596f944be4daf953b82b65d5cec2a P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:01.354355 19807 raft_consensus.cc:740] T b7b596f944be4daf953b82b65d5cec2a P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 371a36d219164974abdf848841528152, State: Initialized, Role: FOLLOWER
I20260130 02:58:01.355131 19807 consensus_queue.cc:260] T b7b596f944be4daf953b82b65d5cec2a P 371a36d219164974abdf848841528152 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:01.362133 19758 ts_tablet_manager.cc:1929] T a46702bef0e6442d85e4ca49a5f7cfbb P 78c2c46c7520492db35d51c244fb4f74: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260130 02:58:01.380383 19856 consensus_queue.cc:1048] T d701152960b14396a6ac1bd10480d2b6 P 78c2c46c7520492db35d51c244fb4f74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260130 02:58:01.362493 19758 log.cc:1199] T a46702bef0e6442d85e4ca49a5f7cfbb P 78c2c46c7520492db35d51c244fb4f74: Deleting WAL directory at /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-1/wal/wals/a46702bef0e6442d85e4ca49a5f7cfbb
I20260130 02:58:01.387377 19758 ts_tablet_manager.cc:1950] T a46702bef0e6442d85e4ca49a5f7cfbb P 78c2c46c7520492db35d51c244fb4f74: Deleting consensus metadata
I20260130 02:58:01.403908 18860 catalog_manager.cc:4985] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767): tablet a46702bef0e6442d85e4ca49a5f7cfbb (table default.Table_97c3def679ff49dcb74a992828c65cde [id=59eb236fbb134702a61a33d9b15c046c]) successfully deleted
I20260130 02:58:01.411266 18867 master_service.cc:954] ReplaceTablet: received request to replace tablet 96584ca174b140f7ad1a2ca698baca71 from {username='slave'} at 127.0.0.1:50284
I20260130 02:58:01.413599 19377 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d701152960b14396a6ac1bd10480d2b6" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74" is_pre_election: true
I20260130 02:58:01.418179 18872 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50284:
table { table_name: "default.Table_b63ca4e4060f48e893b0099fb6433f04" } modify_external_catalogs: true
I20260130 02:58:01.418722 18872 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50284:
table { table_name: "default.Table_b63ca4e4060f48e893b0099fb6433f04" } modify_external_catalogs: true
I20260130 02:58:01.425874 18875 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50284:
name: "default.Table_f8856f06ce97440bae4abdf34164716d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260130 02:58:01.427398 18875 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f8856f06ce97440bae4abdf34164716d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260130 02:58:01.434967 19663 raft_consensus.cc:493] T 955b9547f49f434699fe1d86b260638c P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:01.435427 19663 raft_consensus.cc:515] T 955b9547f49f434699fe1d86b260638c P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:01.442589 19500 tablet_service.cc:1552] Processing DeleteTablet for tablet 7a902d0d571a42bb82ebffd48f7eca3f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:57:57 UTC) from {username='slave'} at 127.0.0.1:59068
I20260130 02:58:01.444036 19501 tablet_service.cc:1552] Processing DeleteTablet for tablet 4035b8a330844dd28af9f3244bf8eb9a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:57:58 UTC) from {username='slave'} at 127.0.0.1:59068
I20260130 02:58:01.451537 19663 leader_election.cc:290] T 955b9547f49f434699fe1d86b260638c P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), d870164263bb44b6b780a557daea3656 (127.18.100.67:39571)
I20260130 02:58:01.466892 20012 raft_consensus.cc:493] T 1ca34772ef484c1f809a99d126e28eff P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:01.474951 20012 raft_consensus.cc:515] T 1ca34772ef484c1f809a99d126e28eff P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:01.477519 20012 leader_election.cc:290] T 1ca34772ef484c1f809a99d126e28eff P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767)
I20260130 02:58:01.478767 20012 raft_consensus.cc:493] T 955b9547f49f434699fe1d86b260638c P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:01.483870 19377 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1ca34772ef484c1f809a99d126e28eff" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74" is_pre_election: true
I20260130 02:58:01.484501 19377 raft_consensus.cc:2468] T 1ca34772ef484c1f809a99d126e28eff P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d870164263bb44b6b780a557daea3656 in term 0.
I20260130 02:58:01.484552 20012 raft_consensus.cc:515] T 955b9547f49f434699fe1d86b260638c P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:01.486512 19377 raft_consensus.cc:1275] T 1bbc55bbf2254d8bb6b721d7e8f69d85 P 78c2c46c7520492db35d51c244fb4f74 [term 2 FOLLOWER]: Refusing update from remote peer d870164263bb44b6b780a557daea3656: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 1. (term mismatch)
I20260130 02:58:01.486881 19377 pending_rounds.cc:85] T 1bbc55bbf2254d8bb6b721d7e8f69d85 P 78c2c46c7520492db35d51c244fb4f74: Aborting all ops after (but not including) 0
I20260130 02:58:01.487118 19377 pending_rounds.cc:107] T 1bbc55bbf2254d8bb6b721d7e8f69d85 P 78c2c46c7520492db35d51c244fb4f74: Aborting uncommitted NO_OP operation due to leader change: 1.1
I20260130 02:58:01.487376 19377 raft_consensus.cc:2889] T 1bbc55bbf2254d8bb6b721d7e8f69d85 P 78c2c46c7520492db35d51c244fb4f74 [term 2 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260130 02:58:01.486647 20012 leader_election.cc:290] T 955b9547f49f434699fe1d86b260638c P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767)
I20260130 02:58:01.488039 18867 catalog_manager.cc:5941] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4: Sending DeleteTablet for 3 replicas of tablet 96584ca174b140f7ad1a2ca698baca71
I20260130 02:58:01.493388 19506 tablet_service.cc:1552] Processing DeleteTablet for tablet a46702bef0e6442d85e4ca49a5f7cfbb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:57:59 UTC) from {username='slave'} at 127.0.0.1:59068
I20260130 02:58:01.493916 19524 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "955b9547f49f434699fe1d86b260638c" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
I20260130 02:58:01.494660 19524 raft_consensus.cc:2468] T 955b9547f49f434699fe1d86b260638c P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78c2c46c7520492db35d51c244fb4f74 in term 0.
I20260130 02:58:01.495602 18867 catalog_manager.cc:6724] ReplaceTablet: tablet 96584ca174b140f7ad1a2ca698baca71 deleted and replaced by tablet 127a84346b9f473d9e7f739b1e65feb8
I20260130 02:58:01.496070 19525 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1ca34772ef484c1f809a99d126e28eff" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
I20260130 02:58:01.496739 19525 raft_consensus.cc:2468] T 1ca34772ef484c1f809a99d126e28eff P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78c2c46c7520492db35d51c244fb4f74 in term 0.
I20260130 02:58:01.499748 19527 raft_consensus.cc:3060] T d701152960b14396a6ac1bd10480d2b6 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:01.500299 18871 catalog_manager.cc:5941] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4: Sending DeleteTablet for 3 replicas of tablet 073b5941850c4140b1751956dd7cf5ef
I20260130 02:58:01.501475 18871 catalog_manager.cc:6724] ReplaceTablet: tablet 073b5941850c4140b1751956dd7cf5ef deleted and replaced by tablet 7d96e93d95f14af79cfb268626559b1e
I20260130 02:58:01.503253 19947 tablet_replica.cc:333] T 7a902d0d571a42bb82ebffd48f7eca3f P d870164263bb44b6b780a557daea3656: stopping tablet replica
I20260130 02:58:01.506829 19360 tablet_service.cc:1552] Processing DeleteTablet for tablet 96584ca174b140f7ad1a2ca698baca71 with delete_type TABLET_DATA_DELETED (replaced by tablet 127a84346b9f473d9e7f739b1e65feb8) from {username='slave'} at 127.0.0.1:60110
I20260130 02:58:01.507488 19758 tablet_replica.cc:333] T 96584ca174b140f7ad1a2ca698baca71 P 78c2c46c7520492db35d51c244fb4f74: stopping tablet replica
I20260130 02:58:01.508188 19758 raft_consensus.cc:2243] T 96584ca174b140f7ad1a2ca698baca71 P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260130 02:58:01.508690 19758 raft_consensus.cc:2272] T 96584ca174b140f7ad1a2ca698baca71 P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260130 02:58:01.508700 19202 tablet_service.cc:1552] Processing DeleteTablet for tablet 073b5941850c4140b1751956dd7cf5ef with delete_type TABLET_DATA_DELETED (replaced by tablet 7d96e93d95f14af79cfb268626559b1e) from {username='slave'} at 127.0.0.1:42870
I20260130 02:58:01.509799 19807 ts_tablet_manager.cc:1434] T b7b596f944be4daf953b82b65d5cec2a P 371a36d219164974abdf848841528152: Time spent starting tablet: real 0.240s user 0.003s sys 0.004s
I20260130 02:58:01.510686 19807 tablet_bootstrap.cc:492] T 1ca34772ef484c1f809a99d126e28eff P 371a36d219164974abdf848841528152: Bootstrap starting.
I20260130 02:58:01.511787 19360 tablet_service.cc:1552] Processing DeleteTablet for tablet 073b5941850c4140b1751956dd7cf5ef with delete_type TABLET_DATA_DELETED (replaced by tablet 7d96e93d95f14af79cfb268626559b1e) from {username='slave'} at 127.0.0.1:60110
I20260130 02:58:01.496235 19313 leader_election.cc:304] T 955b9547f49f434699fe1d86b260638c P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78c2c46c7520492db35d51c244fb4f74, d870164263bb44b6b780a557daea3656; no voters:
I20260130 02:58:01.512920 19663 raft_consensus.cc:2804] T 955b9547f49f434699fe1d86b260638c P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260130 02:58:01.513216 19663 raft_consensus.cc:493] T 955b9547f49f434699fe1d86b260638c P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260130 02:58:01.513482 19663 raft_consensus.cc:3060] T 955b9547f49f434699fe1d86b260638c P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Advancing to term 1
W20260130 02:58:01.377532 19159 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.100.66:40767, user_credentials={real_user=slave}} blocked reactor thread for 237219us
I20260130 02:58:01.515015 19313 leader_election.cc:304] T 1ca34772ef484c1f809a99d126e28eff P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 78c2c46c7520492db35d51c244fb4f74, d870164263bb44b6b780a557daea3656; no voters: 371a36d219164974abdf848841528152
I20260130 02:58:01.515764 20009 raft_consensus.cc:2804] T 1ca34772ef484c1f809a99d126e28eff P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260130 02:58:01.516075 20009 raft_consensus.cc:493] T 1ca34772ef484c1f809a99d126e28eff P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260130 02:58:01.516358 20009 raft_consensus.cc:3060] T 1ca34772ef484c1f809a99d126e28eff P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:01.510501 19758 ts_tablet_manager.cc:1916] T 96584ca174b140f7ad1a2ca698baca71 P 78c2c46c7520492db35d51c244fb4f74: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260130 02:58:01.526830 19221 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d701152960b14396a6ac1bd10480d2b6" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152" is_pre_election: true
I20260130 02:58:01.528786 19221 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1ca34772ef484c1f809a99d126e28eff" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152" is_pre_election: true
I20260130 02:58:01.529376 19223 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "955b9547f49f434699fe1d86b260638c" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152" is_pre_election: true
I20260130 02:58:01.531002 19970 raft_consensus.cc:493] T b7b596f944be4daf953b82b65d5cec2a P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:01.531126 19463 leader_election.cc:304] T d701152960b14396a6ac1bd10480d2b6 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d870164263bb44b6b780a557daea3656; no voters: 371a36d219164974abdf848841528152, 78c2c46c7520492db35d51c244fb4f74
I20260130 02:58:01.531490 19970 raft_consensus.cc:515] T b7b596f944be4daf953b82b65d5cec2a P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:01.533469 19970 leader_election.cc:290] T b7b596f944be4daf953b82b65d5cec2a P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767)
W20260130 02:58:01.539687 19313 consensus_peers.cc:597] T 7a902d0d571a42bb82ebffd48f7eca3f P 78c2c46c7520492db35d51c244fb4f74 -> Peer d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): Couldn't send request to peer d870164263bb44b6b780a557daea3656. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: STOPPING. This is attempt 1: this message will repeat every 5th retry.
I20260130 02:58:01.553435 19899 consensus_queue.cc:1048] T d701152960b14396a6ac1bd10480d2b6 P 78c2c46c7520492db35d51c244fb4f74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260130 02:58:01.556319 19947 raft_consensus.cc:2243] T 7a902d0d571a42bb82ebffd48f7eca3f P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260130 02:58:01.557866 19377 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "955b9547f49f434699fe1d86b260638c" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74" is_pre_election: true
I20260130 02:58:01.558141 19840 tablet_bootstrap.cc:654] T a743a82540014c85aeab0e9d0994b0dc P 78c2c46c7520492db35d51c244fb4f74: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:01.558806 19380 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b7b596f944be4daf953b82b65d5cec2a" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74" is_pre_election: true
I20260130 02:58:01.559360 19380 raft_consensus.cc:2468] T b7b596f944be4daf953b82b65d5cec2a P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d870164263bb44b6b780a557daea3656 in term 0.
I20260130 02:58:01.566866 19998 consensus_queue.cc:1048] T 15918c90ffdc43a5a38b2950e9834188 P 371a36d219164974abdf848841528152 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260130 02:58:01.542783 19663 raft_consensus.cc:515] T 955b9547f49f434699fe1d86b260638c P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:01.572587 19377 raft_consensus.cc:2393] T 955b9547f49f434699fe1d86b260638c P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d870164263bb44b6b780a557daea3656 in current term 1: Already voted for candidate 78c2c46c7520492db35d51c244fb4f74 in this term.
I20260130 02:58:01.572544 19223 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "955b9547f49f434699fe1d86b260638c" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152"
I20260130 02:58:01.574393 19758 ts_tablet_manager.cc:1929] T 96584ca174b140f7ad1a2ca698baca71 P 78c2c46c7520492db35d51c244fb4f74: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.6
I20260130 02:58:01.390260 19754 ts_tablet_manager.cc:1929] T a46702bef0e6442d85e4ca49a5f7cfbb P 371a36d219164974abdf848841528152: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260130 02:58:01.437963 19225 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "955b9547f49f434699fe1d86b260638c" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152" is_pre_election: true
W20260130 02:58:01.574766 19314 leader_election.cc:343] T 955b9547f49f434699fe1d86b260638c P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:01.502903 19205 tablet_service.cc:1552] Processing DeleteTablet for tablet 96584ca174b140f7ad1a2ca698baca71 with delete_type TABLET_DATA_DELETED (replaced by tablet 127a84346b9f473d9e7f739b1e65feb8) from {username='slave'} at 127.0.0.1:42870
I20260130 02:58:01.575816 19754 log.cc:1199] T a46702bef0e6442d85e4ca49a5f7cfbb P 371a36d219164974abdf848841528152: Deleting WAL directory at /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-0/wal/wals/a46702bef0e6442d85e4ca49a5f7cfbb
W20260130 02:58:01.575783 19314 leader_election.cc:343] T 955b9547f49f434699fe1d86b260638c P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:01.577340 19754 ts_tablet_manager.cc:1950] T a46702bef0e6442d85e4ca49a5f7cfbb P 371a36d219164974abdf848841528152: Deleting consensus metadata
I20260130 02:58:01.577848 18867 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50284:
table { table_name: "default.Table_e37f8d618ff244a6af90cc83a9035328" } new_table_name: "default.Table_55e0ba23f44146339c7c6928440d4948" modify_external_catalogs: true
I20260130 02:58:01.578130 19625 tablet_bootstrap.cc:492] T d3ba2df0b15c44f882070570f91b254d P d870164263bb44b6b780a557daea3656: No bootstrap required, opened a new log
I20260130 02:58:01.578639 19625 ts_tablet_manager.cc:1403] T d3ba2df0b15c44f882070570f91b254d P d870164263bb44b6b780a557daea3656: Time spent bootstrapping tablet: real 0.467s user 0.012s sys 0.011s
I20260130 02:58:01.579346 18871 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50284:
table { table_name: "default.Table_be9aa687d9744ac6a2aa335772c6cc9a" } modify_external_catalogs: true
W20260130 02:58:01.579675 19463 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.100.65:42021, user_credentials={real_user=slave}} blocked reactor thread for 48742.3us
I20260130 02:58:01.579850 18871 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50284:
table { table_name: "default.Table_be9aa687d9744ac6a2aa335772c6cc9a" } modify_external_catalogs: true
I20260130 02:58:01.574779 19758 log.cc:1199] T 96584ca174b140f7ad1a2ca698baca71 P 78c2c46c7520492db35d51c244fb4f74: Deleting WAL directory at /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-1/wal/wals/96584ca174b140f7ad1a2ca698baca71
I20260130 02:58:01.581418 19758 ts_tablet_manager.cc:1950] T 96584ca174b140f7ad1a2ca698baca71 P 78c2c46c7520492db35d51c244fb4f74: Deleting consensus metadata
I20260130 02:58:01.581702 19225 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b7b596f944be4daf953b82b65d5cec2a" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152" is_pre_election: true
I20260130 02:58:01.582270 19225 raft_consensus.cc:2468] T b7b596f944be4daf953b82b65d5cec2a P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d870164263bb44b6b780a557daea3656 in term 0.
I20260130 02:58:01.583698 18860 catalog_manager.cc:4985] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767): tablet 96584ca174b140f7ad1a2ca698baca71 (table default.Table_4f2d6e4d9c8e465087d65efc9c2de442 [id=1515db389f284e2f9ed1a3cd0061d0f1]) successfully deleted
I20260130 02:58:01.584787 19625 raft_consensus.cc:359] T d3ba2df0b15c44f882070570f91b254d P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:01.546319 20009 raft_consensus.cc:515] T 1ca34772ef484c1f809a99d126e28eff P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:01.588337 19225 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1ca34772ef484c1f809a99d126e28eff" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152"
I20260130 02:58:01.589246 19754 tablet_replica.cc:333] T 073b5941850c4140b1751956dd7cf5ef P 371a36d219164974abdf848841528152: stopping tablet replica
I20260130 02:58:01.589372 18861 catalog_manager.cc:4985] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021): tablet a46702bef0e6442d85e4ca49a5f7cfbb (table default.Table_97c3def679ff49dcb74a992828c65cde [id=59eb236fbb134702a61a33d9b15c046c]) successfully deleted
I20260130 02:58:01.589989 19754 raft_consensus.cc:2243] T 073b5941850c4140b1751956dd7cf5ef P 371a36d219164974abdf848841528152 [term 1 LEADER]: Raft consensus shutting down.
W20260130 02:58:01.590507 19314 leader_election.cc:343] T 1ca34772ef484c1f809a99d126e28eff P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:01.590837 19754 raft_consensus.cc:2272] T 073b5941850c4140b1751956dd7cf5ef P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260130 02:58:01.593557 19663 leader_election.cc:290] T 955b9547f49f434699fe1d86b260638c P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 election: Requested vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), d870164263bb44b6b780a557daea3656 (127.18.100.67:39571)
I20260130 02:58:01.594228 19758 tablet_replica.cc:333] T 073b5941850c4140b1751956dd7cf5ef P 78c2c46c7520492db35d51c244fb4f74: stopping tablet replica
I20260130 02:58:01.585397 19625 raft_consensus.cc:385] T d3ba2df0b15c44f882070570f91b254d P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:01.615072 19758 raft_consensus.cc:2243] T 073b5941850c4140b1751956dd7cf5ef P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260130 02:58:01.615264 19625 raft_consensus.cc:740] T d3ba2df0b15c44f882070570f91b254d P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d870164263bb44b6b780a557daea3656, State: Initialized, Role: FOLLOWER
I20260130 02:58:01.615624 19758 raft_consensus.cc:2272] T 073b5941850c4140b1751956dd7cf5ef P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260130 02:58:01.616021 19625 consensus_queue.cc:260] T d3ba2df0b15c44f882070570f91b254d P d870164263bb44b6b780a557daea3656 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:01.617573 19758 ts_tablet_manager.cc:1916] T 073b5941850c4140b1751956dd7cf5ef P 78c2c46c7520492db35d51c244fb4f74: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260130 02:58:01.620219 19625 ts_tablet_manager.cc:1434] T d3ba2df0b15c44f882070570f91b254d P d870164263bb44b6b780a557daea3656: Time spent starting tablet: real 0.041s user 0.004s sys 0.001s
I20260130 02:58:01.623396 20009 leader_election.cc:290] T 1ca34772ef484c1f809a99d126e28eff P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 election: Requested vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), d870164263bb44b6b780a557daea3656 (127.18.100.67:39571)
I20260130 02:58:01.652729 19840 tablet_bootstrap.cc:492] T a743a82540014c85aeab0e9d0994b0dc P 78c2c46c7520492db35d51c244fb4f74: No bootstrap required, opened a new log
I20260130 02:58:01.653221 19840 ts_tablet_manager.cc:1403] T a743a82540014c85aeab0e9d0994b0dc P 78c2c46c7520492db35d51c244fb4f74: Time spent bootstrapping tablet: real 0.393s user 0.009s sys 0.000s
I20260130 02:58:01.655540 19840 raft_consensus.cc:359] T a743a82540014c85aeab0e9d0994b0dc P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:01.656212 19840 raft_consensus.cc:385] T a743a82540014c85aeab0e9d0994b0dc P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:01.656458 19840 raft_consensus.cc:740] T a743a82540014c85aeab0e9d0994b0dc P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78c2c46c7520492db35d51c244fb4f74, State: Initialized, Role: FOLLOWER
I20260130 02:58:01.657886 19758 ts_tablet_manager.cc:1929] T 073b5941850c4140b1751956dd7cf5ef P 78c2c46c7520492db35d51c244fb4f74: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260130 02:58:01.658280 19758 log.cc:1199] T 073b5941850c4140b1751956dd7cf5ef P 78c2c46c7520492db35d51c244fb4f74: Deleting WAL directory at /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-1/wal/wals/073b5941850c4140b1751956dd7cf5ef
I20260130 02:58:01.659029 19758 ts_tablet_manager.cc:1950] T 073b5941850c4140b1751956dd7cf5ef P 78c2c46c7520492db35d51c244fb4f74: Deleting consensus metadata
I20260130 02:58:01.597981 19947 raft_consensus.cc:2272] T 7a902d0d571a42bb82ebffd48f7eca3f P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260130 02:58:01.666145 19947 ts_tablet_manager.cc:1916] T 7a902d0d571a42bb82ebffd48f7eca3f P d870164263bb44b6b780a557daea3656: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260130 02:58:01.601282 19971 raft_consensus.cc:493] T bd43c5d148514836869a8ac648c8ad11 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:01.599152 19970 consensus_queue.cc:1048] T e026dee22493424692d00f14848ab57c P d870164263bb44b6b780a557daea3656 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.022s
I20260130 02:58:01.601908 19504 tablet_service.cc:1552] Processing DeleteTablet for tablet 073b5941850c4140b1751956dd7cf5ef with delete_type TABLET_DATA_DELETED (replaced by tablet 7d96e93d95f14af79cfb268626559b1e) from {username='slave'} at 127.0.0.1:59068
I20260130 02:58:01.611727 19525 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "955b9547f49f434699fe1d86b260638c" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656"
I20260130 02:58:01.666833 19971 raft_consensus.cc:515] T bd43c5d148514836869a8ac648c8ad11 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:01.669471 19971 leader_election.cc:290] T bd43c5d148514836869a8ac648c8ad11 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767)
I20260130 02:58:01.671303 19525 raft_consensus.cc:3060] T 955b9547f49f434699fe1d86b260638c P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:01.603116 19502 tablet_service.cc:1552] Processing DeleteTablet for tablet 96584ca174b140f7ad1a2ca698baca71 with delete_type TABLET_DATA_DELETED (replaced by tablet 127a84346b9f473d9e7f739b1e65feb8) from {username='slave'} at 127.0.0.1:59068
W20260130 02:58:01.597674 19462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.100.66:40767, user_credentials={real_user=slave}} blocked reactor thread for 53634.9us
I20260130 02:58:01.649667 19526 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1ca34772ef484c1f809a99d126e28eff" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656"
I20260130 02:58:01.687925 19625 tablet_bootstrap.cc:492] T b3f3177306914e3c8b253db54ee30fb6 P d870164263bb44b6b780a557daea3656: Bootstrap starting.
I20260130 02:58:01.692422 19360 tablet_service.cc:1505] Processing CreateTablet for tablet 111287b5557d4abda1c5cd560fb0d019 (DEFAULT_TABLE table=default.Table_fec49c37fdd045e7801325ace326f367 [id=06f1da4d6e0e46e4a141914a3aa68c07]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:01.693409 19360 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 111287b5557d4abda1c5cd560fb0d019. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:01.693218 19504 tablet_service.cc:1505] Processing CreateTablet for tablet 111287b5557d4abda1c5cd560fb0d019 (DEFAULT_TABLE table=default.Table_fec49c37fdd045e7801325ace326f367 [id=06f1da4d6e0e46e4a141914a3aa68c07]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:01.694356 19504 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 111287b5557d4abda1c5cd560fb0d019. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:01.688764 19462 leader_election.cc:304] T 1ca34772ef484c1f809a99d126e28eff P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78c2c46c7520492db35d51c244fb4f74, d870164263bb44b6b780a557daea3656; no voters:
I20260130 02:58:01.696349 19205 tablet_service.cc:1505] Processing CreateTablet for tablet 111287b5557d4abda1c5cd560fb0d019 (DEFAULT_TABLE table=default.Table_fec49c37fdd045e7801325ace326f367 [id=06f1da4d6e0e46e4a141914a3aa68c07]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:01.700853 19205 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 111287b5557d4abda1c5cd560fb0d019. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:01.704537 19200 tablet_service.cc:1505] Processing CreateTablet for tablet 91f954f3ef4244c9a575a7421a7e75eb (DEFAULT_TABLE table=default.Table_c04335826bdf4c2c94c4cf6e3bca1a7a [id=db23c1be42c44b6f9eb774f14d2b8fd6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:01.705546 19200 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 91f954f3ef4244c9a575a7421a7e75eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:01.706889 19358 tablet_service.cc:1505] Processing CreateTablet for tablet 730bd1d7d15b4cbe945b3ef03b2455f7 (DEFAULT_TABLE table=default.Table_1e9e20a1bfec4bcf81f9eb6219c33d59 [id=f675b7a3ec744533903d8834eeed51d7]), partition=RANGE (key) PARTITION UNBOUNDED
W20260130 02:58:01.707325 18867 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7d76046ee6c8420fa4964decbe8209c1 consensus state has no leader
I20260130 02:58:01.707858 19358 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 730bd1d7d15b4cbe945b3ef03b2455f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:01.708330 18867 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50284:
table { table_name: "default.Table_f03fffd6ca4c48cdaf9b143f1c753710" } new_table_name: "default.Table_e84c6277dfc944059f62cbe0d6b937ee" modify_external_catalogs: true
I20260130 02:58:01.711298 19356 tablet_service.cc:1505] Processing CreateTablet for tablet 91f954f3ef4244c9a575a7421a7e75eb (DEFAULT_TABLE table=default.Table_c04335826bdf4c2c94c4cf6e3bca1a7a [id=db23c1be42c44b6f9eb774f14d2b8fd6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:01.712366 19356 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 91f954f3ef4244c9a575a7421a7e75eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:01.695024 19526 raft_consensus.cc:3060] T 1ca34772ef484c1f809a99d126e28eff P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Advancing to term 1
W20260130 02:58:01.733355 19463 leader_election.cc:343] T 1ca34772ef484c1f809a99d126e28eff P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:01.734740 19462 leader_election.cc:304] T b7b596f944be4daf953b82b65d5cec2a P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78c2c46c7520492db35d51c244fb4f74, d870164263bb44b6b780a557daea3656; no voters:
W20260130 02:58:01.737030 19463 leader_election.cc:343] T 955b9547f49f434699fe1d86b260638c P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:01.737432 19463 leader_election.cc:304] T 955b9547f49f434699fe1d86b260638c P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d870164263bb44b6b780a557daea3656; no voters: 371a36d219164974abdf848841528152, 78c2c46c7520492db35d51c244fb4f74
I20260130 02:58:01.741473 19742 consensus_queue.cc:1048] T 1bbc55bbf2254d8bb6b721d7e8f69d85 P d870164263bb44b6b780a557daea3656 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.006s
I20260130 02:58:01.744439 19199 tablet_service.cc:1505] Processing CreateTablet for tablet 534dc31a9b7e492c82f3672c75742e6c (DEFAULT_TABLE table=default.Table_081113b62fb743189a6e04bf85dcc5fa [id=66e19e6011bc4d86bf579a4a1b3db867]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:01.745378 18866 catalog_manager.cc:5746] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767) does not have the latest schema for tablet 15918c90ffdc43a5a38b2950e9834188 (table default.Table_333d2efbf068485a871f566d50e563ce [id=8b7db40afb894ae99625a756c75ef1de]). Expected version 1 got 0
I20260130 02:58:01.745690 19199 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 534dc31a9b7e492c82f3672c75742e6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:01.747053 18866 catalog_manager.cc:5746] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767) does not have the latest schema for tablet bac739bf4cac4272975f5587ad9c71ff (table default.Table_9e942a1ce3944e9aa028e75656ce066d [id=18ed268eb9ad4450abf9874310ace074]). Expected version 1 got 0
I20260130 02:58:01.747301 19948 raft_consensus.cc:2749] T d701152960b14396a6ac1bd10480d2b6 P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260130 02:58:01.747654 18866 catalog_manager.cc:5746] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767) does not have the latest schema for tablet e364f4e70e5749e083e14e7fa7416ff0 (table default.Table_bc11cb1bce0348ed9ea0912b3ee521b9 [id=df7d3d68e36142ff8564e63ffe0b8797]). Expected version 1 got 0
I20260130 02:58:01.748306 18866 catalog_manager.cc:5746] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767) does not have the latest schema for tablet 2112b0402de5464cb1caf2c2abb67d78 (table default.Table_da3d581b15a94840898777f74611994e [id=b7e241ced53e4160ac6fa1f65ef3674e]). Expected version 1 got 0
I20260130 02:58:01.748222 19506 tablet_service.cc:1505] Processing CreateTablet for tablet 91f954f3ef4244c9a575a7421a7e75eb (DEFAULT_TABLE table=default.Table_c04335826bdf4c2c94c4cf6e3bca1a7a [id=db23c1be42c44b6f9eb774f14d2b8fd6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:01.749893 19502 tablet_service.cc:1505] Processing CreateTablet for tablet 730bd1d7d15b4cbe945b3ef03b2455f7 (DEFAULT_TABLE table=default.Table_1e9e20a1bfec4bcf81f9eb6219c33d59 [id=f675b7a3ec744533903d8834eeed51d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:01.752005 19508 tablet_service.cc:1505] Processing CreateTablet for tablet 534dc31a9b7e492c82f3672c75742e6c (DEFAULT_TABLE table=default.Table_081113b62fb743189a6e04bf85dcc5fa [id=66e19e6011bc4d86bf579a4a1b3db867]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:01.752045 18866 catalog_manager.cc:5654] T c3fce8cc99fb451ebd532fa37390e862 P 78c2c46c7520492db35d51c244fb4f74 reported cstate change: term changed from 0 to 1, leader changed from <none> to 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66). New cstate: current_term: 1 leader_uuid: "78c2c46c7520492db35d51c244fb4f74" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } health_report { overall_health: HEALTHY } } }
I20260130 02:58:01.704495 19525 raft_consensus.cc:2468] T 955b9547f49f434699fe1d86b260638c P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78c2c46c7520492db35d51c244fb4f74 in term 1.
I20260130 02:58:01.754043 19506 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 91f954f3ef4244c9a575a7421a7e75eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:01.742946 20012 raft_consensus.cc:2804] T b7b596f944be4daf953b82b65d5cec2a P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260130 02:58:01.758881 20012 raft_consensus.cc:493] T b7b596f944be4daf953b82b65d5cec2a P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260130 02:58:01.759223 20012 raft_consensus.cc:3060] T b7b596f944be4daf953b82b65d5cec2a P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:01.758919 19502 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 730bd1d7d15b4cbe945b3ef03b2455f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:01.760408 19508 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 534dc31a9b7e492c82f3672c75742e6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:01.760522 19357 tablet_service.cc:1505] Processing CreateTablet for tablet 534dc31a9b7e492c82f3672c75742e6c (DEFAULT_TABLE table=default.Table_081113b62fb743189a6e04bf85dcc5fa [id=66e19e6011bc4d86bf579a4a1b3db867]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:01.761456 19357 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 534dc31a9b7e492c82f3672c75742e6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:01.763613 19313 leader_election.cc:304] T 955b9547f49f434699fe1d86b260638c P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 78c2c46c7520492db35d51c244fb4f74, d870164263bb44b6b780a557daea3656; no voters: 371a36d219164974abdf848841528152
I20260130 02:58:01.770641 19970 raft_consensus.cc:2749] T 955b9547f49f434699fe1d86b260638c P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260130 02:58:01.764439 20009 raft_consensus.cc:2804] T 955b9547f49f434699fe1d86b260638c P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Leader election won for term 1
I20260130 02:58:01.779582 20009 raft_consensus.cc:697] T 955b9547f49f434699fe1d86b260638c P 78c2c46c7520492db35d51c244fb4f74 [term 1 LEADER]: Becoming Leader. State: Replica: 78c2c46c7520492db35d51c244fb4f74, State: Running, Role: LEADER
I20260130 02:58:01.779748 19203 tablet_service.cc:1505] Processing CreateTablet for tablet a90e92d569254bbf8a5fbf286baf5cce (DEFAULT_TABLE table=default.Table_1e42d571631143088cf33d4c3aafed28 [id=30ba2af6dffd4f7e8dcbfdd2f3798157]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:01.780745 19203 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a90e92d569254bbf8a5fbf286baf5cce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:01.699669 19807 tablet_bootstrap.cc:654] T 1ca34772ef484c1f809a99d126e28eff P 371a36d219164974abdf848841528152: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:01.696573 19202 tablet_service.cc:1505] Processing CreateTablet for tablet 730bd1d7d15b4cbe945b3ef03b2455f7 (DEFAULT_TABLE table=default.Table_1e9e20a1bfec4bcf81f9eb6219c33d59 [id=f675b7a3ec744533903d8834eeed51d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:01.783279 19202 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 730bd1d7d15b4cbe945b3ef03b2455f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:01.783316 18875 catalog_manager.cc:5746] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021) does not have the latest schema for tablet bac739bf4cac4272975f5587ad9c71ff (table default.Table_9e942a1ce3944e9aa028e75656ce066d [id=18ed268eb9ad4450abf9874310ace074]). Expected version 1 got 0
I20260130 02:58:01.784000 18875 catalog_manager.cc:5746] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021) does not have the latest schema for tablet e364f4e70e5749e083e14e7fa7416ff0 (table default.Table_bc11cb1bce0348ed9ea0912b3ee521b9 [id=df7d3d68e36142ff8564e63ffe0b8797]). Expected version 1 got 0
I20260130 02:58:01.784619 18875 catalog_manager.cc:5654] T 15918c90ffdc43a5a38b2950e9834188 P 371a36d219164974abdf848841528152 reported cstate change: term changed from 0 to 1, leader changed from <none> to 371a36d219164974abdf848841528152 (127.18.100.65). New cstate: current_term: 1 leader_uuid: "371a36d219164974abdf848841528152" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } health_report { overall_health: UNKNOWN } } }
I20260130 02:58:01.785763 18875 catalog_manager.cc:5746] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021) does not have the latest schema for tablet 15918c90ffdc43a5a38b2950e9834188 (table default.Table_333d2efbf068485a871f566d50e563ce [id=8b7db40afb894ae99625a756c75ef1de]). Expected version 1 got 0
I20260130 02:58:01.787333 18875 catalog_manager.cc:5746] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021) does not have the latest schema for tablet 2112b0402de5464cb1caf2c2abb67d78 (table default.Table_da3d581b15a94840898777f74611994e [id=b7e241ced53e4160ac6fa1f65ef3674e]). Expected version 1 got 0
I20260130 02:58:01.789207 18873 catalog_manager.cc:6199] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4: default.Table_be9aa687d9744ac6a2aa335772c6cc9a [id=4f59d133b0cd4773bb012145927a92a4] alter complete (version 2)
I20260130 02:58:01.797922 18872 catalog_manager.cc:5941] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4: Sending DeleteTablet for 3 replicas of tablet b078934be4bb4fd1915c691eeea9d00f
I20260130 02:58:01.798439 19625 tablet_bootstrap.cc:654] T b3f3177306914e3c8b253db54ee30fb6 P d870164263bb44b6b780a557daea3656: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:01.800081 18872 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50284:
name: "default.Table_f2f790e8e46840ecb199c34fae899ada"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260130 02:58:01.800619 19505 tablet_service.cc:1505] Processing CreateTablet for tablet a90e92d569254bbf8a5fbf286baf5cce (DEFAULT_TABLE table=default.Table_1e42d571631143088cf33d4c3aafed28 [id=30ba2af6dffd4f7e8dcbfdd2f3798157]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:01.801553 19505 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a90e92d569254bbf8a5fbf286baf5cce. 1 dirs total, 0 dirs full, 0 dirs failed
W20260130 02:58:01.801520 18872 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f2f790e8e46840ecb199c34fae899ada in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260130 02:58:01.803320 19197 tablet_service.cc:1552] Processing DeleteTablet for tablet b078934be4bb4fd1915c691eeea9d00f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:58:01 UTC) from {username='slave'} at 127.0.0.1:42870
I20260130 02:58:01.803478 19353 tablet_service.cc:1552] Processing DeleteTablet for tablet b078934be4bb4fd1915c691eeea9d00f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:58:01 UTC) from {username='slave'} at 127.0.0.1:60110
I20260130 02:58:01.805619 19758 tablet_replica.cc:333] T b078934be4bb4fd1915c691eeea9d00f P 78c2c46c7520492db35d51c244fb4f74: stopping tablet replica
I20260130 02:58:01.809408 18860 catalog_manager.cc:4985] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767): tablet 073b5941850c4140b1751956dd7cf5ef (table default.Table_6eb0a92b1926487cbf0b978d26d6a7b1 [id=de7a420d40864f55b156e149308ba12f]) successfully deleted
I20260130 02:58:01.815094 19197 tablet_service.cc:1505] Processing CreateTablet for tablet 0a763e48583c4e748a46a325746ce69a (DEFAULT_TABLE table=default.Table_94b23422c82d419d97dc097a3efb92fc [id=d2f35437eae140dab4150079eae46377]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:01.816175 19197 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0a763e48583c4e748a46a325746ce69a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:01.816392 20025 tablet.cc:1722] T 2112b0402de5464cb1caf2c2abb67d78 P 371a36d219164974abdf848841528152: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260130 02:58:01.821785 19357 tablet_service.cc:1505] Processing CreateTablet for tablet a90e92d569254bbf8a5fbf286baf5cce (DEFAULT_TABLE table=default.Table_1e42d571631143088cf33d4c3aafed28 [id=30ba2af6dffd4f7e8dcbfdd2f3798157]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:01.822992 19357 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a90e92d569254bbf8a5fbf286baf5cce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:01.829361 18872 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50284:
name: "default.Table_ddfb5df90fdb40a087ac434d088bbf0e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260130 02:58:01.830868 18872 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ddfb5df90fdb40a087ac434d088bbf0e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260130 02:58:01.833511 19507 tablet_service.cc:1552] Processing DeleteTablet for tablet b078934be4bb4fd1915c691eeea9d00f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:58:01 UTC) from {username='slave'} at 127.0.0.1:59068
I20260130 02:58:01.835143 18867 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50284:
name: "default.Table_2d345d0f0957486c86c2cf2e34b00941"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260130 02:58:01.836519 18867 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2d345d0f0957486c86c2cf2e34b00941 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260130 02:58:01.843746 18870 catalog_manager.cc:6199] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4: default.Table_007e969f70d345a7bcb3dfbaabbc025d [id=77d4fa8705a3419cb0a84d3ca1bb4ce3] alter complete (version 1)
I20260130 02:58:01.844005 19754 ts_tablet_manager.cc:1916] T 073b5941850c4140b1751956dd7cf5ef P 371a36d219164974abdf848841528152: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260130 02:58:01.848290 19462 leader_election.cc:343] T bd43c5d148514836869a8ac648c8ad11 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767): Not found: Tablet not found: bd43c5d148514836869a8ac648c8ad11
I20260130 02:58:01.849923 19196 tablet_service.cc:1552] Processing DeleteTablet for tablet 93809949599a449ca09b13e9b8bbdef6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:57:57 UTC) from {username='slave'} at 127.0.0.1:42870
I20260130 02:58:01.850490 19204 tablet_service.cc:1552] Processing DeleteTablet for tablet 4035b8a330844dd28af9f3244bf8eb9a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:57:58 UTC) from {username='slave'} at 127.0.0.1:42870
I20260130 02:58:01.850803 19198 tablet_service.cc:1552] Processing DeleteTablet for tablet 3b8cdac4bf4b46efa26236fc930eb9a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:57:54 UTC) from {username='slave'} at 127.0.0.1:42870
I20260130 02:58:01.851240 20034 tablet.cc:1722] T 2112b0402de5464cb1caf2c2abb67d78 P 78c2c46c7520492db35d51c244fb4f74: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20260130 02:58:01.856837 18861 catalog_manager.cc:5058] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021): alter failed for tablet 878f9346d3044d8d8b15c61f86869212 (table default.Table_8edea1fae34d4d18b54b0f801a145225 [id=2009496525c94d7f8ee6ce59694b652d]),no further retry: Not found: Tablet not found: 878f9346d3044d8d8b15c61f86869212
W20260130 02:58:01.861507 18873 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.18.100.66:59691 (request call id 19) took 1984 ms (client timeout 1000 ms). Trace:
W20260130 02:58:01.869302 18869 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.18.100.67:47213 (request call id 18) took 1350 ms (client timeout 1000 ms). Trace:
I20260130 02:58:01.872552 19501 tablet_service.cc:1505] Processing CreateTablet for tablet 0a763e48583c4e748a46a325746ce69a (DEFAULT_TABLE table=default.Table_94b23422c82d419d97dc097a3efb92fc [id=d2f35437eae140dab4150079eae46377]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:01.872970 19352 tablet_service.cc:1552] Processing DeleteTablet for tablet 4035b8a330844dd28af9f3244bf8eb9a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:57:58 UTC) from {username='slave'} at 127.0.0.1:60110
I20260130 02:58:01.873662 19355 tablet_service.cc:1552] Processing DeleteTablet for tablet a46702bef0e6442d85e4ca49a5f7cfbb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:57:59 UTC) from {username='slave'} at 127.0.0.1:60110
I20260130 02:58:01.873773 19501 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0a763e48583c4e748a46a325746ce69a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:01.873899 19353 tablet_service.cc:1552] Processing DeleteTablet for tablet 7a902d0d571a42bb82ebffd48f7eca3f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:57:57 UTC) from {username='slave'} at 127.0.0.1:60110
I20260130 02:58:01.874939 19354 tablet_service.cc:1505] Processing CreateTablet for tablet 0a763e48583c4e748a46a325746ce69a (DEFAULT_TABLE table=default.Table_94b23422c82d419d97dc097a3efb92fc [id=d2f35437eae140dab4150079eae46377]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:01.875248 19500 tablet_service.cc:1552] Processing DeleteTablet for tablet 5b2976108584478a98fd90fe6594af66 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:57:58 UTC) from {username='slave'} at 127.0.0.1:59068
I20260130 02:58:01.874939 19352 tablet_service.cc:1505] Processing CreateTablet for tablet 341041cb0be2456e88403367c0814704 (DEFAULT_TABLE table=default.Table_b2d2da7aae8b430aba32bac2944aa931 [id=8d76f4328edc491291491c78ba205d30]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:01.875946 19354 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0a763e48583c4e748a46a325746ce69a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:01.876312 19500 tablet_service.cc:1552] Processing DeleteTablet for tablet d701152960b14396a6ac1bd10480d2b6 with delete_type TABLET_DATA_DELETED (replaced by tablet 481bd4cca6ee4c178873bf12392e3441) from {username='slave'} at 127.0.0.1:59068
I20260130 02:58:01.876542 19352 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 341041cb0be2456e88403367c0814704. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:01.877018 19503 tablet_service.cc:1552] Processing DeleteTablet for tablet d4f2b54930cc4f5c825562e3e0a79f4d with delete_type TABLET_DATA_DELETED (replaced by tablet 54be22352e404ff9b798a075693f7f21) from {username='slave'} at 127.0.0.1:59068
I20260130 02:58:01.883904 19503 tablet_service.cc:1505] Processing CreateTablet for tablet 341041cb0be2456e88403367c0814704 (DEFAULT_TABLE table=default.Table_b2d2da7aae8b430aba32bac2944aa931 [id=8d76f4328edc491291491c78ba205d30]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:01.885022 19503 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 341041cb0be2456e88403367c0814704. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:01.887233 19360 tablet_service.cc:1552] Processing DeleteTablet for tablet 878f9346d3044d8d8b15c61f86869212 with delete_type TABLET_DATA_DELETED (replaced by tablet 3b148691090e4ee581d46f9845a6e49a) from {username='slave'} at 127.0.0.1:60110
I20260130 02:58:01.887872 19358 tablet_service.cc:1552] Processing DeleteTablet for tablet d4f2b54930cc4f5c825562e3e0a79f4d with delete_type TABLET_DATA_DELETED (replaced by tablet 54be22352e404ff9b798a075693f7f21) from {username='slave'} at 127.0.0.1:60110
I20260130 02:58:01.888217 19971 raft_consensus.cc:493] T a743a82540014c85aeab0e9d0994b0dc P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:01.888650 19971 raft_consensus.cc:515] T a743a82540014c85aeab0e9d0994b0dc P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:01.890465 19971 leader_election.cc:290] T a743a82540014c85aeab0e9d0994b0dc P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767)
W20260130 02:58:01.887511 18868 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.18.100.67:47213 (request call id 17) took 2379 ms (client timeout 1000 ms). Trace:
W20260130 02:58:01.894757 18868 rpcz_store.cc:269] 0130 02:57:59.508060 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:57:59.523144 (+ 15084us) service_pool.cc:225] Handling call
0130 02:57:59.595181 (+ 72037us) spinlock_profiling.cc:243] Waited 68.9 ms on lock 0x7b5400030554. stack: 00007fb0bd3f7373 00007fb0bd3f7270 00007fb0bcf6c574 0000000000508744 00000000005086d9 00007fb0c44490c0 00007fb0c44e7bba 00007fb0c61e6eac 00007fb0c62e110e 00007fb0c3efe4a4 00007fb0c3efe432 00007fb0c3efe361 00007fb0c3efe2dc 00007fb0c3efd592 00007fb0c21dd34c 00007fb0c21dc786
0130 02:58:01.887435 (+2292254us) inbound_call.cc:173] Queueing success response
Metrics: {"reported_tablets":20,"spinlock_wait_cycles":158384000}
I20260130 02:58:01.896680 18868 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50284:
name: "default.Table_1d9d1f3b2b6048f8949fdba26f5fd3d7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260130 02:58:01.898391 18868 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1d9d1f3b2b6048f8949fdba26f5fd3d7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260130 02:58:01.888366 18874 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.18.100.65:51021 (request call id 22) took 2901 ms (client timeout 1000 ms). Trace:
I20260130 02:58:01.871032 19204 tablet_service.cc:1505] Processing CreateTablet for tablet 3cf5154d8cc44f2bb12978202699c193 (DEFAULT_TABLE table=default.Table_bb334df0beee420e883c3225d93892c0 [id=b7b1d250632741a2ad0ed194aae9b197]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:01.868863 19198 tablet_service.cc:1505] Processing CreateTablet for tablet 341041cb0be2456e88403367c0814704 (DEFAULT_TABLE table=default.Table_b2d2da7aae8b430aba32bac2944aa931 [id=8d76f4328edc491291491c78ba205d30]), partition=RANGE (key) PARTITION UNBOUNDED
W20260130 02:58:01.898852 18874 rpcz_store.cc:269] 0130 02:57:58.954970 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:57:58.959529 (+ 4559us) service_pool.cc:225] Handling call
0130 02:57:59.059156 (+ 99627us) spinlock_profiling.cc:243] Waited 96.5 ms on lock 0x7b5400030554. stack: 00007fb0bd3f7373 00007fb0bd3f7270 00007fb0bcf6c574 0000000000508744 00000000005086d9 00007fb0c44490c0 00007fb0c44e7bba 00007fb0c61e6eac 00007fb0c62e110e 00007fb0c3efe4a4 00007fb0c3efe432 00007fb0c3efe361 00007fb0c3efe2dc 00007fb0c3efd592 00007fb0c21dd34c 00007fb0c21dc786
0130 02:58:01.856475 (+2797319us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0130 02:57:59.568388 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0130 02:57:59.568862 (+ 474us) write_op.cc:432] Acquiring schema lock in shared mode
0130 02:57:59.568881 (+ 19us) write_op.cc:435] Acquired schema lock
0130 02:57:59.568893 (+ 12us) tablet.cc:660] Decoding operations
0130 02:57:59.569203 (+ 310us) write_op.cc:620] Acquiring the partition lock for write op
0130 02:57:59.569240 (+ 37us) write_op.cc:641] Partition lock acquired for write op
0130 02:57:59.569263 (+ 23us) tablet.cc:683] Acquiring locks for 2 operations
0130 02:57:59.569516 (+ 253us) tablet.cc:699] Row locks acquired
0130 02:57:59.569529 (+ 13us) write_op.cc:260] PREPARE: finished
0130 02:57:59.569635 (+ 106us) write_op.cc:270] Start()
0130 02:57:59.569709 (+ 74us) write_op.cc:276] Timestamp: P: 1769741879569574 usec, L: 0
0130 02:57:59.569725 (+ 16us) op_driver.cc:348] REPLICATION: starting
0130 02:57:59.570158 (+ 433us) log.cc:844] Serialized 913 byte log entry
0130 02:57:59.951624 (+381466us) op_driver.cc:464] REPLICATION: finished
0130 02:57:59.958766 (+ 7142us) write_op.cc:301] APPLY: starting
0130 02:57:59.958834 (+ 68us) tablet.cc:1365] starting BulkCheckPresence
0130 02:57:59.958956 (+ 122us) tablet.cc:1368] finished BulkCheckPresence
0130 02:57:59.959043 (+ 87us) tablet.cc:1370] starting ApplyRowOperation cycle
0130 02:57:59.959334 (+ 291us) tablet.cc:1381] finished ApplyRowOperation cycle
0130 02:57:59.959473 (+ 139us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0130 02:57:59.959517 (+ 44us) write_op.cc:312] APPLY: finished
0130 02:57:59.959725 (+ 208us) log.cc:844] Serialized 29 byte log entry
0130 02:57:59.959867 (+ 142us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:57:59.959960 (+ 93us) write_op.cc:454] Released schema lock
0130 02:57:59.959994 (+ 34us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"mutex_wait_us":99509,"reported_tablets":18,"spinlock_wait_cycles":221905664,"child_traces":[["op",{"apply.queue_time_us":7007,"apply.run_cpu_time_us":1460,"apply.run_wall_time_us":8328,"num_ops":2,"prepare.queue_time_us":1078,"prepare.run_cpu_time_us":1907,"prepare.run_wall_time_us":2924,"replication_time_us":381763,"thread_start_us":102497,"threads_started":1}]]}
I20260130 02:58:01.859407 19201 tablet_service.cc:1552] Processing DeleteTablet for tablet d4f2b54930cc4f5c825562e3e0a79f4d with delete_type TABLET_DATA_DELETED (replaced by tablet 54be22352e404ff9b798a075693f7f21) from {username='slave'} at 127.0.0.1:42870
W20260130 02:58:01.911314 20022 log.cc:927] Time spent T d701152960b14396a6ac1bd10480d2b6 P d870164263bb44b6b780a557daea3656: Append to log took a long time: real 0.165s user 0.001s sys 0.000s
W20260130 02:58:01.893345 18869 rpcz_store.cc:269] 0130 02:58:00.518655 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:58:00.605686 (+ 87031us) service_pool.cc:225] Handling call
0130 02:58:01.869237 (+1263551us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0130 02:58:01.252438 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0130 02:58:01.252838 (+ 400us) write_op.cc:432] Acquiring schema lock in shared mode
0130 02:58:01.252860 (+ 22us) write_op.cc:435] Acquired schema lock
0130 02:58:01.252872 (+ 12us) tablet.cc:660] Decoding operations
0130 02:58:01.253179 (+ 307us) write_op.cc:620] Acquiring the partition lock for write op
0130 02:58:01.253213 (+ 34us) write_op.cc:641] Partition lock acquired for write op
0130 02:58:01.253235 (+ 22us) tablet.cc:683] Acquiring locks for 2 operations
0130 02:58:01.253492 (+ 257us) tablet.cc:699] Row locks acquired
0130 02:58:01.253506 (+ 14us) write_op.cc:260] PREPARE: finished
0130 02:58:01.253631 (+ 125us) write_op.cc:270] Start()
0130 02:58:01.253712 (+ 81us) write_op.cc:276] Timestamp: P: 1769741881253547 usec, L: 0
0130 02:58:01.253726 (+ 14us) op_driver.cc:348] REPLICATION: starting
0130 02:58:01.254237 (+ 511us) log.cc:844] Serialized 913 byte log entry
0130 02:58:01.343617 (+ 89380us) op_driver.cc:464] REPLICATION: finished
0130 02:58:01.345434 (+ 1817us) write_op.cc:301] APPLY: starting
0130 02:58:01.345480 (+ 46us) tablet.cc:1365] starting BulkCheckPresence
0130 02:58:01.345586 (+ 106us) tablet.cc:1368] finished BulkCheckPresence
0130 02:58:01.345679 (+ 93us) tablet.cc:1370] starting ApplyRowOperation cycle
0130 02:58:01.345904 (+ 225us) tablet.cc:1381] finished ApplyRowOperation cycle
0130 02:58:01.346034 (+ 130us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0130 02:58:01.346075 (+ 41us) write_op.cc:312] APPLY: finished
0130 02:58:01.346239 (+ 164us) log.cc:844] Serialized 29 byte log entry
0130 02:58:01.346364 (+ 125us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:58:01.346424 (+ 60us) write_op.cc:454] Released schema lock
0130 02:58:01.346454 (+ 30us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"mutex_wait_us":6243,"reported_tablets":21,"child_traces":[["op",{"apply.queue_time_us":1712,"apply.run_cpu_time_us":1225,"apply.run_wall_time_us":7436,"num_ops":2,"prepare.queue_time_us":22855,"prepare.run_cpu_time_us":2008,"prepare.run_wall_time_us":2008,"replication_time_us":89809,"thread_start_us":59862,"threads_started":1}]]}
I20260130 02:58:01.913385 18869 master_service.cc:954] ReplaceTablet: received request to replace tablet 96584ca174b140f7ad1a2ca698baca71 from {username='slave'} at 127.0.0.1:50284
I20260130 02:58:01.916162 19376 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a743a82540014c85aeab0e9d0994b0dc" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74" is_pre_election: true
I20260130 02:58:01.919520 18874 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50284:
table { table_name: "default.Table_14c76bd65820453fb5cade67cf66e752" } new_table_name: "default.Table_42cd032296a0408797e7af3a27a103dc" modify_external_catalogs: true
I20260130 02:58:01.840224 19526 raft_consensus.cc:2468] T 1ca34772ef484c1f809a99d126e28eff P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78c2c46c7520492db35d51c244fb4f74 in term 1.
W20260130 02:58:01.861987 18873 rpcz_store.cc:269] 0130 02:57:59.877223 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:57:59.956818 (+ 79595us) service_pool.cc:225] Handling call
0130 02:58:00.077114 (+120296us) spinlock_profiling.cc:243] Waited 94.1 ms on lock 0x7b5400030554. stack: 00007fb0bd3f7373 00007fb0bd3f7270 00007fb0bcf6c574 0000000000508744 00000000005086d9 00007fb0c44490c0 00007fb0c44e7bba 00007fb0c61e6eac 00007fb0c62e110e 00007fb0c3efe4a4 00007fb0c3efe432 00007fb0c3efe361 00007fb0c3efe2dc 00007fb0c3efd592 00007fb0c21dd34c 00007fb0c21dc786
0130 02:58:01.861428 (+1784314us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0130 02:58:01.009210 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0130 02:58:01.009593 (+ 383us) write_op.cc:432] Acquiring schema lock in shared mode
0130 02:58:01.009616 (+ 23us) write_op.cc:435] Acquired schema lock
0130 02:58:01.009629 (+ 13us) tablet.cc:660] Decoding operations
0130 02:58:01.009927 (+ 298us) write_op.cc:620] Acquiring the partition lock for write op
0130 02:58:01.009966 (+ 39us) write_op.cc:641] Partition lock acquired for write op
0130 02:58:01.009987 (+ 21us) tablet.cc:683] Acquiring locks for 2 operations
0130 02:58:01.010229 (+ 242us) tablet.cc:699] Row locks acquired
0130 02:58:01.010241 (+ 12us) write_op.cc:260] PREPARE: finished
0130 02:58:01.010345 (+ 104us) write_op.cc:270] Start()
0130 02:58:01.010420 (+ 75us) write_op.cc:276] Timestamp: P: 1769741881010305 usec, L: 0
0130 02:58:01.010434 (+ 14us) op_driver.cc:348] REPLICATION: starting
0130 02:58:01.016146 (+ 5712us) spinlock_profiling.cc:243] Waited 4.2 ms on lock 0x7b5400030b18. stack: 00007fb0bd3f7373 00007fb0bd3f7270 00007fb0bcf6c574 0000000000508744 00000000005086d9 00007fb0c5253964 00007fb0c449a669 00007fb0c4456651 00007fb0c445632a 00007fb0c44da5e5 00007fb0c44daa89 00007fb0c494c4d1 00007fb0c494bca8 00007fb0c4950941 00007fb0c49508f9 00007fb0c4950889
0130 02:58:01.016395 (+ 249us) log.cc:844] Serialized 913 byte log entry
0130 02:58:01.033981 (+ 17586us) spinlock_profiling.cc:243] Waited 15.5 ms on lock 0x7b580009fed0. stack: 00007fb0bd3f7373 00007fb0bd3f7270 00007fb0bcf6c574 0000000000508744 00000000005086d9 00007fb0c44490c0 00007fb0c443946c 00007fb0c443fd83 00007fb0c443fd09 00007fb0c443fc99 00007fb0c443fc61 00007fb0c443ee2d 00000000005aa2b4 00000000005aa0e9 00007fb0bd429355 00007fb0bd42b6a1
0130 02:58:01.034263 (+ 282us) op_driver.cc:464] REPLICATION: finished
0130 02:58:01.034548 (+ 285us) write_op.cc:301] APPLY: starting
0130 02:58:01.034640 (+ 92us) tablet.cc:1365] starting BulkCheckPresence
0130 02:58:01.034738 (+ 98us) tablet.cc:1368] finished BulkCheckPresence
0130 02:58:01.034757 (+ 19us) tablet.cc:1370] starting ApplyRowOperation cycle
0130 02:58:01.034996 (+ 239us) tablet.cc:1381] finished ApplyRowOperation cycle
0130 02:58:01.035121 (+ 125us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0130 02:58:01.035164 (+ 43us) write_op.cc:312] APPLY: finished
0130 02:58:01.035371 (+ 207us) log.cc:844] Serialized 29 byte log entry
0130 02:58:01.035507 (+ 136us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:58:01.035572 (+ 65us) write_op.cc:454] Released schema lock
0130 02:58:01.035618 (+ 46us) write_op.cc:341] FINISH: Updating metrics
0130 02:58:01.035858 (+ 240us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:58:01.035880 (+ 22us) write_op.cc:454] Released schema lock
Related trace 'op':
0130 02:58:01.168917 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0130 02:58:01.169274 (+ 357us) write_op.cc:432] Acquiring schema lock in shared mode
0130 02:58:01.169297 (+ 23us) write_op.cc:435] Acquired schema lock
0130 02:58:01.169310 (+ 13us) tablet.cc:660] Decoding operations
0130 02:58:01.169545 (+ 235us) write_op.cc:620] Acquiring the partition lock for write op
0130 02:58:01.169583 (+ 38us) write_op.cc:641] Partition lock acquired for write op
0130 02:58:01.169617 (+ 34us) tablet.cc:683] Acquiring locks for 1 operations
0130 02:58:01.169760 (+ 143us) tablet.cc:699] Row locks acquired
0130 02:58:01.169773 (+ 13us) write_op.cc:260] PREPARE: finished
0130 02:58:01.169875 (+ 102us) write_op.cc:270] Start()
0130 02:58:01.169951 (+ 76us) write_op.cc:276] Timestamp: P: 1769741881169818 usec, L: 0
0130 02:58:01.169965 (+ 14us) op_driver.cc:348] REPLICATION: starting
0130 02:58:01.170418 (+ 453us) log.cc:844] Serialized 356 byte log entry
0130 02:58:01.319246 (+148828us) op_driver.cc:464] REPLICATION: finished
0130 02:58:01.319525 (+ 279us) write_op.cc:301] APPLY: starting
0130 02:58:01.319579 (+ 54us) tablet.cc:1365] starting BulkCheckPresence
0130 02:58:01.319680 (+ 101us) tablet.cc:1368] finished BulkCheckPresence
0130 02:58:01.319762 (+ 82us) tablet.cc:1370] starting ApplyRowOperation cycle
0130 02:58:01.319900 (+ 138us) tablet.cc:1381] finished ApplyRowOperation cycle
0130 02:58:01.320021 (+ 121us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0130 02:58:01.320064 (+ 43us) write_op.cc:312] APPLY: finished
0130 02:58:01.320223 (+ 159us) log.cc:844] Serialized 23 byte log entry
0130 02:58:01.320375 (+ 152us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:58:01.320427 (+ 52us) write_op.cc:454] Released schema lock
0130 02:58:01.320460 (+ 33us) write_op.cc:341] FINISH: Updating metrics
0130 02:58:01.320669 (+ 209us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:58:01.320689 (+ 20us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":28547,"reported_tablets":20,"spinlock_wait_cycles":216427904,"child_traces":[["op",{"apply.queue_time_us":150,"apply.run_cpu_time_us":1258,"apply.run_wall_time_us":1274,"num_ops":2,"prepare.queue_time_us":1215,"prepare.run_cpu_time_us":3449,"prepare.run_wall_time_us":17764,"raft.queue_time_us":149,"raft.run_cpu_time_us":1404,"raft.run_wall_time_us":16527,"replication_time_us":23744,"spinlock_wait_cycles":45231872}],["op",{"apply.queue_time_us":151,"apply.run_cpu_time_us":1100,"apply.run_wall_time_us":1100,"num_ops":1,"prepare.queue_time_us":86,"prepare.run_cpu_time_us":1771,"prepare.run_wall_time_us":1772,"raft.queue_time_us":16098,"raft.run_cpu_time_us":628,"raft.run_wall_time_us":628,"replication_time_us":149182}]]}
I20260130 02:58:01.780397 20009 consensus_queue.cc:237] T 955b9547f49f434699fe1d86b260638c P 78c2c46c7520492db35d51c244fb4f74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:01.929200 20011 raft_consensus.cc:2764] T 1ca34772ef484c1f809a99d126e28eff P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20260130 02:58:01.931149 19313 leader_election.cc:304] T 1ca34772ef484c1f809a99d126e28eff P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 78c2c46c7520492db35d51c244fb4f74, d870164263bb44b6b780a557daea3656; no voters: 371a36d219164974abdf848841528152
I20260130 02:58:01.898828 19742 consensus_queue.cc:1048] T 5b2976108584478a98fd90fe6594af66 P d870164263bb44b6b780a557daea3656 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.033s
I20260130 02:58:01.932369 19758 raft_consensus.cc:2243] T b078934be4bb4fd1915c691eeea9d00f P 78c2c46c7520492db35d51c244fb4f74 [term 1 LEADER]: Raft consensus shutting down.
I20260130 02:58:01.933238 19758 raft_consensus.cc:2272] T b078934be4bb4fd1915c691eeea9d00f P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260130 02:58:01.932395 19856 raft_consensus.cc:2804] T 1ca34772ef484c1f809a99d126e28eff P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Leader election won for term 1
I20260130 02:58:01.935042 19856 raft_consensus.cc:697] T 1ca34772ef484c1f809a99d126e28eff P 78c2c46c7520492db35d51c244fb4f74 [term 1 LEADER]: Becoming Leader. State: Replica: 78c2c46c7520492db35d51c244fb4f74, State: Running, Role: LEADER
I20260130 02:58:01.935756 19856 consensus_queue.cc:237] T 1ca34772ef484c1f809a99d126e28eff P 78c2c46c7520492db35d51c244fb4f74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:01.939690 19356 tablet_service.cc:1505] Processing CreateTablet for tablet 6573198830ad45628e71952a7d9c00a0 (DEFAULT_TABLE table=default.Table_1c2b251034c0418cba750ae4d1308b9f [id=9a5fe99c9417426dba0a12441e7191c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:01.940853 19356 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6573198830ad45628e71952a7d9c00a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:01.944100 19758 ts_tablet_manager.cc:1916] T b078934be4bb4fd1915c691eeea9d00f P 78c2c46c7520492db35d51c244fb4f74: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260130 02:58:01.910526 19198 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 341041cb0be2456e88403367c0814704. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:01.933677 19199 tablet_service.cc:1505] Processing CreateTablet for tablet 6573198830ad45628e71952a7d9c00a0 (DEFAULT_TABLE table=default.Table_1c2b251034c0418cba750ae4d1308b9f [id=9a5fe99c9417426dba0a12441e7191c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:01.946967 19199 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6573198830ad45628e71952a7d9c00a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:01.948410 19204 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3cf5154d8cc44f2bb12978202699c193. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:01.967779 19354 tablet_service.cc:1505] Processing CreateTablet for tablet 3cf5154d8cc44f2bb12978202699c193 (DEFAULT_TABLE table=default.Table_bb334df0beee420e883c3225d93892c0 [id=b7b1d250632741a2ad0ed194aae9b197]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:01.969182 19354 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3cf5154d8cc44f2bb12978202699c193. 1 dirs total, 0 dirs full, 0 dirs failed
W20260130 02:58:01.971138 19159 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.100.66:40767, user_credentials={real_user=slave}} blocked reactor thread for 103426us
W20260130 02:58:01.972743 19159 consensus_peers.cc:597] T 96584ca174b140f7ad1a2ca698baca71 P 371a36d219164974abdf848841528152 -> Peer 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767): Couldn't send request to peer 78c2c46c7520492db35d51c244fb4f74. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 96584ca174b140f7ad1a2ca698baca71. This is attempt 1: this message will repeat every 5th retry.
I20260130 02:58:01.983881 19840 consensus_queue.cc:260] T a743a82540014c85aeab0e9d0994b0dc P 78c2c46c7520492db35d51c244fb4f74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:01.987708 19376 raft_consensus.cc:2468] T a743a82540014c85aeab0e9d0994b0dc P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d870164263bb44b6b780a557daea3656 in term 0.
I20260130 02:58:01.989893 19899 raft_consensus.cc:493] T a743a82540014c85aeab0e9d0994b0dc P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:01.990355 19899 raft_consensus.cc:515] T a743a82540014c85aeab0e9d0994b0dc P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:01.993698 19224 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a743a82540014c85aeab0e9d0994b0dc" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152" is_pre_election: true
W20260130 02:58:01.995057 19314 leader_election.cc:343] T a743a82540014c85aeab0e9d0994b0dc P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:01.995553 19899 leader_election.cc:290] T a743a82540014c85aeab0e9d0994b0dc P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), d870164263bb44b6b780a557daea3656 (127.18.100.67:39571)
I20260130 02:58:02.007480 19840 ts_tablet_manager.cc:1434] T a743a82540014c85aeab0e9d0994b0dc P 78c2c46c7520492db35d51c244fb4f74: Time spent starting tablet: real 0.354s user 0.006s sys 0.000s
I20260130 02:58:02.008363 19840 tablet_bootstrap.cc:492] T b3f3177306914e3c8b253db54ee30fb6 P 78c2c46c7520492db35d51c244fb4f74: Bootstrap starting.
I20260130 02:58:02.018028 20012 raft_consensus.cc:515] T b7b596f944be4daf953b82b65d5cec2a P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:02.025267 18869 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50284:
table { table_name: "default.Table_e3b63d7d5d874900aac850654d98d62e" } modify_external_catalogs: true
I20260130 02:58:02.032624 18867 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50284:
name: "default.Table_109c8b6c17d4442b939bde9441d0476f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260130 02:58:02.034119 18867 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_109c8b6c17d4442b939bde9441d0476f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260130 02:58:02.031164 19159 outbound_call.cc:321] RPC callback for RPC call kudu.master.MasterService.Ping -> {remote=127.18.100.124:34629, user_credentials={real_user=slave}} blocked reactor thread for 57726.9us
I20260130 02:58:02.038815 18869 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50284:
table { table_name: "default.Table_e3b63d7d5d874900aac850654d98d62e" } modify_external_catalogs: true
W20260130 02:58:02.044401 18862 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet 7d76046ee6c8420fa4964decbe8209c1 (table default.Table_55e0ba23f44146339c7c6928440d4948 [id=4c70005d1005479fa6c3a6cb6f5ce658], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 7d76046ee6c8420fa4964decbe8209c1 consensus state has no leader
I20260130 02:58:02.046998 18868 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50284:
table { table_name: "default.Table_0658e13596f44598bb5a108797bc06fc" } new_table_name: "default.Table_526677c8ef4144bd9de7a54a5e78e2be" modify_external_catalogs: true
I20260130 02:58:02.050468 20034 tablet.cc:1722] T bac739bf4cac4272975f5587ad9c71ff P 78c2c46c7520492db35d51c244fb4f74: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260130 02:58:02.056649 19358 tablet_service.cc:1505] Processing CreateTablet for tablet 3b148691090e4ee581d46f9845a6e49a (DEFAULT_TABLE table=default.Table_8edea1fae34d4d18b54b0f801a145225 [id=2009496525c94d7f8ee6ce59694b652d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:02.057643 19358 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3b148691090e4ee581d46f9845a6e49a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:02.075464 19202 tablet_service.cc:1505] Processing CreateTablet for tablet b8c579ff5fa84651ba9125457d927162 (DEFAULT_TABLE table=default.Table_3046889233c24f0a8e8058e1591d63b8 [id=edd6505c0bf3462cbf65c3df5649c16c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:02.076695 19202 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b8c579ff5fa84651ba9125457d927162. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:02.078825 19357 tablet_service.cc:1505] Processing CreateTablet for tablet b8c579ff5fa84651ba9125457d927162 (DEFAULT_TABLE table=default.Table_3046889233c24f0a8e8058e1591d63b8 [id=edd6505c0bf3462cbf65c3df5649c16c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:02.087286 19357 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b8c579ff5fa84651ba9125457d927162. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:02.089799 19199 tablet_service.cc:1505] Processing CreateTablet for tablet 3b4908bd1a634ca0907234dd3c39e1c3 (DEFAULT_TABLE table=default.Table_0ad5d205dfea4344a759522f45a55475 [id=5b7d0747e1bd438ea7e2f10abd339c2c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:02.088201 19360 tablet_service.cc:1505] Processing CreateTablet for tablet 54be22352e404ff9b798a075693f7f21 (DEFAULT_TABLE table=default.Table_50c3bb8b3ede494ba9b9f826a2adba72 [id=17150cb73e374ff9a1e92c43e0c3f2c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:02.091739 19199 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3b4908bd1a634ca0907234dd3c39e1c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:02.094830 19201 tablet_service.cc:1505] Processing CreateTablet for tablet 3b148691090e4ee581d46f9845a6e49a (DEFAULT_TABLE table=default.Table_8edea1fae34d4d18b54b0f801a145225 [id=2009496525c94d7f8ee6ce59694b652d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:02.096115 19201 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3b148691090e4ee581d46f9845a6e49a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:02.100999 19840 tablet_bootstrap.cc:654] T b3f3177306914e3c8b253db54ee30fb6 P 78c2c46c7520492db35d51c244fb4f74: Neither blocks nor log segments found. Creating new log.
W20260130 02:58:02.021471 19463 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.100.65:42021, user_credentials={real_user=slave}} blocked reactor thread for 284448us
I20260130 02:58:02.112721 18872 master_service.cc:954] ReplaceTablet: received request to replace tablet 37470ce4356f4542b621e21698551209 from {username='slave'} at 127.0.0.1:50284
W20260130 02:58:02.106423 20022 log.cc:927] Time spent T d701152960b14396a6ac1bd10480d2b6 P d870164263bb44b6b780a557daea3656: Append to log took a long time: real 0.084s user 0.000s sys 0.000s
I20260130 02:58:02.122301 19758 ts_tablet_manager.cc:1929] T b078934be4bb4fd1915c691eeea9d00f P 78c2c46c7520492db35d51c244fb4f74: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20260130 02:58:02.127638 20025 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.AlterSchema from 127.0.0.1:42870 (request call id 94) took 1146 ms. Trace:
I20260130 02:58:02.124146 20042 tablet.cc:1722] T bac739bf4cac4272975f5587ad9c71ff P 371a36d219164974abdf848841528152: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260130 02:58:02.129730 20025 rpcz_store.cc:276] 0130 02:58:00.981503 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:58:00.981650 (+ 147us) service_pool.cc:225] Handling call
0130 02:58:02.127575 (+1145925us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0130 02:58:00.982682 (+ 0us) alter_schema_op.cc:104] PREPARE ALTER-SCHEMA: Starting
0130 02:58:00.983022 (+ 340us) alter_schema_op.cc:63] Acquiring schema lock in exclusive mode
0130 02:58:00.983051 (+ 29us) alter_schema_op.cc:66] Acquired schema lock
0130 02:58:00.983091 (+ 40us) alter_schema_op.cc:119] PREPARE ALTER-SCHEMA: finished
0130 02:58:00.983187 (+ 96us) alter_schema_op.cc:128] START. Timestamp: 1769741880983081
0130 02:58:00.983205 (+ 18us) op_driver.cc:348] REPLICATION: starting
0130 02:58:00.983681 (+ 476us) log.cc:844] Serialized 173 byte log entry
0130 02:58:01.633954 (+650273us) op_driver.cc:464] REPLICATION: finished
0130 02:58:01.816286 (+182332us) alter_schema_op.cc:133] APPLY ALTER-SCHEMA: Starting
0130 02:58:01.817463 (+ 1177us) log.cc:844] Serialized 5 byte log entry
0130 02:58:02.126551 (+309088us) tablet_metadata.cc:641] Metadata flushed
0130 02:58:02.127368 (+ 817us) log.cc:844] Serialized 14 byte log entry
0130 02:58:02.127454 (+ 86us) alter_schema_op.cc:73] Released schema lock
0130 02:58:02.127482 (+ 28us) alter_schema_op.cc:186] AlterSchemaCommitCallback: making alter schema visible
Metrics: {"child_traces":[["op",{"apply.queue_time_us":182159,"prepare.queue_time_us":154,"prepare.run_cpu_time_us":13473,"prepare.run_wall_time_us":194150,"replication_time_us":650646,"thread_start_us":376550,"threads_started":2,"wal-append.queue_time_us":193167}]]}
I20260130 02:58:02.141739 20025 tablet.cc:1722] T e364f4e70e5749e083e14e7fa7416ff0 P 371a36d219164974abdf848841528152: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260130 02:58:02.162418 19500 tablet_service.cc:1505] Processing CreateTablet for tablet 3b4908bd1a634ca0907234dd3c39e1c3 (DEFAULT_TABLE table=default.Table_0ad5d205dfea4344a759522f45a55475 [id=5b7d0747e1bd438ea7e2f10abd339c2c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:02.174291 19506 tablet_service.cc:1505] Processing CreateTablet for tablet 6573198830ad45628e71952a7d9c00a0 (DEFAULT_TABLE table=default.Table_1c2b251034c0418cba750ae4d1308b9f [id=9a5fe99c9417426dba0a12441e7191c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:02.175340 19506 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6573198830ad45628e71952a7d9c00a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:02.148643 19507 tablet_service.cc:1505] Processing CreateTablet for tablet 3cf5154d8cc44f2bb12978202699c193 (DEFAULT_TABLE table=default.Table_bb334df0beee420e883c3225d93892c0 [id=b7b1d250632741a2ad0ed194aae9b197]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:02.183102 19204 tablet_service.cc:1505] Processing CreateTablet for tablet 54be22352e404ff9b798a075693f7f21 (DEFAULT_TABLE table=default.Table_50c3bb8b3ede494ba9b9f826a2adba72 [id=17150cb73e374ff9a1e92c43e0c3f2c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:02.192456 18867 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50284:
name: "default.Table_4c2511f2cf3f40ee9c134faafc7c8383"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260130 02:58:02.128510 19360 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 54be22352e404ff9b798a075693f7f21. 1 dirs total, 0 dirs full, 0 dirs failed
W20260130 02:58:02.194034 18867 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4c2511f2cf3f40ee9c134faafc7c8383 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260130 02:58:02.108670 19526 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a743a82540014c85aeab0e9d0994b0dc" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
I20260130 02:58:02.130915 20047 tablet.cc:1722] T e364f4e70e5749e083e14e7fa7416ff0 P 78c2c46c7520492db35d51c244fb4f74: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260130 02:58:02.133011 19352 tablet_service.cc:1505] Processing CreateTablet for tablet 3b4908bd1a634ca0907234dd3c39e1c3 (DEFAULT_TABLE table=default.Table_0ad5d205dfea4344a759522f45a55475 [id=5b7d0747e1bd438ea7e2f10abd339c2c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:02.143116 19840 tablet_bootstrap.cc:492] T b3f3177306914e3c8b253db54ee30fb6 P 78c2c46c7520492db35d51c244fb4f74: No bootstrap required, opened a new log
I20260130 02:58:02.207376 19840 ts_tablet_manager.cc:1403] T b3f3177306914e3c8b253db54ee30fb6 P 78c2c46c7520492db35d51c244fb4f74: Time spent bootstrapping tablet: real 0.199s user 0.010s sys 0.002s
I20260130 02:58:02.209583 19840 raft_consensus.cc:359] T b3f3177306914e3c8b253db54ee30fb6 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:02.210197 19840 raft_consensus.cc:385] T b3f3177306914e3c8b253db54ee30fb6 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:02.210425 19840 raft_consensus.cc:740] T b3f3177306914e3c8b253db54ee30fb6 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78c2c46c7520492db35d51c244fb4f74, State: Initialized, Role: FOLLOWER
I20260130 02:58:02.211371 19840 consensus_queue.cc:260] T b3f3177306914e3c8b253db54ee30fb6 P 78c2c46c7520492db35d51c244fb4f74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:02.219043 19840 ts_tablet_manager.cc:1434] T b3f3177306914e3c8b253db54ee30fb6 P 78c2c46c7520492db35d51c244fb4f74: Time spent starting tablet: real 0.011s user 0.005s sys 0.000s
I20260130 02:58:02.220367 19352 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3b4908bd1a634ca0907234dd3c39e1c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:02.222723 19758 log.cc:1199] T b078934be4bb4fd1915c691eeea9d00f P 78c2c46c7520492db35d51c244fb4f74: Deleting WAL directory at /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-1/wal/wals/b078934be4bb4fd1915c691eeea9d00f
I20260130 02:58:02.226981 19507 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3cf5154d8cc44f2bb12978202699c193. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:02.228314 19204 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 54be22352e404ff9b798a075693f7f21. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:02.228209 19499 tablet_service.cc:1505] Processing CreateTablet for tablet 3b148691090e4ee581d46f9845a6e49a (DEFAULT_TABLE table=default.Table_8edea1fae34d4d18b54b0f801a145225 [id=2009496525c94d7f8ee6ce59694b652d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:02.233243 18872 catalog_manager.cc:5941] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4: Sending DeleteTablet for 3 replicas of tablet 37470ce4356f4542b621e21698551209
I20260130 02:58:02.234489 18872 catalog_manager.cc:6724] ReplaceTablet: tablet 37470ce4356f4542b621e21698551209 deleted and replaced by tablet 7caf7cbad5054396b5617094e4d536a0
I20260130 02:58:02.234767 19758 ts_tablet_manager.cc:1950] T b078934be4bb4fd1915c691eeea9d00f P 78c2c46c7520492db35d51c244fb4f74: Deleting consensus metadata
I20260130 02:58:02.235374 18872 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50284:
name: "default.Table_b85cb876f25740bfaaab01949ac26b38"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260130 02:58:02.236816 18872 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b85cb876f25740bfaaab01949ac26b38 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260130 02:58:02.237475 20012 leader_election.cc:290] T b7b596f944be4daf953b82b65d5cec2a P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 election: Requested vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767)
I20260130 02:58:02.205196 19526 raft_consensus.cc:2468] T a743a82540014c85aeab0e9d0994b0dc P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78c2c46c7520492db35d51c244fb4f74 in term 0.
I20260130 02:58:02.238898 19313 leader_election.cc:304] T a743a82540014c85aeab0e9d0994b0dc P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 78c2c46c7520492db35d51c244fb4f74, d870164263bb44b6b780a557daea3656; no voters: 371a36d219164974abdf848841528152
I20260130 02:58:02.239555 19856 raft_consensus.cc:2804] T a743a82540014c85aeab0e9d0994b0dc P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260130 02:58:02.239900 19856 raft_consensus.cc:493] T a743a82540014c85aeab0e9d0994b0dc P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260130 02:58:02.240250 19856 raft_consensus.cc:3060] T a743a82540014c85aeab0e9d0994b0dc P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:02.241367 18860 catalog_manager.cc:4985] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767): tablet b078934be4bb4fd1915c691eeea9d00f (table default.Table_b63ca4e4060f48e893b0099fb6433f04 [id=720decfb55c24f34983ef636f06e3169]) successfully deleted
I20260130 02:58:02.245492 19498 tablet_service.cc:1505] Processing CreateTablet for tablet b8c579ff5fa84651ba9125457d927162 (DEFAULT_TABLE table=default.Table_3046889233c24f0a8e8058e1591d63b8 [id=edd6505c0bf3462cbf65c3df5649c16c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:02.247777 19856 raft_consensus.cc:515] T a743a82540014c85aeab0e9d0994b0dc P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:02.249598 19807 tablet_bootstrap.cc:492] T 1ca34772ef484c1f809a99d126e28eff P 371a36d219164974abdf848841528152: No bootstrap required, opened a new log
I20260130 02:58:02.249993 19807 ts_tablet_manager.cc:1403] T 1ca34772ef484c1f809a99d126e28eff P 371a36d219164974abdf848841528152: Time spent bootstrapping tablet: real 0.740s user 0.009s sys 0.006s
I20260130 02:58:02.250054 19856 leader_election.cc:290] T a743a82540014c85aeab0e9d0994b0dc P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 election: Requested vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), d870164263bb44b6b780a557daea3656 (127.18.100.67:39571)
I20260130 02:58:02.250700 19358 tablet_service.cc:1552] Processing DeleteTablet for tablet 37470ce4356f4542b621e21698551209 with delete_type TABLET_DATA_DELETED (replaced by tablet 7caf7cbad5054396b5617094e4d536a0) from {username='slave'} at 127.0.0.1:60110
I20260130 02:58:02.251895 19500 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3b4908bd1a634ca0907234dd3c39e1c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:02.251963 19840 tablet_bootstrap.cc:492] T d3ba2df0b15c44f882070570f91b254d P 78c2c46c7520492db35d51c244fb4f74: Bootstrap starting.
I20260130 02:58:02.252193 19807 raft_consensus.cc:359] T 1ca34772ef484c1f809a99d126e28eff P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:02.252785 19807 raft_consensus.cc:385] T 1ca34772ef484c1f809a99d126e28eff P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:02.252997 19224 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a743a82540014c85aeab0e9d0994b0dc" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152"
W20260130 02:58:02.254300 19314 leader_election.cc:343] T a743a82540014c85aeab0e9d0994b0dc P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:02.259933 18867 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50284:
name: "default.Table_8896365e792f4d24b2c16c2d269b0a93"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260130 02:58:02.261742 18867 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8896365e792f4d24b2c16c2d269b0a93 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260130 02:58:02.262729 19199 tablet_service.cc:1552] Processing DeleteTablet for tablet 37470ce4356f4542b621e21698551209 with delete_type TABLET_DATA_DELETED (replaced by tablet 7caf7cbad5054396b5617094e4d536a0) from {username='slave'} at 127.0.0.1:42870
I20260130 02:58:02.264587 19526 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a743a82540014c85aeab0e9d0994b0dc" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656"
I20260130 02:58:02.265154 19526 raft_consensus.cc:3060] T a743a82540014c85aeab0e9d0994b0dc P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:02.273317 19840 tablet_bootstrap.cc:654] T d3ba2df0b15c44f882070570f91b254d P 78c2c46c7520492db35d51c244fb4f74: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:02.275697 18873 catalog_manager.cc:5746] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767) does not have the latest schema for tablet 15918c90ffdc43a5a38b2950e9834188 (table default.Table_333d2efbf068485a871f566d50e563ce [id=8b7db40afb894ae99625a756c75ef1de]). Expected version 1 got 0
I20260130 02:58:02.278055 19499 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3b148691090e4ee581d46f9845a6e49a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:02.288410 19498 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b8c579ff5fa84651ba9125457d927162. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:02.293699 19856 raft_consensus.cc:493] T b3f3177306914e3c8b253db54ee30fb6 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:02.294137 19856 raft_consensus.cc:515] T b3f3177306914e3c8b253db54ee30fb6 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:02.295403 19526 raft_consensus.cc:2468] T a743a82540014c85aeab0e9d0994b0dc P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78c2c46c7520492db35d51c244fb4f74 in term 1.
I20260130 02:58:02.296373 19313 leader_election.cc:304] T a743a82540014c85aeab0e9d0994b0dc P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 78c2c46c7520492db35d51c244fb4f74, d870164263bb44b6b780a557daea3656; no voters: 371a36d219164974abdf848841528152
I20260130 02:58:02.296895 19224 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b3f3177306914e3c8b253db54ee30fb6" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152" is_pre_election: true
W20260130 02:58:02.298285 19314 leader_election.cc:343] T b3f3177306914e3c8b253db54ee30fb6 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:02.298918 20009 raft_consensus.cc:2804] T a743a82540014c85aeab0e9d0994b0dc P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Leader election won for term 1
I20260130 02:58:02.298879 20042 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.AlterSchema from 127.0.0.1:42870 (request call id 97) took 1132 ms. Trace:
W20260130 02:58:02.298863 19463 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.100.65:42021, user_credentials={real_user=slave}} blocked reactor thread for 135206us
I20260130 02:58:02.300436 20009 raft_consensus.cc:697] T a743a82540014c85aeab0e9d0994b0dc P 78c2c46c7520492db35d51c244fb4f74 [term 1 LEADER]: Becoming Leader. State: Replica: 78c2c46c7520492db35d51c244fb4f74, State: Running, Role: LEADER
I20260130 02:58:02.300298 19974 consensus_queue.cc:1048] T 15918c90ffdc43a5a38b2950e9834188 P 371a36d219164974abdf848841528152 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260130 02:58:02.301167 20009 consensus_queue.cc:237] T a743a82540014c85aeab0e9d0994b0dc P 78c2c46c7520492db35d51c244fb4f74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:02.304186 19625 tablet_bootstrap.cc:492] T b3f3177306914e3c8b253db54ee30fb6 P d870164263bb44b6b780a557daea3656: No bootstrap required, opened a new log
I20260130 02:58:02.304618 19625 ts_tablet_manager.cc:1403] T b3f3177306914e3c8b253db54ee30fb6 P d870164263bb44b6b780a557daea3656: Time spent bootstrapping tablet: real 0.617s user 0.016s sys 0.010s
I20260130 02:58:02.305987 18873 catalog_manager.cc:5746] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767) does not have the latest schema for tablet df2c8989d6c1480c97baa23db01141eb (table default.Table_e84c6277dfc944059f62cbe0d6b937ee [id=747af828147149efb7b7b8d33c49286d]). Expected version 2 got 1
I20260130 02:58:02.299229 20042 rpcz_store.cc:276] 0130 02:58:01.166408 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:58:01.166567 (+ 159us) service_pool.cc:225] Handling call
0130 02:58:02.298808 (+1132241us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0130 02:58:01.176967 (+ 0us) alter_schema_op.cc:104] PREPARE ALTER-SCHEMA: Starting
0130 02:58:01.177327 (+ 360us) alter_schema_op.cc:63] Acquiring schema lock in exclusive mode
0130 02:58:01.177364 (+ 37us) alter_schema_op.cc:66] Acquired schema lock
0130 02:58:01.177425 (+ 61us) alter_schema_op.cc:119] PREPARE ALTER-SCHEMA: finished
0130 02:58:01.177571 (+ 146us) alter_schema_op.cc:128] START. Timestamp: 1769741881160341
0130 02:58:01.177593 (+ 22us) op_driver.cc:348] REPLICATION: starting
0130 02:58:01.178092 (+ 499us) log.cc:844] Serialized 173 byte log entry
0130 02:58:01.968860 (+790768us) op_driver.cc:464] REPLICATION: finished
0130 02:58:02.081539 (+112679us) alter_schema_op.cc:133] APPLY ALTER-SCHEMA: Starting
0130 02:58:02.132020 (+ 50481us) log.cc:844] Serialized 5 byte log entry
0130 02:58:02.280386 (+148366us) tablet_metadata.cc:641] Metadata flushed
0130 02:58:02.281658 (+ 1272us) log.cc:844] Serialized 14 byte log entry
0130 02:58:02.298671 (+ 17013us) alter_schema_op.cc:73] Released schema lock
0130 02:58:02.298711 (+ 40us) alter_schema_op.cc:186] AlterSchemaCommitCallback: making alter schema visible
Metrics: {"child_traces":[["op",{"apply.queue_time_us":112497,"prepare.queue_time_us":9558,"prepare.run_cpu_time_us":9516,"prepare.run_wall_time_us":206816,"raft.queue_time_us":1331,"raft.run_cpu_time_us":793,"raft.run_wall_time_us":792,"replication_time_us":791160,"thread_start_us":457265,"threads_started":3,"wal-append.queue_time_us":203447}]]}
I20260130 02:58:02.253026 19807 raft_consensus.cc:740] T 1ca34772ef484c1f809a99d126e28eff P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 371a36d219164974abdf848841528152, State: Initialized, Role: FOLLOWER
I20260130 02:58:02.332700 19807 consensus_queue.cc:260] T 1ca34772ef484c1f809a99d126e28eff P 371a36d219164974abdf848841528152 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:02.340765 18873 catalog_manager.cc:5746] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767) does not have the latest schema for tablet e364f4e70e5749e083e14e7fa7416ff0 (table default.Table_bc11cb1bce0348ed9ea0912b3ee521b9 [id=df7d3d68e36142ff8564e63ffe0b8797]). Expected version 1 got 0
I20260130 02:58:02.349634 20009 consensus_queue.cc:1048] T c7c621b99deb457998c44ab1644d3939 P 78c2c46c7520492db35d51c244fb4f74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260130 02:58:02.342518 18873 catalog_manager.cc:5654] T c7c621b99deb457998c44ab1644d3939 P 78c2c46c7520492db35d51c244fb4f74 reported cstate change: term changed from 0 to 1, leader changed from <none> to 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66). New cstate: current_term: 1 leader_uuid: "78c2c46c7520492db35d51c244fb4f74" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } health_report { overall_health: UNKNOWN } } }
I20260130 02:58:02.354290 19758 tablet_replica.cc:333] T 7a902d0d571a42bb82ebffd48f7eca3f P 78c2c46c7520492db35d51c244fb4f74: stopping tablet replica
W20260130 02:58:02.353518 18860 catalog_manager.cc:4952] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767): delete failed for tablet 4035b8a330844dd28af9f3244bf8eb9a because the tablet was not found. No further retry: Not found: Tablet not found: 4035b8a330844dd28af9f3244bf8eb9a
I20260130 02:58:02.355067 19758 raft_consensus.cc:2243] T 7a902d0d571a42bb82ebffd48f7eca3f P 78c2c46c7520492db35d51c244fb4f74 [term 1 LEADER]: Raft consensus shutting down.
W20260130 02:58:02.355207 18860 catalog_manager.cc:4952] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767): delete failed for tablet a46702bef0e6442d85e4ca49a5f7cfbb because the tablet was not found. No further retry: Not found: Tablet not found: a46702bef0e6442d85e4ca49a5f7cfbb
I20260130 02:58:02.355809 19758 raft_consensus.cc:2272] T 7a902d0d571a42bb82ebffd48f7eca3f P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260130 02:58:02.358721 19462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.100.66:40767, user_credentials={real_user=slave}} blocked reactor thread for 442937us
I20260130 02:58:02.359102 19856 leader_election.cc:290] T b3f3177306914e3c8b253db54ee30fb6 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), d870164263bb44b6b780a557daea3656 (127.18.100.67:39571)
I20260130 02:58:02.360654 19462 leader_election.cc:304] T a743a82540014c85aeab0e9d0994b0dc P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78c2c46c7520492db35d51c244fb4f74, d870164263bb44b6b780a557daea3656; no voters:
I20260130 02:58:02.361124 19377 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b7b596f944be4daf953b82b65d5cec2a" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74"
I20260130 02:58:02.361749 19377 raft_consensus.cc:3060] T b7b596f944be4daf953b82b65d5cec2a P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:02.362675 19527 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b3f3177306914e3c8b253db54ee30fb6" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
I20260130 02:58:02.368014 19971 raft_consensus.cc:2764] T a743a82540014c85aeab0e9d0994b0dc P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20260130 02:58:02.371614 19625 raft_consensus.cc:359] T b3f3177306914e3c8b253db54ee30fb6 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:02.372228 19625 raft_consensus.cc:385] T b3f3177306914e3c8b253db54ee30fb6 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:02.372484 19625 raft_consensus.cc:740] T b3f3177306914e3c8b253db54ee30fb6 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d870164263bb44b6b780a557daea3656, State: Initialized, Role: FOLLOWER
I20260130 02:58:02.373014 19625 consensus_queue.cc:260] T b3f3177306914e3c8b253db54ee30fb6 P d870164263bb44b6b780a557daea3656 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:02.374653 19377 raft_consensus.cc:2468] T b7b596f944be4daf953b82b65d5cec2a P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d870164263bb44b6b780a557daea3656 in term 1.
I20260130 02:58:02.376330 19527 raft_consensus.cc:2468] T b3f3177306914e3c8b253db54ee30fb6 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78c2c46c7520492db35d51c244fb4f74 in term 0.
I20260130 02:58:02.377255 19313 leader_election.cc:304] T b3f3177306914e3c8b253db54ee30fb6 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 78c2c46c7520492db35d51c244fb4f74, d870164263bb44b6b780a557daea3656; no voters: 371a36d219164974abdf848841528152
I20260130 02:58:02.377983 20009 raft_consensus.cc:2804] T b3f3177306914e3c8b253db54ee30fb6 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260130 02:58:02.378378 20009 raft_consensus.cc:493] T b3f3177306914e3c8b253db54ee30fb6 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260130 02:58:02.378729 20009 raft_consensus.cc:3060] T b3f3177306914e3c8b253db54ee30fb6 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:02.378885 19625 ts_tablet_manager.cc:1434] T b3f3177306914e3c8b253db54ee30fb6 P d870164263bb44b6b780a557daea3656: Time spent starting tablet: real 0.074s user 0.005s sys 0.002s
I20260130 02:58:02.388191 20009 raft_consensus.cc:515] T b3f3177306914e3c8b253db54ee30fb6 P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:02.391396 19462 leader_election.cc:304] T b7b596f944be4daf953b82b65d5cec2a P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78c2c46c7520492db35d51c244fb4f74, d870164263bb44b6b780a557daea3656; no voters:
I20260130 02:58:02.396672 20009 leader_election.cc:290] T b3f3177306914e3c8b253db54ee30fb6 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 election: Requested vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), d870164263bb44b6b780a557daea3656 (127.18.100.67:39571)
I20260130 02:58:02.400565 19527 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b3f3177306914e3c8b253db54ee30fb6" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656"
I20260130 02:58:02.401096 19527 raft_consensus.cc:3060] T b3f3177306914e3c8b253db54ee30fb6 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:02.403460 19225 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b3f3177306914e3c8b253db54ee30fb6" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152"
I20260130 02:58:02.405193 19625 tablet_bootstrap.cc:492] T ec28ff12927d450ead16d26cf016138d P d870164263bb44b6b780a557daea3656: Bootstrap starting.
W20260130 02:58:02.405187 19314 leader_election.cc:343] T b3f3177306914e3c8b253db54ee30fb6 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:02.407020 18872 master_service.cc:954] ReplaceTablet: received request to replace tablet bd43c5d148514836869a8ac648c8ad11 from {username='slave'} at 127.0.0.1:50284
I20260130 02:58:02.407037 19524 raft_consensus.cc:3060] T c7c621b99deb457998c44ab1644d3939 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:02.408601 18872 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50284:
name: "default.Table_6a7faee4c75c4eb7862fb4c05afc17cb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260130 02:58:02.410154 18872 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6a7faee4c75c4eb7862fb4c05afc17cb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260130 02:58:02.298432 19947 ts_tablet_manager.cc:1929] T 7a902d0d571a42bb82ebffd48f7eca3f P d870164263bb44b6b780a557daea3656: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260130 02:58:02.411334 19225 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bd43c5d148514836869a8ac648c8ad11" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152" is_pre_election: true
I20260130 02:58:02.414273 19223 raft_consensus.cc:1275] T 1bbc55bbf2254d8bb6b721d7e8f69d85 P 371a36d219164974abdf848841528152 [term 2 FOLLOWER]: Refusing update from remote peer d870164263bb44b6b780a557daea3656: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20260130 02:58:02.416117 19223 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a743a82540014c85aeab0e9d0994b0dc" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152" is_pre_election: true
I20260130 02:58:02.419368 19221 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b7b596f944be4daf953b82b65d5cec2a" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152"
I20260130 02:58:02.419874 19221 raft_consensus.cc:3060] T b7b596f944be4daf953b82b65d5cec2a P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:02.435039 19947 log.cc:1199] T 7a902d0d571a42bb82ebffd48f7eca3f P d870164263bb44b6b780a557daea3656: Deleting WAL directory at /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-2/wal/wals/7a902d0d571a42bb82ebffd48f7eca3f
I20260130 02:58:02.437219 19840 tablet_bootstrap.cc:492] T d3ba2df0b15c44f882070570f91b254d P 78c2c46c7520492db35d51c244fb4f74: No bootstrap required, opened a new log
I20260130 02:58:02.437306 19505 tablet_service.cc:1505] Processing CreateTablet for tablet 54be22352e404ff9b798a075693f7f21 (DEFAULT_TABLE table=default.Table_50c3bb8b3ede494ba9b9f826a2adba72 [id=17150cb73e374ff9a1e92c43e0c3f2c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:02.437680 19840 ts_tablet_manager.cc:1403] T d3ba2df0b15c44f882070570f91b254d P 78c2c46c7520492db35d51c244fb4f74: Time spent bootstrapping tablet: real 0.186s user 0.010s sys 0.000s
I20260130 02:58:02.438330 19505 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 54be22352e404ff9b798a075693f7f21. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:02.438630 19758 ts_tablet_manager.cc:1916] T 7a902d0d571a42bb82ebffd48f7eca3f P 78c2c46c7520492db35d51c244fb4f74: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260130 02:58:02.438892 19462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.100.66:40767, user_credentials={real_user=slave}} blocked reactor thread for 47574.1us
I20260130 02:58:02.442358 20044 raft_consensus.cc:2804] T b7b596f944be4daf953b82b65d5cec2a P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Leader election won for term 1
I20260130 02:58:02.444272 19502 tablet_service.cc:1552] Processing DeleteTablet for tablet 37470ce4356f4542b621e21698551209 with delete_type TABLET_DATA_DELETED (replaced by tablet 7caf7cbad5054396b5617094e4d536a0) from {username='slave'} at 127.0.0.1:59068
I20260130 02:58:02.440145 19840 raft_consensus.cc:359] T d3ba2df0b15c44f882070570f91b254d P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:02.459511 20044 raft_consensus.cc:697] T b7b596f944be4daf953b82b65d5cec2a P d870164263bb44b6b780a557daea3656 [term 1 LEADER]: Becoming Leader. State: Replica: d870164263bb44b6b780a557daea3656, State: Running, Role: LEADER
I20260130 02:58:02.458496 19840 raft_consensus.cc:385] T d3ba2df0b15c44f882070570f91b254d P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:02.460212 19840 raft_consensus.cc:740] T d3ba2df0b15c44f882070570f91b254d P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78c2c46c7520492db35d51c244fb4f74, State: Initialized, Role: FOLLOWER
I20260130 02:58:02.461107 19221 raft_consensus.cc:2468] T b7b596f944be4daf953b82b65d5cec2a P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d870164263bb44b6b780a557daea3656 in term 1.
I20260130 02:58:02.461993 19840 consensus_queue.cc:260] T d3ba2df0b15c44f882070570f91b254d P 78c2c46c7520492db35d51c244fb4f74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:02.467499 19840 ts_tablet_manager.cc:1434] T d3ba2df0b15c44f882070570f91b254d P 78c2c46c7520492db35d51c244fb4f74: Time spent starting tablet: real 0.029s user 0.004s sys 0.001s
W20260130 02:58:02.470827 20049 log.cc:927] Time spent T 2112b0402de5464cb1caf2c2abb67d78 P d870164263bb44b6b780a557daea3656: Append to log took a long time: real 0.055s user 0.002s sys 0.000s
I20260130 02:58:02.471325 18872 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50284:
name: "default.Table_003c5fa197e648259fe978a5ee4b44c8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260130 02:58:02.440249 19947 ts_tablet_manager.cc:1950] T 7a902d0d571a42bb82ebffd48f7eca3f P d870164263bb44b6b780a557daea3656: Deleting consensus metadata
I20260130 02:58:02.472456 19840 tablet_bootstrap.cc:492] T 2ca0df71836145748cada4e4559a5656 P 78c2c46c7520492db35d51c244fb4f74: Bootstrap starting.
W20260130 02:58:02.473480 18872 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_003c5fa197e648259fe978a5ee4b44c8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260130 02:58:02.484421 19754 ts_tablet_manager.cc:1929] T 073b5941850c4140b1751956dd7cf5ef P 371a36d219164974abdf848841528152: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260130 02:58:02.484808 19754 log.cc:1199] T 073b5941850c4140b1751956dd7cf5ef P 371a36d219164974abdf848841528152: Deleting WAL directory at /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-0/wal/wals/073b5941850c4140b1751956dd7cf5ef
I20260130 02:58:02.485472 19754 ts_tablet_manager.cc:1950] T 073b5941850c4140b1751956dd7cf5ef P 371a36d219164974abdf848841528152: Deleting consensus metadata
I20260130 02:58:02.486635 18861 catalog_manager.cc:4985] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021): tablet 073b5941850c4140b1751956dd7cf5ef (table default.Table_6eb0a92b1926487cbf0b978d26d6a7b1 [id=de7a420d40864f55b156e149308ba12f]) successfully deleted
I20260130 02:58:02.487218 19527 raft_consensus.cc:2468] T b3f3177306914e3c8b253db54ee30fb6 P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78c2c46c7520492db35d51c244fb4f74 in term 1.
I20260130 02:58:02.490739 19840 tablet_bootstrap.cc:654] T 2ca0df71836145748cada4e4559a5656 P 78c2c46c7520492db35d51c244fb4f74: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:02.493362 19807 ts_tablet_manager.cc:1434] T 1ca34772ef484c1f809a99d126e28eff P 371a36d219164974abdf848841528152: Time spent starting tablet: real 0.243s user 0.005s sys 0.001s
I20260130 02:58:02.376442 18873 catalog_manager.cc:5654] T 1ca34772ef484c1f809a99d126e28eff P 78c2c46c7520492db35d51c244fb4f74 reported cstate change: term changed from 0 to 1, leader changed from <none> to 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66). New cstate: current_term: 1 leader_uuid: "78c2c46c7520492db35d51c244fb4f74" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } health_report { overall_health: UNKNOWN } } }
I20260130 02:58:02.499145 18873 catalog_manager.cc:5654] T 955b9547f49f434699fe1d86b260638c P 78c2c46c7520492db35d51c244fb4f74 reported cstate change: term changed from 0 to 1, leader changed from <none> to 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66). New cstate: current_term: 1 leader_uuid: "78c2c46c7520492db35d51c244fb4f74" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } health_report { overall_health: UNKNOWN } } }
I20260130 02:58:02.494293 20025 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.AlterSchema from 127.0.0.1:42870 (request call id 101) took 1249 ms. Trace:
W20260130 02:58:02.497455 19159 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.100.67:39571, user_credentials={real_user=slave}} blocked reactor thread for 162194us
W20260130 02:58:02.506131 19159 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260130 02:58:02.508764 19807 tablet_bootstrap.cc:492] T bd43c5d148514836869a8ac648c8ad11 P 371a36d219164974abdf848841528152: Bootstrap starting.
I20260130 02:58:02.506794 20047 tablet.cc:1722] T df2c8989d6c1480c97baa23db01141eb P 78c2c46c7520492db35d51c244fb4f74: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20260130 02:58:02.526786 19313 leader_election.cc:304] T b3f3177306914e3c8b253db54ee30fb6 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 78c2c46c7520492db35d51c244fb4f74, d870164263bb44b6b780a557daea3656; no voters: 371a36d219164974abdf848841528152
I20260130 02:58:02.527761 19856 raft_consensus.cc:2804] T b3f3177306914e3c8b253db54ee30fb6 P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Leader election won for term 1
I20260130 02:58:02.528225 19856 raft_consensus.cc:697] T b3f3177306914e3c8b253db54ee30fb6 P 78c2c46c7520492db35d51c244fb4f74 [term 1 LEADER]: Becoming Leader. State: Replica: 78c2c46c7520492db35d51c244fb4f74, State: Running, Role: LEADER
I20260130 02:58:02.528966 19856 consensus_queue.cc:237] T b3f3177306914e3c8b253db54ee30fb6 P 78c2c46c7520492db35d51c244fb4f74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:02.501845 20025 rpcz_store.cc:276] 0130 02:58:01.244734 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:58:01.246319 (+ 1585us) service_pool.cc:225] Handling call
0130 02:58:02.494233 (+1247914us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0130 02:58:01.340096 (+ 0us) alter_schema_op.cc:104] PREPARE ALTER-SCHEMA: Starting
0130 02:58:01.340440 (+ 344us) alter_schema_op.cc:63] Acquiring schema lock in exclusive mode
0130 02:58:01.340469 (+ 29us) alter_schema_op.cc:66] Acquired schema lock
0130 02:58:01.340540 (+ 71us) alter_schema_op.cc:119] PREPARE ALTER-SCHEMA: finished
0130 02:58:01.340662 (+ 122us) alter_schema_op.cc:128] START. Timestamp: 1769741881323446
0130 02:58:01.340682 (+ 20us) op_driver.cc:348] REPLICATION: starting
0130 02:58:01.402619 (+ 61937us) log.cc:844] Serialized 173 byte log entry
0130 02:58:01.405330 (+ 2711us) spinlock_profiling.cc:243] Waited 61.3 ms on lock 0x7b54000b1b54. stack: 00007ff548de3373 00007ff548de3270 00007ff548958574 0000000000508744 00000000005086d9 00007ff54fe345a1 00007ff54fec6aa8 00007ff5503384d1 00007ff550337ca8 00007ff55033c941 00007ff55033c8f9 00007ff55033c889 00007ff55033c851 00007ff55033ba7d 00000000005aa2b4 00000000005aa0e9
0130 02:58:02.087629 (+682299us) op_driver.cc:464] REPLICATION: finished
0130 02:58:02.141658 (+ 54029us) alter_schema_op.cc:133] APPLY ALTER-SCHEMA: Starting
0130 02:58:02.217243 (+ 75585us) log.cc:844] Serialized 5 byte log entry
0130 02:58:02.489098 (+271855us) tablet_metadata.cc:641] Metadata flushed
0130 02:58:02.494022 (+ 4924us) log.cc:844] Serialized 14 byte log entry
0130 02:58:02.494109 (+ 87us) alter_schema_op.cc:73] Released schema lock
0130 02:58:02.494137 (+ 28us) alter_schema_op.cc:186] AlterSchemaCommitCallback: making alter schema visible
Metrics: {"child_traces":[["op",{"apply.queue_time_us":53885,"mutex_wait_us":933,"prepare.queue_time_us":64252,"prepare.run_cpu_time_us":3697,"prepare.run_wall_time_us":67198,"raft.queue_time_us":1061,"raft.run_cpu_time_us":638,"raft.run_wall_time_us":643,"replication_time_us":746839,"spinlock_wait_cycles":141060992,"thread_start_us":50627,"threads_started":1}]]}
I20260130 02:58:02.544881 19754 tablet_replica.cc:333] T 96584ca174b140f7ad1a2ca698baca71 P 371a36d219164974abdf848841528152: stopping tablet replica
I20260130 02:58:02.549876 19754 raft_consensus.cc:2243] T 96584ca174b140f7ad1a2ca698baca71 P 371a36d219164974abdf848841528152 [term 1 LEADER]: Raft consensus shutting down.
I20260130 02:58:02.550876 19754 raft_consensus.cc:2272] T 96584ca174b140f7ad1a2ca698baca71 P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260130 02:58:02.552340 19840 tablet_bootstrap.cc:492] T 2ca0df71836145748cada4e4559a5656 P 78c2c46c7520492db35d51c244fb4f74: No bootstrap required, opened a new log
I20260130 02:58:02.552757 19840 ts_tablet_manager.cc:1403] T 2ca0df71836145748cada4e4559a5656 P 78c2c46c7520492db35d51c244fb4f74: Time spent bootstrapping tablet: real 0.080s user 0.012s sys 0.002s
I20260130 02:58:02.555063 19840 raft_consensus.cc:359] T 2ca0df71836145748cada4e4559a5656 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:02.555630 19840 raft_consensus.cc:385] T 2ca0df71836145748cada4e4559a5656 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:02.555894 19840 raft_consensus.cc:740] T 2ca0df71836145748cada4e4559a5656 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78c2c46c7520492db35d51c244fb4f74, State: Initialized, Role: FOLLOWER
I20260130 02:58:02.557483 19840 consensus_queue.cc:260] T 2ca0df71836145748cada4e4559a5656 P 78c2c46c7520492db35d51c244fb4f74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:02.560422 19754 ts_tablet_manager.cc:1916] T 96584ca174b140f7ad1a2ca698baca71 P 371a36d219164974abdf848841528152: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260130 02:58:02.561590 20047 tablet.cc:1722] T 15918c90ffdc43a5a38b2950e9834188 P 78c2c46c7520492db35d51c244fb4f74: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260130 02:58:02.564517 19840 ts_tablet_manager.cc:1434] T 2ca0df71836145748cada4e4559a5656 P 78c2c46c7520492db35d51c244fb4f74: Time spent starting tablet: real 0.011s user 0.005s sys 0.001s
I20260130 02:58:02.565521 19840 tablet_bootstrap.cc:492] T e47b22aafe5247c4b62c4193c24f427c P 78c2c46c7520492db35d51c244fb4f74: Bootstrap starting.
I20260130 02:58:02.507903 19758 ts_tablet_manager.cc:1929] T 7a902d0d571a42bb82ebffd48f7eca3f P 78c2c46c7520492db35d51c244fb4f74: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260130 02:58:02.568483 19758 log.cc:1199] T 7a902d0d571a42bb82ebffd48f7eca3f P 78c2c46c7520492db35d51c244fb4f74: Deleting WAL directory at /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-1/wal/wals/7a902d0d571a42bb82ebffd48f7eca3f
I20260130 02:58:02.569275 19758 ts_tablet_manager.cc:1950] T 7a902d0d571a42bb82ebffd48f7eca3f P 78c2c46c7520492db35d51c244fb4f74: Deleting consensus metadata
I20260130 02:58:02.474558 19947 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59068 (request call id 93) took 1062 ms. Trace:
I20260130 02:58:02.584085 20009 raft_consensus.cc:493] T d3ba2df0b15c44f882070570f91b254d P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:02.584560 20009 raft_consensus.cc:515] T d3ba2df0b15c44f882070570f91b254d P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:02.586261 19754 ts_tablet_manager.cc:1929] T 96584ca174b140f7ad1a2ca698baca71 P 371a36d219164974abdf848841528152: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.6
I20260130 02:58:02.586337 20009 leader_election.cc:290] T d3ba2df0b15c44f882070570f91b254d P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), d870164263bb44b6b780a557daea3656 (127.18.100.67:39571)
I20260130 02:58:02.590734 19754 log.cc:1199] T 96584ca174b140f7ad1a2ca698baca71 P 371a36d219164974abdf848841528152: Deleting WAL directory at /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-0/wal/wals/96584ca174b140f7ad1a2ca698baca71
I20260130 02:58:02.591904 19754 ts_tablet_manager.cc:1950] T 96584ca174b140f7ad1a2ca698baca71 P 371a36d219164974abdf848841528152: Deleting consensus metadata
I20260130 02:58:02.594717 19807 tablet_bootstrap.cc:654] T bd43c5d148514836869a8ac648c8ad11 P 371a36d219164974abdf848841528152: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:02.584097 19840 tablet_bootstrap.cc:654] T e47b22aafe5247c4b62c4193c24f427c P 78c2c46c7520492db35d51c244fb4f74: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:02.599366 18860 catalog_manager.cc:4985] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767): tablet 7a902d0d571a42bb82ebffd48f7eca3f (table default.Table_298b1af220674b6ab20d3652e8a637a1 [id=742a870a99554fc48fb4f369e68fc6d6]) successfully deleted
I20260130 02:58:02.599531 20025 tablet.cc:1722] T 15918c90ffdc43a5a38b2950e9834188 P 371a36d219164974abdf848841528152: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20260130 02:58:02.601679 18860 catalog_manager.cc:4952] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767): delete failed for tablet 878f9346d3044d8d8b15c61f86869212 because the tablet was not found. No further retry: Not found: Tablet not found: 878f9346d3044d8d8b15c61f86869212
W20260130 02:58:02.603367 18860 catalog_manager.cc:4952] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767): delete failed for tablet d4f2b54930cc4f5c825562e3e0a79f4d because the tablet was not found. No further retry: Not found: Tablet not found: d4f2b54930cc4f5c825562e3e0a79f4d
I20260130 02:58:02.603022 19221 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d3ba2df0b15c44f882070570f91b254d" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152" is_pre_election: true
W20260130 02:58:02.604938 19314 leader_election.cc:343] T d3ba2df0b15c44f882070570f91b254d P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:02.606992 19758 tablet_replica.cc:333] T 37470ce4356f4542b621e21698551209 P 78c2c46c7520492db35d51c244fb4f74: stopping tablet replica
I20260130 02:58:02.611600 19754 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42870 (request call id 102) took 1116 ms. Trace:
I20260130 02:58:02.612440 19754 rpcz_store.cc:276] 0130 02:58:01.495527 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:58:01.502734 (+ 7207us) service_pool.cc:225] Handling call
0130 02:58:02.544740 (+1042006us) ts_tablet_manager.cc:1151] Deleting tablet 96584ca174b140f7ad1a2ca698baca71
0130 02:58:02.544780 (+ 40us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0130 02:58:02.573378 (+ 28598us) tablet_metadata.cc:641] Metadata flushed
0130 02:58:02.586061 (+ 12683us) tablet_metadata.cc:641] Metadata flushed
0130 02:58:02.594893 (+ 8832us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":969089}
I20260130 02:58:02.613951 19754 tablet_replica.cc:333] T b078934be4bb4fd1915c691eeea9d00f P 371a36d219164974abdf848841528152: stopping tablet replica
I20260130 02:58:02.613786 18861 catalog_manager.cc:4985] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021): tablet 96584ca174b140f7ad1a2ca698baca71 (table default.Table_4f2d6e4d9c8e465087d65efc9c2de442 [id=1515db389f284e2f9ed1a3cd0061d0f1]) successfully deleted
I20260130 02:58:02.614171 19758 raft_consensus.cc:2243] T 37470ce4356f4542b621e21698551209 P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260130 02:58:02.614733 19758 raft_consensus.cc:2272] T 37470ce4356f4542b621e21698551209 P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260130 02:58:02.614837 19754 raft_consensus.cc:2243] T b078934be4bb4fd1915c691eeea9d00f P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260130 02:58:02.616339 19754 raft_consensus.cc:2272] T b078934be4bb4fd1915c691eeea9d00f P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260130 02:58:02.616554 19758 ts_tablet_manager.cc:1916] T 37470ce4356f4542b621e21698551209 P 78c2c46c7520492db35d51c244fb4f74: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260130 02:58:02.618171 19754 ts_tablet_manager.cc:1916] T b078934be4bb4fd1915c691eeea9d00f P 371a36d219164974abdf848841528152: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260130 02:58:02.616616 19840 tablet_bootstrap.cc:492] T e47b22aafe5247c4b62c4193c24f427c P 78c2c46c7520492db35d51c244fb4f74: No bootstrap required, opened a new log
I20260130 02:58:02.620355 19840 ts_tablet_manager.cc:1403] T e47b22aafe5247c4b62c4193c24f427c P 78c2c46c7520492db35d51c244fb4f74: Time spent bootstrapping tablet: real 0.055s user 0.008s sys 0.002s
I20260130 02:58:02.622545 19840 raft_consensus.cc:359] T e47b22aafe5247c4b62c4193c24f427c P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:02.623158 19840 raft_consensus.cc:385] T e47b22aafe5247c4b62c4193c24f427c P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:02.623406 19840 raft_consensus.cc:740] T e47b22aafe5247c4b62c4193c24f427c P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78c2c46c7520492db35d51c244fb4f74, State: Initialized, Role: FOLLOWER
I20260130 02:58:02.624053 19840 consensus_queue.cc:260] T e47b22aafe5247c4b62c4193c24f427c P 78c2c46c7520492db35d51c244fb4f74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:02.626487 19840 ts_tablet_manager.cc:1434] T e47b22aafe5247c4b62c4193c24f427c P 78c2c46c7520492db35d51c244fb4f74: Time spent starting tablet: real 0.006s user 0.001s sys 0.004s
I20260130 02:58:02.630326 19840 tablet_bootstrap.cc:492] T 7b80b22bd44c4233837c951aa97e6996 P 78c2c46c7520492db35d51c244fb4f74: Bootstrap starting.
I20260130 02:58:02.637648 19807 tablet_bootstrap.cc:492] T bd43c5d148514836869a8ac648c8ad11 P 371a36d219164974abdf848841528152: No bootstrap required, opened a new log
I20260130 02:58:02.638082 19807 ts_tablet_manager.cc:1403] T bd43c5d148514836869a8ac648c8ad11 P 371a36d219164974abdf848841528152: Time spent bootstrapping tablet: real 0.130s user 0.009s sys 0.001s
I20260130 02:58:02.641113 19807 raft_consensus.cc:359] T bd43c5d148514836869a8ac648c8ad11 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:02.641728 19807 raft_consensus.cc:385] T bd43c5d148514836869a8ac648c8ad11 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:02.641973 19807 raft_consensus.cc:740] T bd43c5d148514836869a8ac648c8ad11 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 371a36d219164974abdf848841528152, State: Initialized, Role: FOLLOWER
I20260130 02:58:02.646836 19807 consensus_queue.cc:260] T bd43c5d148514836869a8ac648c8ad11 P 371a36d219164974abdf848841528152 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:02.651216 19754 ts_tablet_manager.cc:1929] T b078934be4bb4fd1915c691eeea9d00f P 371a36d219164974abdf848841528152: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20260130 02:58:02.651942 19754 log.cc:1199] T b078934be4bb4fd1915c691eeea9d00f P 371a36d219164974abdf848841528152: Deleting WAL directory at /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-0/wal/wals/b078934be4bb4fd1915c691eeea9d00f
I20260130 02:58:02.656104 19807 ts_tablet_manager.cc:1434] T bd43c5d148514836869a8ac648c8ad11 P 371a36d219164974abdf848841528152: Time spent starting tablet: real 0.018s user 0.005s sys 0.001s
I20260130 02:58:02.657509 19754 ts_tablet_manager.cc:1950] T b078934be4bb4fd1915c691eeea9d00f P 371a36d219164974abdf848841528152: Deleting consensus metadata
I20260130 02:58:02.659556 18861 catalog_manager.cc:4985] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021): tablet b078934be4bb4fd1915c691eeea9d00f (table default.Table_b63ca4e4060f48e893b0099fb6433f04 [id=720decfb55c24f34983ef636f06e3169]) successfully deleted
W20260130 02:58:02.660336 18861 catalog_manager.cc:4952] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021): delete failed for tablet 93809949599a449ca09b13e9b8bbdef6 because the tablet was not found. No further retry: Not found: Tablet not found: 93809949599a449ca09b13e9b8bbdef6
W20260130 02:58:02.660965 18861 catalog_manager.cc:4952] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021): delete failed for tablet 4035b8a330844dd28af9f3244bf8eb9a because the tablet was not found. No further retry: Not found: Tablet not found: 4035b8a330844dd28af9f3244bf8eb9a
I20260130 02:58:02.662462 19840 tablet_bootstrap.cc:654] T 7b80b22bd44c4233837c951aa97e6996 P 78c2c46c7520492db35d51c244fb4f74: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:02.463883 20044 consensus_queue.cc:237] T b7b596f944be4daf953b82b65d5cec2a P d870164263bb44b6b780a557daea3656 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
W20260130 02:58:02.668017 18861 catalog_manager.cc:4952] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021): delete failed for tablet 3b8cdac4bf4b46efa26236fc930eb9a6 because the tablet was not found. No further retry: Not found: Tablet not found: 3b8cdac4bf4b46efa26236fc930eb9a6
I20260130 02:58:02.668610 19528 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d3ba2df0b15c44f882070570f91b254d" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
I20260130 02:58:02.669195 19528 raft_consensus.cc:2468] T d3ba2df0b15c44f882070570f91b254d P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78c2c46c7520492db35d51c244fb4f74 in term 0.
I20260130 02:58:02.671178 19807 tablet_bootstrap.cc:492] T 955b9547f49f434699fe1d86b260638c P 371a36d219164974abdf848841528152: Bootstrap starting.
W20260130 02:58:02.672590 18861 catalog_manager.cc:4952] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021): delete failed for tablet d4f2b54930cc4f5c825562e3e0a79f4d because the tablet was not found. No further retry: Not found: Tablet not found: d4f2b54930cc4f5c825562e3e0a79f4d
I20260130 02:58:02.676600 19754 tablet_replica.cc:333] T 37470ce4356f4542b621e21698551209 P 371a36d219164974abdf848841528152: stopping tablet replica
I20260130 02:58:02.681198 19840 tablet_bootstrap.cc:492] T 7b80b22bd44c4233837c951aa97e6996 P 78c2c46c7520492db35d51c244fb4f74: No bootstrap required, opened a new log
I20260130 02:58:02.681674 19840 ts_tablet_manager.cc:1403] T 7b80b22bd44c4233837c951aa97e6996 P 78c2c46c7520492db35d51c244fb4f74: Time spent bootstrapping tablet: real 0.052s user 0.011s sys 0.000s
I20260130 02:58:02.683976 19840 raft_consensus.cc:359] T 7b80b22bd44c4233837c951aa97e6996 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:02.684733 19840 raft_consensus.cc:385] T 7b80b22bd44c4233837c951aa97e6996 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:02.685746 19754 raft_consensus.cc:2243] T 37470ce4356f4542b621e21698551209 P 371a36d219164974abdf848841528152 [term 1 LEADER]: Raft consensus shutting down.
I20260130 02:58:02.686112 19840 raft_consensus.cc:740] T 7b80b22bd44c4233837c951aa97e6996 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78c2c46c7520492db35d51c244fb4f74, State: Initialized, Role: FOLLOWER
I20260130 02:58:02.686614 19754 raft_consensus.cc:2272] T 37470ce4356f4542b621e21698551209 P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260130 02:58:02.686894 19840 consensus_queue.cc:260] T 7b80b22bd44c4233837c951aa97e6996 P 78c2c46c7520492db35d51c244fb4f74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:02.688364 19754 ts_tablet_manager.cc:1916] T 37470ce4356f4542b621e21698551209 P 371a36d219164974abdf848841528152: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260130 02:58:02.692243 19758 ts_tablet_manager.cc:1929] T 37470ce4356f4542b621e21698551209 P 78c2c46c7520492db35d51c244fb4f74: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260130 02:58:02.692585 19758 log.cc:1199] T 37470ce4356f4542b621e21698551209 P 78c2c46c7520492db35d51c244fb4f74: Deleting WAL directory at /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-1/wal/wals/37470ce4356f4542b621e21698551209
I20260130 02:58:02.693240 19758 ts_tablet_manager.cc:1950] T 37470ce4356f4542b621e21698551209 P 78c2c46c7520492db35d51c244fb4f74: Deleting consensus metadata
I20260130 02:58:02.693666 19840 ts_tablet_manager.cc:1434] T 7b80b22bd44c4233837c951aa97e6996 P 78c2c46c7520492db35d51c244fb4f74: Time spent starting tablet: real 0.012s user 0.005s sys 0.000s
I20260130 02:58:02.596294 19947 rpcz_store.cc:276] 0130 02:58:01.412582 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:58:01.442132 (+ 29550us) service_pool.cc:225] Handling call
0130 02:58:01.502859 (+ 60727us) ts_tablet_manager.cc:1151] Deleting tablet 7a902d0d571a42bb82ebffd48f7eca3f
0130 02:58:01.502957 (+ 98us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0130 02:58:02.233898 (+730941us) tablet_metadata.cc:641] Metadata flushed
0130 02:58:02.298196 (+ 64298us) tablet_metadata.cc:641] Metadata flushed
0130 02:58:02.474472 (+176276us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":92777,"tablet-delete.queue_time_us":59355}
I20260130 02:58:02.694639 19947 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59068 (request call id 92) took 1282 ms. Trace:
I20260130 02:58:02.695061 19947 rpcz_store.cc:276] 0130 02:58:01.412185 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:58:01.443737 (+ 31552us) service_pool.cc:225] Handling call
0130 02:58:02.694308 (+1250571us) ts_tablet_manager.cc:1151] Deleting tablet 4035b8a330844dd28af9f3244bf8eb9a
0130 02:58:02.694362 (+ 54us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0130 02:58:02.694552 (+ 190us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1249795}
I20260130 02:58:02.697486 19807 tablet_bootstrap.cc:654] T 955b9547f49f434699fe1d86b260638c P 371a36d219164974abdf848841528152: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:02.697425 18860 catalog_manager.cc:4985] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): tablet 7a902d0d571a42bb82ebffd48f7eca3f (table default.Table_298b1af220674b6ab20d3652e8a637a1 [id=742a870a99554fc48fb4f369e68fc6d6]) successfully deleted
W20260130 02:58:02.698160 18860 catalog_manager.cc:4952] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): delete failed for tablet 4035b8a330844dd28af9f3244bf8eb9a because the tablet was not found. No further retry: Not found: Tablet not found: 4035b8a330844dd28af9f3244bf8eb9a
I20260130 02:58:02.699287 19840 tablet_bootstrap.cc:492] T 5ca9919f80954e74a4d91c94b74de0e4 P 78c2c46c7520492db35d51c244fb4f74: Bootstrap starting.
I20260130 02:58:02.704459 19754 ts_tablet_manager.cc:1929] T 37470ce4356f4542b621e21698551209 P 371a36d219164974abdf848841528152: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260130 02:58:02.704811 19754 log.cc:1199] T 37470ce4356f4542b621e21698551209 P 371a36d219164974abdf848841528152: Deleting WAL directory at /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-0/wal/wals/37470ce4356f4542b621e21698551209
I20260130 02:58:02.705440 19754 ts_tablet_manager.cc:1950] T 37470ce4356f4542b621e21698551209 P 371a36d219164974abdf848841528152: Deleting consensus metadata
I20260130 02:58:02.705943 19625 tablet_bootstrap.cc:654] T ec28ff12927d450ead16d26cf016138d P d870164263bb44b6b780a557daea3656: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:02.707664 18861 catalog_manager.cc:4985] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021): tablet 37470ce4356f4542b621e21698551209 (table default.Table_007e969f70d345a7bcb3dfbaabbc025d [id=77d4fa8705a3419cb0a84d3ca1bb4ce3]) successfully deleted
I20260130 02:58:02.708446 18860 catalog_manager.cc:4985] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767): tablet 37470ce4356f4542b621e21698551209 (table default.Table_007e969f70d345a7bcb3dfbaabbc025d [id=77d4fa8705a3419cb0a84d3ca1bb4ce3]) successfully deleted
I20260130 02:58:02.710697 19807 tablet_bootstrap.cc:492] T 955b9547f49f434699fe1d86b260638c P 371a36d219164974abdf848841528152: No bootstrap required, opened a new log
I20260130 02:58:02.711102 19807 ts_tablet_manager.cc:1403] T 955b9547f49f434699fe1d86b260638c P 371a36d219164974abdf848841528152: Time spent bootstrapping tablet: real 0.040s user 0.006s sys 0.004s
I20260130 02:58:02.713289 19807 raft_consensus.cc:359] T 955b9547f49f434699fe1d86b260638c P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:02.713894 19807 raft_consensus.cc:385] T 955b9547f49f434699fe1d86b260638c P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:02.713471 20059 tablet.cc:1722] T df2c8989d6c1480c97baa23db01141eb P d870164263bb44b6b780a557daea3656: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20260130 02:58:02.714146 19807 raft_consensus.cc:740] T 955b9547f49f434699fe1d86b260638c P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 371a36d219164974abdf848841528152, State: Initialized, Role: FOLLOWER
I20260130 02:58:02.714723 19807 consensus_queue.cc:260] T 955b9547f49f434699fe1d86b260638c P 371a36d219164974abdf848841528152 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:02.709949 19947 tablet_replica.cc:333] T a46702bef0e6442d85e4ca49a5f7cfbb P d870164263bb44b6b780a557daea3656: stopping tablet replica
W20260130 02:58:02.663643 19463 leader_election.cc:343] T bd43c5d148514836869a8ac648c8ad11 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:02.723806 19840 tablet_bootstrap.cc:654] T 5ca9919f80954e74a4d91c94b74de0e4 P 78c2c46c7520492db35d51c244fb4f74: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:02.733146 19807 ts_tablet_manager.cc:1434] T 955b9547f49f434699fe1d86b260638c P 371a36d219164974abdf848841528152: Time spent starting tablet: real 0.022s user 0.005s sys 0.000s
I20260130 02:58:02.739933 19807 tablet_bootstrap.cc:492] T a743a82540014c85aeab0e9d0994b0dc P 371a36d219164974abdf848841528152: Bootstrap starting.
W20260130 02:58:02.742233 20039 log.cc:927] Time spent T e026dee22493424692d00f14848ab57c P d870164263bb44b6b780a557daea3656: Append to log took a long time: real 0.082s user 0.000s sys 0.000s
I20260130 02:58:02.743494 19840 tablet_bootstrap.cc:492] T 5ca9919f80954e74a4d91c94b74de0e4 P 78c2c46c7520492db35d51c244fb4f74: No bootstrap required, opened a new log
I20260130 02:58:02.743878 19840 ts_tablet_manager.cc:1403] T 5ca9919f80954e74a4d91c94b74de0e4 P 78c2c46c7520492db35d51c244fb4f74: Time spent bootstrapping tablet: real 0.045s user 0.008s sys 0.001s
I20260130 02:58:02.731153 19463 leader_election.cc:304] T bd43c5d148514836869a8ac648c8ad11 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d870164263bb44b6b780a557daea3656; no voters: 371a36d219164974abdf848841528152, 78c2c46c7520492db35d51c244fb4f74
I20260130 02:58:02.746047 20029 raft_consensus.cc:2749] T bd43c5d148514836869a8ac648c8ad11 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260130 02:58:02.746218 19840 raft_consensus.cc:359] T 5ca9919f80954e74a4d91c94b74de0e4 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:02.747309 19840 raft_consensus.cc:385] T 5ca9919f80954e74a4d91c94b74de0e4 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:02.747556 19840 raft_consensus.cc:740] T 5ca9919f80954e74a4d91c94b74de0e4 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78c2c46c7520492db35d51c244fb4f74, State: Initialized, Role: FOLLOWER
I20260130 02:58:02.748399 19947 raft_consensus.cc:2243] T a46702bef0e6442d85e4ca49a5f7cfbb P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260130 02:58:02.748289 19840 consensus_queue.cc:260] T 5ca9919f80954e74a4d91c94b74de0e4 P 78c2c46c7520492db35d51c244fb4f74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:02.749049 19947 raft_consensus.cc:2272] T a46702bef0e6442d85e4ca49a5f7cfbb P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260130 02:58:02.750074 19463 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.100.65:42021, user_credentials={real_user=slave}} blocked reactor thread for 90007.3us
I20260130 02:58:02.755129 20029 consensus_queue.cc:1048] T 1bbc55bbf2254d8bb6b721d7e8f69d85 P d870164263bb44b6b780a557daea3656 [LEADER]: Connected to new peer: Peer: permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260130 02:58:02.756455 19840 ts_tablet_manager.cc:1434] T 5ca9919f80954e74a4d91c94b74de0e4 P 78c2c46c7520492db35d51c244fb4f74: Time spent starting tablet: real 0.012s user 0.005s sys 0.000s
W20260130 02:58:02.756577 19463 leader_election.cc:343] T a743a82540014c85aeab0e9d0994b0dc P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:02.757405 19840 tablet_bootstrap.cc:492] T 7d76046ee6c8420fa4964decbe8209c1 P 78c2c46c7520492db35d51c244fb4f74: Bootstrap starting.
I20260130 02:58:02.757827 20029 consensus_queue.cc:1048] T e026dee22493424692d00f14848ab57c P d870164263bb44b6b780a557daea3656 [LEADER]: Connected to new peer: Peer: permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260130 02:58:02.758421 19947 ts_tablet_manager.cc:1916] T a46702bef0e6442d85e4ca49a5f7cfbb P d870164263bb44b6b780a557daea3656: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260130 02:58:02.759691 19807 tablet_bootstrap.cc:654] T a743a82540014c85aeab0e9d0994b0dc P 371a36d219164974abdf848841528152: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:02.762090 20029 consensus_queue.cc:1048] T 5b2976108584478a98fd90fe6594af66 P d870164263bb44b6b780a557daea3656 [LEADER]: Connected to new peer: Peer: permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260130 02:58:02.769183 19840 tablet_bootstrap.cc:654] T 7d76046ee6c8420fa4964decbe8209c1 P 78c2c46c7520492db35d51c244fb4f74: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:02.777470 19807 tablet_bootstrap.cc:492] T a743a82540014c85aeab0e9d0994b0dc P 371a36d219164974abdf848841528152: No bootstrap required, opened a new log
I20260130 02:58:02.777894 19807 ts_tablet_manager.cc:1403] T a743a82540014c85aeab0e9d0994b0dc P 371a36d219164974abdf848841528152: Time spent bootstrapping tablet: real 0.038s user 0.010s sys 0.002s
I20260130 02:58:02.780372 19807 raft_consensus.cc:359] T a743a82540014c85aeab0e9d0994b0dc P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:02.781004 19807 raft_consensus.cc:385] T a743a82540014c85aeab0e9d0994b0dc P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:02.781273 19807 raft_consensus.cc:740] T a743a82540014c85aeab0e9d0994b0dc P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 371a36d219164974abdf848841528152, State: Initialized, Role: FOLLOWER
I20260130 02:58:02.782300 19807 consensus_queue.cc:260] T a743a82540014c85aeab0e9d0994b0dc P 371a36d219164974abdf848841528152 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:02.784984 19625 tablet_bootstrap.cc:492] T ec28ff12927d450ead16d26cf016138d P d870164263bb44b6b780a557daea3656: No bootstrap required, opened a new log
I20260130 02:58:02.785465 19625 ts_tablet_manager.cc:1403] T ec28ff12927d450ead16d26cf016138d P d870164263bb44b6b780a557daea3656: Time spent bootstrapping tablet: real 0.380s user 0.011s sys 0.003s
I20260130 02:58:02.790901 19807 ts_tablet_manager.cc:1434] T a743a82540014c85aeab0e9d0994b0dc P 371a36d219164974abdf848841528152: Time spent starting tablet: real 0.013s user 0.007s sys 0.000s
I20260130 02:58:02.791744 20025 tablet.cc:1722] T df2c8989d6c1480c97baa23db01141eb P 371a36d219164974abdf848841528152: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20260130 02:58:02.795806 19861 raft_consensus.cc:493] T 1ca34772ef484c1f809a99d126e28eff P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:02.796377 19861 raft_consensus.cc:515] T 1ca34772ef484c1f809a99d126e28eff P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:02.799382 19861 leader_election.cc:290] T 1ca34772ef484c1f809a99d126e28eff P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767), d870164263bb44b6b780a557daea3656 (127.18.100.67:39571)
I20260130 02:58:02.802999 19378 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1ca34772ef484c1f809a99d126e28eff" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74" is_pre_election: true
I20260130 02:58:02.816920 19313 leader_election.cc:304] T d3ba2df0b15c44f882070570f91b254d P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 78c2c46c7520492db35d51c244fb4f74, d870164263bb44b6b780a557daea3656; no voters: 371a36d219164974abdf848841528152
I20260130 02:58:02.818743 20009 raft_consensus.cc:2804] T d3ba2df0b15c44f882070570f91b254d P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260130 02:58:02.819124 20009 raft_consensus.cc:493] T d3ba2df0b15c44f882070570f91b254d P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260130 02:58:02.819453 20009 raft_consensus.cc:3060] T d3ba2df0b15c44f882070570f91b254d P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:02.826273 20009 raft_consensus.cc:515] T d3ba2df0b15c44f882070570f91b254d P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:02.826421 20059 tablet.cc:1722] T e364f4e70e5749e083e14e7fa7416ff0 P d870164263bb44b6b780a557daea3656: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260130 02:58:02.829138 20009 leader_election.cc:290] T d3ba2df0b15c44f882070570f91b254d P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 election: Requested vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), d870164263bb44b6b780a557daea3656 (127.18.100.67:39571)
I20260130 02:58:02.830965 19807 tablet_bootstrap.cc:492] T ec28ff12927d450ead16d26cf016138d P 371a36d219164974abdf848841528152: Bootstrap starting.
I20260130 02:58:02.836966 19625 raft_consensus.cc:359] T ec28ff12927d450ead16d26cf016138d P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:02.837634 19625 raft_consensus.cc:385] T ec28ff12927d450ead16d26cf016138d P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:02.837880 19625 raft_consensus.cc:740] T ec28ff12927d450ead16d26cf016138d P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d870164263bb44b6b780a557daea3656, State: Initialized, Role: FOLLOWER
I20260130 02:58:02.836508 19223 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d3ba2df0b15c44f882070570f91b254d" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152"
W20260130 02:58:02.848512 19314 leader_election.cc:343] T d3ba2df0b15c44f882070570f91b254d P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:02.859584 19223 raft_consensus.cc:3060] T 1ca34772ef484c1f809a99d126e28eff P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Advancing to term 1
W20260130 02:58:02.866894 19313 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.100.67:39571, user_credentials={real_user=slave}} blocked reactor thread for 50051us
I20260130 02:58:02.868017 19856 consensus_queue.cc:1048] T c7c621b99deb457998c44ab1644d3939 P 78c2c46c7520492db35d51c244fb4f74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260130 02:58:02.848906 20077 tablet.cc:1722] T bac739bf4cac4272975f5587ad9c71ff P d870164263bb44b6b780a557daea3656: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260130 02:58:02.861925 20079 tablet.cc:1722] T 2112b0402de5464cb1caf2c2abb67d78 P d870164263bb44b6b780a557daea3656: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260130 02:58:02.846030 19527 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1ca34772ef484c1f809a99d126e28eff" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
I20260130 02:58:02.875612 19527 raft_consensus.cc:2393] T 1ca34772ef484c1f809a99d126e28eff P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 371a36d219164974abdf848841528152 in current term 1: Already voted for candidate 78c2c46c7520492db35d51c244fb4f74 in this term.
I20260130 02:58:02.863132 20078 tablet.cc:1722] T 15918c90ffdc43a5a38b2950e9834188 P d870164263bb44b6b780a557daea3656: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260130 02:58:02.878479 19159 leader_election.cc:304] T 1ca34772ef484c1f809a99d126e28eff P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 371a36d219164974abdf848841528152; no voters: 78c2c46c7520492db35d51c244fb4f74, d870164263bb44b6b780a557daea3656
I20260130 02:58:02.879204 19861 raft_consensus.cc:2749] T 1ca34772ef484c1f809a99d126e28eff P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260130 02:58:02.879570 19856 consensus_queue.cc:1048] T 1ca34772ef484c1f809a99d126e28eff P 78c2c46c7520492db35d51c244fb4f74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260130 02:58:02.880012 19807 tablet_bootstrap.cc:654] T ec28ff12927d450ead16d26cf016138d P 371a36d219164974abdf848841528152: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:02.881307 19899 consensus_queue.cc:1048] T 1ca34772ef484c1f809a99d126e28eff P 78c2c46c7520492db35d51c244fb4f74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260130 02:58:02.838547 19625 consensus_queue.cc:260] T ec28ff12927d450ead16d26cf016138d P d870164263bb44b6b780a557daea3656 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:02.894883 19625 ts_tablet_manager.cc:1434] T ec28ff12927d450ead16d26cf016138d P d870164263bb44b6b780a557daea3656: Time spent starting tablet: real 0.109s user 0.004s sys 0.002s
I20260130 02:58:02.650075 18867 catalog_manager.cc:5746] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571) does not have the latest schema for tablet bac739bf4cac4272975f5587ad9c71ff (table default.Table_9e942a1ce3944e9aa028e75656ce066d [id=18ed268eb9ad4450abf9874310ace074]). Expected version 1 got 0
I20260130 02:58:02.898532 18867 catalog_manager.cc:5746] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571) does not have the latest schema for tablet e364f4e70e5749e083e14e7fa7416ff0 (table default.Table_bc11cb1bce0348ed9ea0912b3ee521b9 [id=df7d3d68e36142ff8564e63ffe0b8797]). Expected version 1 got 0
I20260130 02:58:02.899993 18867 catalog_manager.cc:5746] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571) does not have the latest schema for tablet 2112b0402de5464cb1caf2c2abb67d78 (table default.Table_da3d581b15a94840898777f74611994e [id=b7e241ced53e4160ac6fa1f65ef3674e]). Expected version 1 got 0
I20260130 02:58:02.900410 19840 tablet_bootstrap.cc:492] T 7d76046ee6c8420fa4964decbe8209c1 P 78c2c46c7520492db35d51c244fb4f74: No bootstrap required, opened a new log
I20260130 02:58:02.900949 19840 ts_tablet_manager.cc:1403] T 7d76046ee6c8420fa4964decbe8209c1 P 78c2c46c7520492db35d51c244fb4f74: Time spent bootstrapping tablet: real 0.144s user 0.020s sys 0.008s
I20260130 02:58:02.901149 18867 catalog_manager.cc:5746] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571) does not have the latest schema for tablet df2c8989d6c1480c97baa23db01141eb (table default.Table_e84c6277dfc944059f62cbe0d6b937ee [id=747af828147149efb7b7b8d33c49286d]). Expected version 2 got 1
I20260130 02:58:02.903319 19840 raft_consensus.cc:359] T 7d76046ee6c8420fa4964decbe8209c1 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:02.903959 19840 raft_consensus.cc:385] T 7d76046ee6c8420fa4964decbe8209c1 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:02.904206 19840 raft_consensus.cc:740] T 7d76046ee6c8420fa4964decbe8209c1 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78c2c46c7520492db35d51c244fb4f74, State: Initialized, Role: FOLLOWER
I20260130 02:58:02.908178 19224 raft_consensus.cc:3060] T 955b9547f49f434699fe1d86b260638c P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:02.912509 19527 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d3ba2df0b15c44f882070570f91b254d" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656"
I20260130 02:58:02.913012 19527 raft_consensus.cc:3060] T d3ba2df0b15c44f882070570f91b254d P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:02.916286 19856 consensus_queue.cc:1048] T 955b9547f49f434699fe1d86b260638c P 78c2c46c7520492db35d51c244fb4f74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260130 02:58:02.919230 19527 raft_consensus.cc:2468] T d3ba2df0b15c44f882070570f91b254d P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78c2c46c7520492db35d51c244fb4f74 in term 1.
I20260130 02:58:02.927089 19625 tablet_bootstrap.cc:492] T 7d76046ee6c8420fa4964decbe8209c1 P d870164263bb44b6b780a557daea3656: Bootstrap starting.
I20260130 02:58:02.931370 19840 consensus_queue.cc:260] T 7d76046ee6c8420fa4964decbe8209c1 P 78c2c46c7520492db35d51c244fb4f74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:02.939121 19840 ts_tablet_manager.cc:1434] T 7d76046ee6c8420fa4964decbe8209c1 P 78c2c46c7520492db35d51c244fb4f74: Time spent starting tablet: real 0.038s user 0.005s sys 0.000s
I20260130 02:58:02.947907 19840 tablet_bootstrap.cc:492] T ec28ff12927d450ead16d26cf016138d P 78c2c46c7520492db35d51c244fb4f74: Bootstrap starting.
I20260130 02:58:02.954712 19807 tablet_bootstrap.cc:492] T ec28ff12927d450ead16d26cf016138d P 371a36d219164974abdf848841528152: No bootstrap required, opened a new log
I20260130 02:58:02.955159 19807 ts_tablet_manager.cc:1403] T ec28ff12927d450ead16d26cf016138d P 371a36d219164974abdf848841528152: Time spent bootstrapping tablet: real 0.124s user 0.012s sys 0.000s
I20260130 02:58:02.957561 19807 raft_consensus.cc:359] T ec28ff12927d450ead16d26cf016138d P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:02.958191 19807 raft_consensus.cc:385] T ec28ff12927d450ead16d26cf016138d P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:02.958448 19807 raft_consensus.cc:740] T ec28ff12927d450ead16d26cf016138d P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 371a36d219164974abdf848841528152, State: Initialized, Role: FOLLOWER
I20260130 02:58:02.958447 19840 tablet_bootstrap.cc:654] T ec28ff12927d450ead16d26cf016138d P 78c2c46c7520492db35d51c244fb4f74: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:02.959204 19807 consensus_queue.cc:260] T ec28ff12927d450ead16d26cf016138d P 371a36d219164974abdf848841528152 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:02.968724 19807 ts_tablet_manager.cc:1434] T ec28ff12927d450ead16d26cf016138d P 371a36d219164974abdf848841528152: Time spent starting tablet: real 0.013s user 0.005s sys 0.000s
I20260130 02:58:02.969581 19807 tablet_bootstrap.cc:492] T d3ba2df0b15c44f882070570f91b254d P 371a36d219164974abdf848841528152: Bootstrap starting.
I20260130 02:58:02.996661 20083 raft_consensus.cc:493] T e47b22aafe5247c4b62c4193c24f427c P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:02.997150 20083 raft_consensus.cc:515] T e47b22aafe5247c4b62c4193c24f427c P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:02.999989 20083 leader_election.cc:290] T e47b22aafe5247c4b62c4193c24f427c P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), d870164263bb44b6b780a557daea3656 (127.18.100.67:39571)
I20260130 02:58:03.007324 19221 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e47b22aafe5247c4b62c4193c24f427c" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152" is_pre_election: true
W20260130 02:58:03.008773 19314 leader_election.cc:343] T e47b22aafe5247c4b62c4193c24f427c P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:02.902172 18867 catalog_manager.cc:5746] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571) does not have the latest schema for tablet 15918c90ffdc43a5a38b2950e9834188 (table default.Table_333d2efbf068485a871f566d50e563ce [id=8b7db40afb894ae99625a756c75ef1de]). Expected version 1 got 0
I20260130 02:58:03.033746 19807 tablet_bootstrap.cc:654] T d3ba2df0b15c44f882070570f91b254d P 371a36d219164974abdf848841528152: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:03.033736 18872 catalog_manager.cc:5746] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021) does not have the latest schema for tablet 15918c90ffdc43a5a38b2950e9834188 (table default.Table_333d2efbf068485a871f566d50e563ce [id=8b7db40afb894ae99625a756c75ef1de]). Expected version 1 got 0
I20260130 02:58:03.034515 18872 catalog_manager.cc:5746] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021) does not have the latest schema for tablet df2c8989d6c1480c97baa23db01141eb (table default.Table_e84c6277dfc944059f62cbe0d6b937ee [id=747af828147149efb7b7b8d33c49286d]). Expected version 2 got 1
I20260130 02:58:03.034961 19527 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e47b22aafe5247c4b62c4193c24f427c" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
I20260130 02:58:03.036633 19313 leader_election.cc:304] T d3ba2df0b15c44f882070570f91b254d P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 78c2c46c7520492db35d51c244fb4f74, d870164263bb44b6b780a557daea3656; no voters: 371a36d219164974abdf848841528152
I20260130 02:58:03.037356 20083 raft_consensus.cc:2804] T d3ba2df0b15c44f882070570f91b254d P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Leader election won for term 1
I20260130 02:58:03.037814 20083 raft_consensus.cc:697] T d3ba2df0b15c44f882070570f91b254d P 78c2c46c7520492db35d51c244fb4f74 [term 1 LEADER]: Becoming Leader. State: Replica: 78c2c46c7520492db35d51c244fb4f74, State: Running, Role: LEADER
I20260130 02:58:03.038036 19840 tablet_bootstrap.cc:492] T ec28ff12927d450ead16d26cf016138d P 78c2c46c7520492db35d51c244fb4f74: No bootstrap required, opened a new log
I20260130 02:58:03.038599 19840 ts_tablet_manager.cc:1403] T ec28ff12927d450ead16d26cf016138d P 78c2c46c7520492db35d51c244fb4f74: Time spent bootstrapping tablet: real 0.091s user 0.018s sys 0.006s
I20260130 02:58:03.038635 20083 consensus_queue.cc:237] T d3ba2df0b15c44f882070570f91b254d P 78c2c46c7520492db35d51c244fb4f74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:03.041532 19840 raft_consensus.cc:359] T ec28ff12927d450ead16d26cf016138d P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:03.042690 19840 raft_consensus.cc:385] T ec28ff12927d450ead16d26cf016138d P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:03.044248 19840 raft_consensus.cc:740] T ec28ff12927d450ead16d26cf016138d P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78c2c46c7520492db35d51c244fb4f74, State: Initialized, Role: FOLLOWER
I20260130 02:58:03.045526 18871 catalog_manager.cc:5941] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4: Sending DeleteTablet for 3 replicas of tablet ceda64471a92476f9ba183d2beab30d1
I20260130 02:58:03.046979 18871 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:50284 (request call id 130) took 1597 ms. Trace:
I20260130 02:58:03.048763 19352 tablet_service.cc:1552] Processing DeleteTablet for tablet ceda64471a92476f9ba183d2beab30d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:58:01 UTC) from {username='slave'} at 127.0.0.1:60110
I20260130 02:58:03.047277 18871 rpcz_store.cc:276] 0130 02:58:01.449515 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:58:01.501909 (+ 52394us) service_pool.cc:225] Handling call
0130 02:58:01.579236 (+ 77327us) spinlock_profiling.cc:243] Waited 72.6 ms on lock 0x7b5400030554. stack: 00007fb0bd3f7373 00007fb0bd3f7270 00007fb0bcf6c574 0000000000508744 00000000005086d9 00007fb0c44490c0 00007fb0c44e7bba 00007fb0c61e6eac 00007fb0c62e2966 00007fb0c3f0a024 00007fb0c3f09fb2 00007fb0c3f09ee1 00007fb0c3f09e5c 00007fb0c3f09112 00007fb0c21dd34c 00007fb0c21dc786
0130 02:58:01.580244 (+ 1008us) catalog_manager.cc:2393] Looking up, locking, and authorizing table
0130 02:58:01.797485 (+217241us) catalog_manager.cc:2815] Modifying in-memory table state
0130 02:58:01.797630 (+ 145us) catalog_manager.cc:2824] Locking tablets
0130 02:58:02.004881 (+207251us) catalog_manager.cc:2838] Removing table and tablets from system table
0130 02:58:02.158254 (+153373us) catalog_manager.cc:2856] Removing table from by-name map
0130 02:58:02.173214 (+ 14960us) catalog_manager.cc:2872] Committing in-memory state
0130 02:58:03.045425 (+872211us) catalog_manager.cc:2876] Aborting table tasks
0130 02:58:03.046905 (+ 1480us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0130 02:58:02.022446 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0130 02:58:02.022862 (+ 416us) write_op.cc:432] Acquiring schema lock in shared mode
0130 02:58:02.022886 (+ 24us) write_op.cc:435] Acquired schema lock
0130 02:58:02.022898 (+ 12us) tablet.cc:660] Decoding operations
0130 02:58:02.023187 (+ 289us) write_op.cc:620] Acquiring the partition lock for write op
0130 02:58:02.023224 (+ 37us) write_op.cc:641] Partition lock acquired for write op
0130 02:58:02.023248 (+ 24us) tablet.cc:683] Acquiring locks for 2 operations
0130 02:58:02.023505 (+ 257us) tablet.cc:699] Row locks acquired
0130 02:58:02.023518 (+ 13us) write_op.cc:260] PREPARE: finished
0130 02:58:02.023622 (+ 104us) write_op.cc:270] Start()
0130 02:58:02.023699 (+ 77us) write_op.cc:276] Timestamp: P: 1769741882023582 usec, L: 0
0130 02:58:02.023713 (+ 14us) op_driver.cc:348] REPLICATION: starting
0130 02:58:02.024192 (+ 479us) log.cc:844] Serialized 771 byte log entry
0130 02:58:02.154683 (+130491us) op_driver.cc:464] REPLICATION: finished
0130 02:58:02.156575 (+ 1892us) write_op.cc:301] APPLY: starting
0130 02:58:02.156619 (+ 44us) tablet.cc:1365] starting BulkCheckPresence
0130 02:58:02.156717 (+ 98us) tablet.cc:1368] finished BulkCheckPresence
0130 02:58:02.156815 (+ 98us) tablet.cc:1370] starting ApplyRowOperation cycle
0130 02:58:02.157034 (+ 219us) tablet.cc:1381] finished ApplyRowOperation cycle
0130 02:58:02.157168 (+ 134us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0130 02:58:02.157209 (+ 41us) write_op.cc:312] APPLY: finished
0130 02:58:02.157362 (+ 153us) log.cc:844] Serialized 29 byte log entry
0130 02:58:02.157482 (+ 120us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:58:02.157541 (+ 59us) write_op.cc:454] Released schema lock
0130 02:58:02.157573 (+ 32us) write_op.cc:341] FINISH: Updating metrics
0130 02:58:02.157767 (+ 194us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:58:02.157792 (+ 25us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":166873984,"child_traces":[["op",{"apply.queue_time_us":1780,"apply.run_cpu_time_us":1166,"apply.run_wall_time_us":1166,"num_ops":2,"prepare.queue_time_us":15890,"prepare.run_cpu_time_us":1941,"prepare.run_wall_time_us":1941,"replication_time_us":130900}]]}
I20260130 02:58:03.050045 19758 tablet_replica.cc:333] T ceda64471a92476f9ba183d2beab30d1 P 78c2c46c7520492db35d51c244fb4f74: stopping tablet replica
I20260130 02:58:03.050770 20009 consensus_queue.cc:1048] T 955b9547f49f434699fe1d86b260638c P 78c2c46c7520492db35d51c244fb4f74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260130 02:58:03.052821 19204 tablet_service.cc:1552] Processing DeleteTablet for tablet ceda64471a92476f9ba183d2beab30d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:58:01 UTC) from {username='slave'} at 127.0.0.1:42870
I20260130 02:58:03.053314 19505 tablet_service.cc:1552] Processing DeleteTablet for tablet ceda64471a92476f9ba183d2beab30d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:58:01 UTC) from {username='slave'} at 127.0.0.1:59068
I20260130 02:58:03.053486 19754 tablet_replica.cc:333] T ceda64471a92476f9ba183d2beab30d1 P 371a36d219164974abdf848841528152: stopping tablet replica
I20260130 02:58:03.050555 18871 master_service.cc:954] ReplaceTablet: received request to replace tablet b7b596f944be4daf953b82b65d5cec2a from {username='slave'} at 127.0.0.1:50284
I20260130 02:58:03.053825 19758 raft_consensus.cc:2243] T ceda64471a92476f9ba183d2beab30d1 P 78c2c46c7520492db35d51c244fb4f74 [term 1 LEADER]: Raft consensus shutting down.
I20260130 02:58:03.054198 19754 raft_consensus.cc:2243] T ceda64471a92476f9ba183d2beab30d1 P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260130 02:58:03.054668 19758 raft_consensus.cc:2272] T ceda64471a92476f9ba183d2beab30d1 P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260130 02:58:03.054703 19754 raft_consensus.cc:2272] T ceda64471a92476f9ba183d2beab30d1 P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260130 02:58:03.055712 18868 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:50284 (request call id 140) took 1009 ms. Trace:
I20260130 02:58:03.056383 19758 ts_tablet_manager.cc:1916] T ceda64471a92476f9ba183d2beab30d1 P 78c2c46c7520492db35d51c244fb4f74: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260130 02:58:03.056392 19754 ts_tablet_manager.cc:1916] T ceda64471a92476f9ba183d2beab30d1 P 371a36d219164974abdf848841528152: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260130 02:58:03.063282 19840 consensus_queue.cc:260] T ec28ff12927d450ead16d26cf016138d P 78c2c46c7520492db35d51c244fb4f74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:03.056017 18868 rpcz_store.cc:276] 0130 02:58:02.046516 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:58:02.046677 (+ 161us) service_pool.cc:225] Handling call
0130 02:58:02.047709 (+ 1032us) catalog_manager.cc:2393] Looking up, locking, and authorizing table
0130 02:58:02.049257 (+ 1548us) catalog_manager.cc:3708] Apply alter schema
0130 02:58:02.049797 (+ 540us) catalog_manager.cc:3740] Acquired catalog manager lock
0130 02:58:02.049984 (+ 187us) catalog_manager.cc:3898] Updating metadata on disk
0130 02:58:02.172979 (+122995us) catalog_manager.cc:3929] Committing alterations to in-memory state
0130 02:58:03.055646 (+882667us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0130 02:58:02.095152 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0130 02:58:02.097043 (+ 1891us) write_op.cc:432] Acquiring schema lock in shared mode
0130 02:58:02.097067 (+ 24us) write_op.cc:435] Acquired schema lock
0130 02:58:02.097079 (+ 12us) tablet.cc:660] Decoding operations
0130 02:58:02.097330 (+ 251us) write_op.cc:620] Acquiring the partition lock for write op
0130 02:58:02.097382 (+ 52us) write_op.cc:641] Partition lock acquired for write op
0130 02:58:02.097405 (+ 23us) tablet.cc:683] Acquiring locks for 1 operations
0130 02:58:02.097565 (+ 160us) tablet.cc:699] Row locks acquired
0130 02:58:02.097578 (+ 13us) write_op.cc:260] PREPARE: finished
0130 02:58:02.097703 (+ 125us) write_op.cc:270] Start()
0130 02:58:02.097783 (+ 80us) write_op.cc:276] Timestamp: P: 1769741882097650 usec, L: 0
0130 02:58:02.097797 (+ 14us) op_driver.cc:348] REPLICATION: starting
0130 02:58:02.110247 (+ 12450us) spinlock_profiling.cc:243] Waited 9.12 ms on lock 0x7b5400030aa0. stack: 00007fb0bd3f7373 00007fb0bd3f7270 00007fb0bcf6c574 0000000000508744 00000000005086d9 00007fb0c5253964 00007fb0c44565db 00007fb0c445632a 00007fb0c44da5e5 00007fb0c44daa89 00007fb0c494c4d1 00007fb0c494bca8 00007fb0c4950941 00007fb0c49508f9 00007fb0c4950889 00007fb0c4950851
0130 02:58:02.110738 (+ 491us) log.cc:844] Serialized 405 byte log entry
0130 02:58:02.167601 (+ 56863us) op_driver.cc:464] REPLICATION: finished
0130 02:58:02.170709 (+ 3108us) write_op.cc:301] APPLY: starting
0130 02:58:02.170755 (+ 46us) tablet.cc:1365] starting BulkCheckPresence
0130 02:58:02.170844 (+ 89us) tablet.cc:1368] finished BulkCheckPresence
0130 02:58:02.170864 (+ 20us) tablet.cc:1370] starting ApplyRowOperation cycle
0130 02:58:02.170973 (+ 109us) tablet.cc:1381] finished ApplyRowOperation cycle
0130 02:58:02.171092 (+ 119us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0130 02:58:02.171134 (+ 42us) write_op.cc:312] APPLY: finished
0130 02:58:02.171287 (+ 153us) log.cc:844] Serialized 23 byte log entry
0130 02:58:02.171395 (+ 108us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:58:02.171451 (+ 56us) write_op.cc:454] Released schema lock
0130 02:58:02.171481 (+ 30us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":2955,"apply.run_cpu_time_us":944,"apply.run_wall_time_us":944,"num_ops":1,"prepare.queue_time_us":35168,"prepare.run_cpu_time_us":3515,"prepare.run_wall_time_us":23174,"raft.queue_time_us":7131,"raft.run_cpu_time_us":15,"raft.run_wall_time_us":15,"replication_time_us":69732,"spinlock_wait_cycles":20964608,"thread_start_us":22067,"threads_started":1}]]}
I20260130 02:58:03.070214 19947 ts_tablet_manager.cc:1929] T a46702bef0e6442d85e4ca49a5f7cfbb P d870164263bb44b6b780a557daea3656: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260130 02:58:03.071252 18868 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50284:
name: "default.Table_5588d44386964492b3b8a4f7bd19b520"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260130 02:58:03.072897 18868 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5588d44386964492b3b8a4f7bd19b520 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260130 02:58:03.076171 18869 catalog_manager.cc:5941] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4: Sending DeleteTablet for 3 replicas of tablet 481bd4cca6ee4c178873bf12392e3441
I20260130 02:58:03.076133 20025 tablet.cc:1722] T 1bbc55bbf2254d8bb6b721d7e8f69d85 P 371a36d219164974abdf848841528152: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260130 02:58:03.077477 18869 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:50284 (request call id 138) took 1056 ms. Trace:
I20260130 02:58:03.080947 19360 tablet_service.cc:1552] Processing DeleteTablet for tablet b078934be4bb4fd1915c691eeea9d00f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:58:01 UTC) from {username='slave'} at 127.0.0.1:60110
I20260130 02:58:03.081179 19352 tablet_service.cc:1552] Processing DeleteTablet for tablet 481bd4cca6ee4c178873bf12392e3441 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:58:02 UTC) from {username='slave'} at 127.0.0.1:60110
I20260130 02:58:03.081712 19354 tablet_service.cc:1552] Processing DeleteTablet for tablet 26560f746d244e58a182eb6bb24c8720 with delete_type TABLET_DATA_DELETED (replaced by tablet 341041cb0be2456e88403367c0814704) from {username='slave'} at 127.0.0.1:60110
I20260130 02:58:03.082288 19357 tablet_service.cc:1552] Processing DeleteTablet for tablet 5b2976108584478a98fd90fe6594af66 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:57:58 UTC) from {username='slave'} at 127.0.0.1:60110
I20260130 02:58:03.077757 18869 rpcz_store.cc:276] 0130 02:58:02.020584 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:58:02.024972 (+ 4388us) service_pool.cc:225] Handling call
0130 02:58:02.039263 (+ 14291us) catalog_manager.cc:2393] Looking up, locking, and authorizing table
0130 02:58:02.046003 (+ 6740us) catalog_manager.cc:2815] Modifying in-memory table state
0130 02:58:02.053059 (+ 7056us) catalog_manager.cc:2824] Locking tablets
0130 02:58:02.053708 (+ 649us) catalog_manager.cc:2838] Removing table and tablets from system table
0130 02:58:02.174882 (+121174us) catalog_manager.cc:2856] Removing table from by-name map
0130 02:58:02.175054 (+ 172us) catalog_manager.cc:2872] Committing in-memory state
0130 02:58:03.076080 (+901026us) catalog_manager.cc:2876] Aborting table tasks
0130 02:58:03.077419 (+ 1339us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0130 02:58:02.060297 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0130 02:58:02.060661 (+ 364us) write_op.cc:432] Acquiring schema lock in shared mode
0130 02:58:02.060684 (+ 23us) write_op.cc:435] Acquired schema lock
0130 02:58:02.060697 (+ 13us) tablet.cc:660] Decoding operations
0130 02:58:02.060997 (+ 300us) write_op.cc:620] Acquiring the partition lock for write op
0130 02:58:02.061032 (+ 35us) write_op.cc:641] Partition lock acquired for write op
0130 02:58:02.061055 (+ 23us) tablet.cc:683] Acquiring locks for 2 operations
0130 02:58:02.061290 (+ 235us) tablet.cc:699] Row locks acquired
0130 02:58:02.061304 (+ 14us) write_op.cc:260] PREPARE: finished
0130 02:58:02.061409 (+ 105us) write_op.cc:270] Start()
0130 02:58:02.061488 (+ 79us) write_op.cc:276] Timestamp: P: 1769741882061364 usec, L: 0
0130 02:58:02.061504 (+ 16us) op_driver.cc:348] REPLICATION: starting
0130 02:58:02.083588 (+ 22084us) log.cc:844] Serialized 731 byte log entry
0130 02:58:02.167408 (+ 83820us) op_driver.cc:464] REPLICATION: finished
0130 02:58:02.169930 (+ 2522us) write_op.cc:301] APPLY: starting
0130 02:58:02.169977 (+ 47us) tablet.cc:1365] starting BulkCheckPresence
0130 02:58:02.170071 (+ 94us) tablet.cc:1368] finished BulkCheckPresence
0130 02:58:02.170165 (+ 94us) tablet.cc:1370] starting ApplyRowOperation cycle
0130 02:58:02.170379 (+ 214us) tablet.cc:1381] finished ApplyRowOperation cycle
0130 02:58:02.170503 (+ 124us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0130 02:58:02.170544 (+ 41us) write_op.cc:312] APPLY: finished
0130 02:58:02.174143 (+ 3599us) log.cc:844] Serialized 29 byte log entry
0130 02:58:02.174275 (+ 132us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:58:02.174332 (+ 57us) write_op.cc:454] Released schema lock
0130 02:58:02.174361 (+ 29us) write_op.cc:341] FINISH: Updating metrics
0130 02:58:02.174554 (+ 193us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:58:02.174604 (+ 50us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":2405,"apply.run_cpu_time_us":1187,"apply.run_wall_time_us":4597,"num_ops":2,"prepare.queue_time_us":4996,"prepare.run_cpu_time_us":2079,"prepare.run_wall_time_us":34725,"raft.queue_time_us":673,"raft.run_cpu_time_us":27,"raft.run_wall_time_us":28,"replication_time_us":105827}]]}
I20260130 02:58:03.087693 19199 tablet_service.cc:1552] Processing DeleteTablet for tablet a46702bef0e6442d85e4ca49a5f7cfbb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:57:59 UTC) from {username='slave'} at 127.0.0.1:42870
I20260130 02:58:03.087663 18869 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50284:
name: "default.Table_ea2e0aa925a0486bb292304ca21bb819"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260130 02:58:03.088455 19201 tablet_service.cc:1552] Processing DeleteTablet for tablet d701152960b14396a6ac1bd10480d2b6 with delete_type TABLET_DATA_DELETED (replaced by tablet 481bd4cca6ee4c178873bf12392e3441) from {username='slave'} at 127.0.0.1:42870
I20260130 02:58:03.089064 19204 tablet_service.cc:1552] Processing DeleteTablet for tablet 481bd4cca6ee4c178873bf12392e3441 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:58:02 UTC) from {username='slave'} at 127.0.0.1:42870
W20260130 02:58:03.089583 18869 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ea2e0aa925a0486bb292304ca21bb819 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260130 02:58:03.089645 19198 tablet_service.cc:1552] Processing DeleteTablet for tablet 878f9346d3044d8d8b15c61f86869212 with delete_type TABLET_DATA_DELETED (replaced by tablet 3b148691090e4ee581d46f9845a6e49a) from {username='slave'} at 127.0.0.1:42870
W20260130 02:58:03.083209 18870 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.18.100.65:51021 (request call id 23) took 3113 ms (client timeout 1000 ms). Trace:
I20260130 02:58:03.092229 19358 tablet_service.cc:1552] Processing DeleteTablet for tablet d701152960b14396a6ac1bd10480d2b6 with delete_type TABLET_DATA_DELETED (replaced by tablet 481bd4cca6ee4c178873bf12392e3441) from {username='slave'} at 127.0.0.1:60110
I20260130 02:58:03.092854 19352 tablet_service.cc:1552] Processing DeleteTablet for tablet a46702bef0e6442d85e4ca49a5f7cfbb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:57:59 UTC) from {username='slave'} at 127.0.0.1:60110
I20260130 02:58:03.093441 19354 tablet_service.cc:1552] Processing DeleteTablet for tablet 073b5941850c4140b1751956dd7cf5ef with delete_type TABLET_DATA_DELETED (replaced by tablet 7d96e93d95f14af79cfb268626559b1e) from {username='slave'} at 127.0.0.1:60110
I20260130 02:58:03.094071 19357 tablet_service.cc:1552] Processing DeleteTablet for tablet 96584ca174b140f7ad1a2ca698baca71 with delete_type TABLET_DATA_DELETED (replaced by tablet 127a84346b9f473d9e7f739b1e65feb8) from {username='slave'} at 127.0.0.1:60110
I20260130 02:58:03.094620 19360 tablet_service.cc:1552] Processing DeleteTablet for tablet 7a902d0d571a42bb82ebffd48f7eca3f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:57:57 UTC) from {username='slave'} at 127.0.0.1:60110
W20260130 02:58:03.095852 18866 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.18.100.66:59691 (request call id 20) took 2132 ms (client timeout 1000 ms). Trace:
W20260130 02:58:03.091588 18870 rpcz_store.cc:269] 0130 02:57:59.970236 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:58:00.032242 (+ 62006us) service_pool.cc:225] Handling call
0130 02:58:00.058730 (+ 26488us) spinlock_profiling.cc:243] Waited 18.8 ms on lock 0x7b5400030554. stack: 00007fb0bd3f7373 00007fb0bd3f7270 00007fb0bcf6c574 0000000000508744 00000000005086d9 00007fb0c44490c0 00007fb0c44e7bba 00007fb0c61e6eac 00007fb0c62e110e 00007fb0c3efe4a4 00007fb0c3efe432 00007fb0c3efe361 00007fb0c3efe2dc 00007fb0c3efd592 00007fb0c21dd34c 00007fb0c21dc786
0130 02:58:03.083133 (+3024403us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0130 02:58:00.760540 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0130 02:58:00.760959 (+ 419us) write_op.cc:432] Acquiring schema lock in shared mode
0130 02:58:00.760982 (+ 23us) write_op.cc:435] Acquired schema lock
0130 02:58:00.760994 (+ 12us) tablet.cc:660] Decoding operations
0130 02:58:00.761298 (+ 304us) write_op.cc:620] Acquiring the partition lock for write op
0130 02:58:00.761338 (+ 40us) write_op.cc:641] Partition lock acquired for write op
0130 02:58:00.761360 (+ 22us) tablet.cc:683] Acquiring locks for 2 operations
0130 02:58:00.761641 (+ 281us) tablet.cc:699] Row locks acquired
0130 02:58:00.761655 (+ 14us) write_op.cc:260] PREPARE: finished
0130 02:58:00.761770 (+ 115us) write_op.cc:270] Start()
0130 02:58:00.761862 (+ 92us) write_op.cc:276] Timestamp: P: 1769741880761687 usec, L: 0
0130 02:58:00.761878 (+ 16us) op_driver.cc:348] REPLICATION: starting
0130 02:58:00.762358 (+ 480us) log.cc:844] Serialized 913 byte log entry
0130 02:58:00.941502 (+179144us) op_driver.cc:464] REPLICATION: finished
0130 02:58:00.941993 (+ 491us) write_op.cc:301] APPLY: starting
0130 02:58:00.942047 (+ 54us) tablet.cc:1365] starting BulkCheckPresence
0130 02:58:00.942145 (+ 98us) tablet.cc:1368] finished BulkCheckPresence
0130 02:58:00.942847 (+ 702us) tablet.cc:1370] starting ApplyRowOperation cycle
0130 02:58:00.943084 (+ 237us) tablet.cc:1381] finished ApplyRowOperation cycle
0130 02:58:00.943210 (+ 126us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0130 02:58:00.943249 (+ 39us) write_op.cc:312] APPLY: finished
0130 02:58:00.943413 (+ 164us) log.cc:844] Serialized 29 byte log entry
0130 02:58:00.943541 (+ 128us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:58:00.943620 (+ 79us) write_op.cc:454] Released schema lock
0130 02:58:00.943649 (+ 29us) write_op.cc:341] FINISH: Updating metrics
0130 02:58:00.943845 (+ 196us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:58:00.943866 (+ 21us) write_op.cc:454] Released schema lock
Related trace 'op':
0130 02:58:01.222859 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0130 02:58:01.223210 (+ 351us) write_op.cc:432] Acquiring schema lock in shared mode
0130 02:58:01.223233 (+ 23us) write_op.cc:435] Acquired schema lock
0130 02:58:01.223245 (+ 12us) tablet.cc:660] Decoding operations
0130 02:58:01.223473 (+ 228us) write_op.cc:620] Acquiring the partition lock for write op
0130 02:58:01.223507 (+ 34us) write_op.cc:641] Partition lock acquired for write op
0130 02:58:01.223528 (+ 21us) tablet.cc:683] Acquiring locks for 1 operations
0130 02:58:01.223666 (+ 138us) tablet.cc:699] Row locks acquired
0130 02:58:01.223679 (+ 13us) write_op.cc:260] PREPARE: finished
0130 02:58:01.223782 (+ 103us) write_op.cc:270] Start()
0130 02:58:01.223859 (+ 77us) write_op.cc:276] Timestamp: P: 1769741881223718 usec, L: 0
0130 02:58:01.223873 (+ 14us) op_driver.cc:348] REPLICATION: starting
0130 02:58:01.224315 (+ 442us) log.cc:844] Serialized 356 byte log entry
0130 02:58:01.343382 (+119067us) op_driver.cc:464] REPLICATION: finished
0130 02:58:01.344072 (+ 690us) write_op.cc:301] APPLY: starting
0130 02:58:01.344131 (+ 59us) tablet.cc:1365] starting BulkCheckPresence
0130 02:58:01.344211 (+ 80us) tablet.cc:1368] finished BulkCheckPresence
0130 02:58:01.344296 (+ 85us) tablet.cc:1370] starting ApplyRowOperation cycle
0130 02:58:01.344451 (+ 155us) tablet.cc:1381] finished ApplyRowOperation cycle
0130 02:58:01.344572 (+ 121us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0130 02:58:01.344621 (+ 49us) write_op.cc:312] APPLY: finished
0130 02:58:01.344785 (+ 164us) log.cc:844] Serialized 23 byte log entry
0130 02:58:01.344926 (+ 141us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:58:01.344985 (+ 59us) write_op.cc:454] Released schema lock
0130 02:58:01.345014 (+ 29us) write_op.cc:341] FINISH: Updating metrics
0130 02:58:01.345212 (+ 198us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:58:01.345234 (+ 22us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":37089,"reported_tablets":19,"spinlock_wait_cycles":43300736,"child_traces":[["op",{"apply.queue_time_us":360,"apply.run_cpu_time_us":1818,"apply.run_wall_time_us":1817,"num_ops":2,"prepare.queue_time_us":49337,"prepare.run_cpu_time_us":2158,"prepare.run_wall_time_us":2158,"raft.queue_time_us":522,"raft.run_cpu_time_us":678,"raft.run_wall_time_us":677,"replication_time_us":179540}],["op",{"apply.queue_time_us":538,"apply.run_cpu_time_us":1114,"apply.run_wall_time_us":1114,"num_ops":1,"prepare.queue_time_us":139,"prepare.run_cpu_time_us":1634,"prepare.run_wall_time_us":1635,"replication_time_us":119421}]]}
I20260130 02:58:03.073877 20079 tablet.cc:1722] T 1bbc55bbf2254d8bb6b721d7e8f69d85 P d870164263bb44b6b780a557daea3656: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260130 02:58:03.097512 19947 log.cc:1199] T a46702bef0e6442d85e4ca49a5f7cfbb P d870164263bb44b6b780a557daea3656: Deleting WAL directory at /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-2/wal/wals/a46702bef0e6442d85e4ca49a5f7cfbb
I20260130 02:58:03.098290 19947 ts_tablet_manager.cc:1950] T a46702bef0e6442d85e4ca49a5f7cfbb P d870164263bb44b6b780a557daea3656: Deleting consensus metadata
W20260130 02:58:03.096211 18866 rpcz_store.cc:269] 0130 02:58:00.963293 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:58:01.173015 (+209722us) service_pool.cc:225] Handling call
0130 02:58:01.772502 (+599487us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:58:01.772531 (+ 29us) write_op.cc:454] Released schema lock
0130 02:58:03.095788 (+1323257us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0130 02:58:01.755152 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0130 02:58:01.755515 (+ 363us) write_op.cc:432] Acquiring schema lock in shared mode
0130 02:58:01.755542 (+ 27us) write_op.cc:435] Acquired schema lock
0130 02:58:01.755555 (+ 13us) tablet.cc:660] Decoding operations
0130 02:58:01.755816 (+ 261us) write_op.cc:620] Acquiring the partition lock for write op
0130 02:58:01.755855 (+ 39us) write_op.cc:641] Partition lock acquired for write op
0130 02:58:01.755877 (+ 22us) tablet.cc:683] Acquiring locks for 1 operations
0130 02:58:01.756012 (+ 135us) tablet.cc:699] Row locks acquired
0130 02:58:01.756025 (+ 13us) write_op.cc:260] PREPARE: finished
0130 02:58:01.756129 (+ 104us) write_op.cc:270] Start()
0130 02:58:01.756206 (+ 77us) write_op.cc:276] Timestamp: P: 1769741881756070 usec, L: 0
0130 02:58:01.756220 (+ 14us) op_driver.cc:348] REPLICATION: starting
0130 02:58:01.756695 (+ 475us) log.cc:844] Serialized 533 byte log entry
0130 02:58:01.765708 (+ 9013us) op_driver.cc:464] REPLICATION: finished
0130 02:58:01.766181 (+ 473us) write_op.cc:301] APPLY: starting
0130 02:58:01.766237 (+ 56us) tablet.cc:1365] starting BulkCheckPresence
0130 02:58:01.766322 (+ 85us) tablet.cc:1368] finished BulkCheckPresence
0130 02:58:01.766409 (+ 87us) tablet.cc:1370] starting ApplyRowOperation cycle
0130 02:58:01.766599 (+ 190us) tablet.cc:1381] finished ApplyRowOperation cycle
0130 02:58:01.766717 (+ 118us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0130 02:58:01.766759 (+ 42us) write_op.cc:312] APPLY: finished
0130 02:58:01.766926 (+ 167us) log.cc:844] Serialized 23 byte log entry
0130 02:58:01.767048 (+ 122us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:58:01.767107 (+ 59us) write_op.cc:454] Released schema lock
0130 02:58:01.767135 (+ 28us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"reported_tablets":26,"child_traces":[["op",{"apply.queue_time_us":323,"apply.run_cpu_time_us":1114,"apply.run_wall_time_us":1113,"num_ops":1,"prepare.queue_time_us":98,"prepare.run_cpu_time_us":1833,"prepare.run_wall_time_us":1833,"raft.queue_time_us":2345,"raft.run_cpu_time_us":662,"raft.run_wall_time_us":665,"replication_time_us":9388}]]}
I20260130 02:58:03.098991 18866 master_service.cc:954] ReplaceTablet: received request to replace tablet 730bd1d7d15b4cbe945b3ef03b2455f7 from {username='slave'} at 127.0.0.1:50284
I20260130 02:58:03.101119 19947 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59068 (request call id 91) took 1689 ms. Trace:
I20260130 02:58:03.101455 19947 rpcz_store.cc:276] 0130 02:58:01.411745 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:58:01.493141 (+ 81396us) service_pool.cc:225] Handling call
0130 02:58:02.697541 (+1204400us) ts_tablet_manager.cc:1151] Deleting tablet a46702bef0e6442d85e4ca49a5f7cfbb
0130 02:58:02.697585 (+ 44us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0130 02:58:02.865080 (+167495us) tablet_metadata.cc:641] Metadata flushed
0130 02:58:03.069972 (+204892us) tablet_metadata.cc:641] Metadata flushed
0130 02:58:03.101058 (+ 31086us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":6107,"tablet-delete.queue_time_us":1202095}
I20260130 02:58:03.080056 19505 tablet_service.cc:1552] Processing DeleteTablet for tablet 481bd4cca6ee4c178873bf12392e3441 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:58:02 UTC) from {username='slave'} at 127.0.0.1:59068
I20260130 02:58:03.102430 19840 ts_tablet_manager.cc:1434] T ec28ff12927d450ead16d26cf016138d P 78c2c46c7520492db35d51c244fb4f74: Time spent starting tablet: real 0.063s user 0.005s sys 0.000s
I20260130 02:58:03.082204 19625 tablet_bootstrap.cc:654] T 7d76046ee6c8420fa4964decbe8209c1 P d870164263bb44b6b780a557daea3656: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:03.102947 18874 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:50284 (request call id 137) took 1188 ms. Trace:
I20260130 02:58:03.103672 19840 tablet_bootstrap.cc:492] T 481bd4cca6ee4c178873bf12392e3441 P 78c2c46c7520492db35d51c244fb4f74: Bootstrap starting.
I20260130 02:58:03.103219 18874 rpcz_store.cc:276] 0130 02:58:01.914161 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:58:01.919202 (+ 5041us) service_pool.cc:225] Handling call
0130 02:58:01.920223 (+ 1021us) catalog_manager.cc:2393] Looking up, locking, and authorizing table
0130 02:58:01.920997 (+ 774us) catalog_manager.cc:3708] Apply alter schema
0130 02:58:01.921549 (+ 552us) catalog_manager.cc:3740] Acquired catalog manager lock
0130 02:58:01.921741 (+ 192us) catalog_manager.cc:3898] Updating metadata on disk
0130 02:58:02.158378 (+236637us) catalog_manager.cc:3929] Committing alterations to in-memory state
0130 02:58:03.102884 (+944506us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0130 02:58:01.993214 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0130 02:58:01.993603 (+ 389us) write_op.cc:432] Acquiring schema lock in shared mode
0130 02:58:01.993632 (+ 29us) write_op.cc:435] Acquired schema lock
0130 02:58:01.993647 (+ 15us) tablet.cc:660] Decoding operations
0130 02:58:01.993928 (+ 281us) write_op.cc:620] Acquiring the partition lock for write op
0130 02:58:01.993966 (+ 38us) write_op.cc:641] Partition lock acquired for write op
0130 02:58:01.993987 (+ 21us) tablet.cc:683] Acquiring locks for 1 operations
0130 02:58:01.994132 (+ 145us) tablet.cc:699] Row locks acquired
0130 02:58:01.994144 (+ 12us) write_op.cc:260] PREPARE: finished
0130 02:58:01.994253 (+ 109us) write_op.cc:270] Start()
0130 02:58:01.994341 (+ 88us) write_op.cc:276] Timestamp: P: 1769741881994211 usec, L: 0
0130 02:58:01.994358 (+ 17us) op_driver.cc:348] REPLICATION: starting
0130 02:58:01.997895 (+ 3537us) log.cc:844] Serialized 405 byte log entry
0130 02:58:02.154422 (+156527us) op_driver.cc:464] REPLICATION: finished
0130 02:58:02.155075 (+ 653us) write_op.cc:301] APPLY: starting
0130 02:58:02.155133 (+ 58us) tablet.cc:1365] starting BulkCheckPresence
0130 02:58:02.155223 (+ 90us) tablet.cc:1368] finished BulkCheckPresence
0130 02:58:02.155332 (+ 109us) tablet.cc:1370] starting ApplyRowOperation cycle
0130 02:58:02.155495 (+ 163us) tablet.cc:1381] finished ApplyRowOperation cycle
0130 02:58:02.155625 (+ 130us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0130 02:58:02.155671 (+ 46us) write_op.cc:312] APPLY: finished
0130 02:58:02.155850 (+ 179us) log.cc:844] Serialized 23 byte log entry
0130 02:58:02.156007 (+ 157us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:58:02.156065 (+ 58us) write_op.cc:454] Released schema lock
0130 02:58:02.156107 (+ 42us) write_op.cc:341] FINISH: Updating metrics
0130 02:58:02.156321 (+ 214us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:58:02.156345 (+ 24us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":512,"apply.run_cpu_time_us":1215,"apply.run_wall_time_us":1214,"mutex_wait_us":70232,"num_ops":1,"prepare.queue_time_us":70380,"prepare.run_cpu_time_us":1902,"prepare.run_wall_time_us":9956,"replication_time_us":159972}]]}
I20260130 02:58:03.113240 18860 catalog_manager.cc:4985] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): tablet a46702bef0e6442d85e4ca49a5f7cfbb (table default.Table_97c3def679ff49dcb74a992828c65cde [id=59eb236fbb134702a61a33d9b15c046c]) successfully deleted
I20260130 02:58:03.106468 18874 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50284:
name: "default.Table_f4b80e13bbe24163a77efb7b60f5dc2c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260130 02:58:03.115954 19861 raft_consensus.cc:493] T ec28ff12927d450ead16d26cf016138d P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260130 02:58:03.116256 18874 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f4b80e13bbe24163a77efb7b60f5dc2c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260130 02:58:03.116398 19861 raft_consensus.cc:515] T ec28ff12927d450ead16d26cf016138d P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:03.118175 19861 leader_election.cc:290] T ec28ff12927d450ead16d26cf016138d P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d870164263bb44b6b780a557daea3656 (127.18.100.67:39571), 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767)
I20260130 02:58:03.120193 19378 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ec28ff12927d450ead16d26cf016138d" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74" is_pre_election: true
I20260130 02:58:03.120790 19378 raft_consensus.cc:2468] T ec28ff12927d450ead16d26cf016138d P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 371a36d219164974abdf848841528152 in term 0.
I20260130 02:58:03.121477 19527 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ec28ff12927d450ead16d26cf016138d" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
I20260130 02:58:03.123854 19159 leader_election.cc:304] T ec28ff12927d450ead16d26cf016138d P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 371a36d219164974abdf848841528152, 78c2c46c7520492db35d51c244fb4f74; no voters:
I20260130 02:58:03.124616 19861 raft_consensus.cc:2804] T ec28ff12927d450ead16d26cf016138d P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260130 02:58:03.124784 19527 raft_consensus.cc:2468] T ec28ff12927d450ead16d26cf016138d P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 371a36d219164974abdf848841528152 in term 0.
I20260130 02:58:03.124974 19861 raft_consensus.cc:493] T ec28ff12927d450ead16d26cf016138d P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260130 02:58:03.125265 19861 raft_consensus.cc:3060] T ec28ff12927d450ead16d26cf016138d P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:03.129359 19204 tablet_service.cc:1552] Processing DeleteTablet for tablet b078934be4bb4fd1915c691eeea9d00f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:58:01 UTC) from {username='slave'} at 127.0.0.1:42870
I20260130 02:58:03.130004 19198 tablet_service.cc:1552] Processing DeleteTablet for tablet 26560f746d244e58a182eb6bb24c8720 with delete_type TABLET_DATA_DELETED (replaced by tablet 341041cb0be2456e88403367c0814704) from {username='slave'} at 127.0.0.1:42870
I20260130 02:58:03.130622 19754 ts_tablet_manager.cc:1929] T ceda64471a92476f9ba183d2beab30d1 P 371a36d219164974abdf848841528152: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20260130 02:58:03.130973 19754 log.cc:1199] T ceda64471a92476f9ba183d2beab30d1 P 371a36d219164974abdf848841528152: Deleting WAL directory at /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-0/wal/wals/ceda64471a92476f9ba183d2beab30d1
I20260130 02:58:03.131065 19198 tablet_service.cc:1552] Processing DeleteTablet for tablet 5b2976108584478a98fd90fe6594af66 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:57:58 UTC) from {username='slave'} at 127.0.0.1:42870
I20260130 02:58:03.131657 19754 ts_tablet_manager.cc:1950] T ceda64471a92476f9ba183d2beab30d1 P 371a36d219164974abdf848841528152: Deleting consensus metadata
I20260130 02:58:03.131965 19198 tablet_service.cc:1552] Processing DeleteTablet for tablet 96584ca174b140f7ad1a2ca698baca71 with delete_type TABLET_DATA_DELETED (replaced by tablet 127a84346b9f473d9e7f739b1e65feb8) from {username='slave'} at 127.0.0.1:42870
I20260130 02:58:03.132934 19861 raft_consensus.cc:515] T ec28ff12927d450ead16d26cf016138d P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:03.133975 18861 catalog_manager.cc:4985] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021): tablet ceda64471a92476f9ba183d2beab30d1 (table default.Table_be9aa687d9744ac6a2aa335772c6cc9a [id=4f59d133b0cd4773bb012145927a92a4]) successfully deleted
I20260130 02:58:03.134690 19840 tablet_bootstrap.cc:654] T 481bd4cca6ee4c178873bf12392e3441 P 78c2c46c7520492db35d51c244fb4f74: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:03.137368 19758 ts_tablet_manager.cc:1929] T ceda64471a92476f9ba183d2beab30d1 P 78c2c46c7520492db35d51c244fb4f74: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20260130 02:58:03.137715 19758 log.cc:1199] T ceda64471a92476f9ba183d2beab30d1 P 78c2c46c7520492db35d51c244fb4f74: Deleting WAL directory at /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-1/wal/wals/ceda64471a92476f9ba183d2beab30d1
I20260130 02:58:03.138378 19758 ts_tablet_manager.cc:1950] T ceda64471a92476f9ba183d2beab30d1 P 78c2c46c7520492db35d51c244fb4f74: Deleting consensus metadata
I20260130 02:58:03.141021 18860 catalog_manager.cc:4985] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767): tablet ceda64471a92476f9ba183d2beab30d1 (table default.Table_be9aa687d9744ac6a2aa335772c6cc9a [id=4f59d133b0cd4773bb012145927a92a4]) successfully deleted
I20260130 02:58:03.135255 19807 tablet_bootstrap.cc:492] T d3ba2df0b15c44f882070570f91b254d P 371a36d219164974abdf848841528152: No bootstrap required, opened a new log
I20260130 02:58:03.141842 19807 ts_tablet_manager.cc:1403] T d3ba2df0b15c44f882070570f91b254d P 371a36d219164974abdf848841528152: Time spent bootstrapping tablet: real 0.172s user 0.010s sys 0.004s
I20260130 02:58:03.143785 19758 tablet_replica.cc:333] T 5b2976108584478a98fd90fe6594af66 P 78c2c46c7520492db35d51c244fb4f74: stopping tablet replica
I20260130 02:58:03.144506 19758 raft_consensus.cc:2243] T 5b2976108584478a98fd90fe6594af66 P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260130 02:58:03.144971 19758 raft_consensus.cc:2272] T 5b2976108584478a98fd90fe6594af66 P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260130 02:58:03.146670 19807 raft_consensus.cc:359] T d3ba2df0b15c44f882070570f91b254d P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:03.147279 19807 raft_consensus.cc:385] T d3ba2df0b15c44f882070570f91b254d P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20260130 02:58:03.147226 18860 catalog_manager.cc:4952] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767): delete failed for tablet b078934be4bb4fd1915c691eeea9d00f because the tablet was not found. No further retry: Not found: Tablet not found: b078934be4bb4fd1915c691eeea9d00f
I20260130 02:58:03.147541 19807 raft_consensus.cc:740] T d3ba2df0b15c44f882070570f91b254d P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 371a36d219164974abdf848841528152, State: Initialized, Role: FOLLOWER
W20260130 02:58:03.147913 18860 catalog_manager.cc:4965] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767): delete failed for tablet 481bd4cca6ee4c178873bf12392e3441 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 481bd4cca6ee4c178873bf12392e3441 already in progress: creating tablet
W20260130 02:58:03.148458 18860 catalog_manager.cc:4952] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767): delete failed for tablet 26560f746d244e58a182eb6bb24c8720 because the tablet was not found. No further retry: Not found: Tablet not found: 26560f746d244e58a182eb6bb24c8720
I20260130 02:58:03.148289 19807 consensus_queue.cc:260] T d3ba2df0b15c44f882070570f91b254d P 371a36d219164974abdf848841528152 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:03.150228 19807 ts_tablet_manager.cc:1434] T d3ba2df0b15c44f882070570f91b254d P 371a36d219164974abdf848841528152: Time spent starting tablet: real 0.008s user 0.004s sys 0.000s
I20260130 02:58:03.151162 19861 leader_election.cc:290] T ec28ff12927d450ead16d26cf016138d P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 election: Requested vote from peers d870164263bb44b6b780a557daea3656 (127.18.100.67:39571), 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767)
I20260130 02:58:03.152762 19378 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ec28ff12927d450ead16d26cf016138d" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74"
I20260130 02:58:03.152902 19527 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ec28ff12927d450ead16d26cf016138d" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656"
I20260130 02:58:03.153311 19378 raft_consensus.cc:3060] T ec28ff12927d450ead16d26cf016138d P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:03.153437 19527 raft_consensus.cc:3060] T ec28ff12927d450ead16d26cf016138d P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:03.153887 19807 tablet_bootstrap.cc:492] T b3f3177306914e3c8b253db54ee30fb6 P 371a36d219164974abdf848841528152: Bootstrap starting.
I20260130 02:58:03.163262 19378 raft_consensus.cc:2468] T ec28ff12927d450ead16d26cf016138d P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 371a36d219164974abdf848841528152 in term 1.
I20260130 02:58:03.164454 19159 leader_election.cc:304] T ec28ff12927d450ead16d26cf016138d P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 371a36d219164974abdf848841528152, 78c2c46c7520492db35d51c244fb4f74; no voters:
I20260130 02:58:03.165669 19861 raft_consensus.cc:2804] T ec28ff12927d450ead16d26cf016138d P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Leader election won for term 1
I20260130 02:58:03.166348 19861 raft_consensus.cc:697] T ec28ff12927d450ead16d26cf016138d P 371a36d219164974abdf848841528152 [term 1 LEADER]: Becoming Leader. State: Replica: 371a36d219164974abdf848841528152, State: Running, Role: LEADER
I20260130 02:58:03.175055 19861 consensus_queue.cc:237] T ec28ff12927d450ead16d26cf016138d P 371a36d219164974abdf848841528152 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:03.181161 19807 tablet_bootstrap.cc:654] T b3f3177306914e3c8b253db54ee30fb6 P 371a36d219164974abdf848841528152: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:03.185384 19947 tablet_replica.cc:333] T 073b5941850c4140b1751956dd7cf5ef P d870164263bb44b6b780a557daea3656: stopping tablet replica
I20260130 02:58:03.185741 18870 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50284:
name: "default.Table_09baf25d18fa489e9af0d7e289070082"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260130 02:58:03.186396 19947 raft_consensus.cc:2243] T 073b5941850c4140b1751956dd7cf5ef P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260130 02:58:03.189188 19625 tablet_bootstrap.cc:492] T 7d76046ee6c8420fa4964decbe8209c1 P d870164263bb44b6b780a557daea3656: No bootstrap required, opened a new log
I20260130 02:58:03.191358 19625 ts_tablet_manager.cc:1403] T 7d76046ee6c8420fa4964decbe8209c1 P d870164263bb44b6b780a557daea3656: Time spent bootstrapping tablet: real 0.264s user 0.006s sys 0.006s
I20260130 02:58:03.191879 19947 raft_consensus.cc:2272] T 073b5941850c4140b1751956dd7cf5ef P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260130 02:58:03.200212 19758 ts_tablet_manager.cc:1916] T 5b2976108584478a98fd90fe6594af66 P 78c2c46c7520492db35d51c244fb4f74: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260130 02:58:03.162351 19527 raft_consensus.cc:2468] T ec28ff12927d450ead16d26cf016138d P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 371a36d219164974abdf848841528152 in term 1.
W20260130 02:58:03.204249 18870 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_09baf25d18fa489e9af0d7e289070082 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260130 02:58:03.206202 19754 tablet_replica.cc:333] T d701152960b14396a6ac1bd10480d2b6 P 371a36d219164974abdf848841528152: stopping tablet replica
I20260130 02:58:03.207032 19754 raft_consensus.cc:2243] T d701152960b14396a6ac1bd10480d2b6 P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260130 02:58:03.207482 19754 raft_consensus.cc:2272] T d701152960b14396a6ac1bd10480d2b6 P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260130 02:58:03.193954 19625 raft_consensus.cc:359] T 7d76046ee6c8420fa4964decbe8209c1 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:03.219834 19625 raft_consensus.cc:385] T 7d76046ee6c8420fa4964decbe8209c1 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:03.219940 19840 tablet_bootstrap.cc:492] T 481bd4cca6ee4c178873bf12392e3441 P 78c2c46c7520492db35d51c244fb4f74: No bootstrap required, opened a new log
I20260130 02:58:03.220379 19840 ts_tablet_manager.cc:1403] T 481bd4cca6ee4c178873bf12392e3441 P 78c2c46c7520492db35d51c244fb4f74: Time spent bootstrapping tablet: real 0.117s user 0.008s sys 0.003s
I20260130 02:58:03.220706 19625 raft_consensus.cc:740] T 7d76046ee6c8420fa4964decbe8209c1 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d870164263bb44b6b780a557daea3656, State: Initialized, Role: FOLLOWER
I20260130 02:58:03.223044 19840 raft_consensus.cc:359] T 481bd4cca6ee4c178873bf12392e3441 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:03.223685 19840 raft_consensus.cc:385] T 481bd4cca6ee4c178873bf12392e3441 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20260130 02:58:03.223862 19313 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.100.67:39571, user_credentials={real_user=slave}} blocked reactor thread for 173495us
I20260130 02:58:03.222684 19947 ts_tablet_manager.cc:1916] T 073b5941850c4140b1751956dd7cf5ef P d870164263bb44b6b780a557daea3656: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260130 02:58:03.224983 19313 leader_election.cc:343] T e47b22aafe5247c4b62c4193c24f427c P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:03.225337 19313 leader_election.cc:304] T e47b22aafe5247c4b62c4193c24f427c P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78c2c46c7520492db35d51c244fb4f74; no voters: 371a36d219164974abdf848841528152, d870164263bb44b6b780a557daea3656
I20260130 02:58:03.225960 20083 raft_consensus.cc:2749] T e47b22aafe5247c4b62c4193c24f427c P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260130 02:58:03.229212 18875 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.18.100.65:51021 (request call id 24) took 1808 ms (client timeout 1000 ms). Trace:
W20260130 02:58:03.211241 18861 catalog_manager.cc:4952] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021): delete failed for tablet a46702bef0e6442d85e4ca49a5f7cfbb because the tablet was not found. No further retry: Not found: Tablet not found: a46702bef0e6442d85e4ca49a5f7cfbb
I20260130 02:58:03.231945 19198 tablet_service.cc:1552] Processing DeleteTablet for tablet 073b5941850c4140b1751956dd7cf5ef with delete_type TABLET_DATA_DELETED (replaced by tablet 7d96e93d95f14af79cfb268626559b1e) from {username='slave'} at 127.0.0.1:42870
I20260130 02:58:03.224677 19625 consensus_queue.cc:260] T 7d76046ee6c8420fa4964decbe8209c1 P d870164263bb44b6b780a557daea3656 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:03.232626 19198 tablet_service.cc:1552] Processing DeleteTablet for tablet 7a902d0d571a42bb82ebffd48f7eca3f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:57:57 UTC) from {username='slave'} at 127.0.0.1:42870
I20260130 02:58:03.223974 19840 raft_consensus.cc:740] T 481bd4cca6ee4c178873bf12392e3441 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78c2c46c7520492db35d51c244fb4f74, State: Initialized, Role: FOLLOWER
I20260130 02:58:03.237313 20079 tablet.cc:1722] T f2d01bfdc9c4494386bd02bed655fe63 P d870164263bb44b6b780a557daea3656: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20260130 02:58:03.239570 19840 consensus_queue.cc:260] T 481bd4cca6ee4c178873bf12392e3441 P 78c2c46c7520492db35d51c244fb4f74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:03.241607 19840 ts_tablet_manager.cc:1434] T 481bd4cca6ee4c178873bf12392e3441 P 78c2c46c7520492db35d51c244fb4f74: Time spent starting tablet: real 0.021s user 0.005s sys 0.000s
I20260130 02:58:03.252283 20044 raft_consensus.cc:493] T 7d76046ee6c8420fa4964decbe8209c1 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:03.252723 20044 raft_consensus.cc:515] T 7d76046ee6c8420fa4964decbe8209c1 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:03.256281 19754 ts_tablet_manager.cc:1916] T d701152960b14396a6ac1bd10480d2b6 P 371a36d219164974abdf848841528152: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260130 02:58:03.263007 19223 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7d76046ee6c8420fa4964decbe8209c1" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152" is_pre_election: true
I20260130 02:58:03.264051 20025 tablet.cc:1722] T f2d01bfdc9c4494386bd02bed655fe63 P 371a36d219164974abdf848841528152: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20260130 02:58:03.265348 20044 leader_election.cc:290] T 7d76046ee6c8420fa4964decbe8209c1 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767)
W20260130 02:58:03.267575 19463 leader_election.cc:343] T 7d76046ee6c8420fa4964decbe8209c1 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:03.272148 19625 ts_tablet_manager.cc:1434] T 7d76046ee6c8420fa4964decbe8209c1 P d870164263bb44b6b780a557daea3656: Time spent starting tablet: real 0.080s user 0.004s sys 0.002s
W20260130 02:58:03.230197 18875 rpcz_store.cc:269] 0130 02:58:01.421062 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:58:01.484213 (+ 63151us) service_pool.cc:225] Handling call
0130 02:58:01.580718 (+ 96505us) spinlock_profiling.cc:243] Waited 77.3 ms on lock 0x7b5400030554. stack: 00007fb0bd3f7373 00007fb0bd3f7270 00007fb0bcf6c574 0000000000508744 00000000005086d9 00007fb0c44490c0 00007fb0c44e7bba 00007fb0c61e6eac 00007fb0c62e110e 00007fb0c3efe4a4 00007fb0c3efe432 00007fb0c3efe361 00007fb0c3efe2dc 00007fb0c3efd592 00007fb0c21dd34c 00007fb0c21dc786
0130 02:58:03.229140 (+1648422us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0130 02:58:01.948744 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0130 02:58:01.949096 (+ 352us) write_op.cc:432] Acquiring schema lock in shared mode
0130 02:58:01.949118 (+ 22us) write_op.cc:435] Acquired schema lock
0130 02:58:01.949129 (+ 11us) tablet.cc:660] Decoding operations
0130 02:58:01.949358 (+ 229us) write_op.cc:620] Acquiring the partition lock for write op
0130 02:58:01.949392 (+ 34us) write_op.cc:641] Partition lock acquired for write op
0130 02:58:01.949412 (+ 20us) tablet.cc:683] Acquiring locks for 1 operations
0130 02:58:01.949539 (+ 127us) tablet.cc:699] Row locks acquired
0130 02:58:01.949551 (+ 12us) write_op.cc:260] PREPARE: finished
0130 02:58:01.949645 (+ 94us) write_op.cc:270] Start()
0130 02:58:01.949718 (+ 73us) write_op.cc:276] Timestamp: P: 1769741881949566 usec, L: 0
0130 02:58:01.949733 (+ 15us) op_driver.cc:348] REPLICATION: starting
0130 02:58:01.950150 (+ 417us) log.cc:844] Serialized 533 byte log entry
0130 02:58:01.999303 (+ 49153us) op_driver.cc:464] REPLICATION: finished
0130 02:58:02.000993 (+ 1690us) write_op.cc:301] APPLY: starting
0130 02:58:02.001043 (+ 50us) tablet.cc:1365] starting BulkCheckPresence
0130 02:58:02.001128 (+ 85us) tablet.cc:1368] finished BulkCheckPresence
0130 02:58:02.001212 (+ 84us) tablet.cc:1370] starting ApplyRowOperation cycle
0130 02:58:02.001363 (+ 151us) tablet.cc:1381] finished ApplyRowOperation cycle
0130 02:58:02.001480 (+ 117us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0130 02:58:02.001518 (+ 38us) write_op.cc:312] APPLY: finished
0130 02:58:02.001664 (+ 146us) log.cc:844] Serialized 23 byte log entry
0130 02:58:02.001778 (+ 114us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:58:02.001831 (+ 53us) write_op.cc:454] Released schema lock
0130 02:58:02.001873 (+ 42us) write_op.cc:341] FINISH: Updating metrics
0130 02:58:02.002078 (+ 205us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:58:02.002102 (+ 24us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":24,"spinlock_wait_cycles":177903872,"child_traces":[["op",{"apply.queue_time_us":1563,"apply.run_cpu_time_us":1053,"apply.run_wall_time_us":1054,"mutex_wait_us":44511,"num_ops":1,"prepare.queue_time_us":44654,"prepare.run_cpu_time_us":1658,"prepare.run_wall_time_us":5542,"raft.queue_time_us":69,"raft.run_cpu_time_us":11,"raft.run_wall_time_us":13,"replication_time_us":49497}]]}
I20260130 02:58:03.283521 19625 tablet_bootstrap.cc:492] T 5ca9919f80954e74a4d91c94b74de0e4 P d870164263bb44b6b780a557daea3656: Bootstrap starting.
I20260130 02:58:03.285077 20083 raft_consensus.cc:493] T 7d76046ee6c8420fa4964decbe8209c1 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:03.285531 20083 raft_consensus.cc:515] T 7d76046ee6c8420fa4964decbe8209c1 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:03.293586 19807 tablet_bootstrap.cc:492] T b3f3177306914e3c8b253db54ee30fb6 P 371a36d219164974abdf848841528152: No bootstrap required, opened a new log
I20260130 02:58:03.294019 19807 ts_tablet_manager.cc:1403] T b3f3177306914e3c8b253db54ee30fb6 P 371a36d219164974abdf848841528152: Time spent bootstrapping tablet: real 0.140s user 0.009s sys 0.009s
I20260130 02:58:03.296463 19840 tablet_bootstrap.cc:492] T 534dc31a9b7e492c82f3672c75742e6c P 78c2c46c7520492db35d51c244fb4f74: Bootstrap starting.
I20260130 02:58:03.297660 19807 raft_consensus.cc:359] T b3f3177306914e3c8b253db54ee30fb6 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:03.302793 20089 raft_consensus.cc:493] T 5ca9919f80954e74a4d91c94b74de0e4 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:03.303248 20089 raft_consensus.cc:515] T 5ca9919f80954e74a4d91c94b74de0e4 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:03.307658 20089 leader_election.cc:290] T 5ca9919f80954e74a4d91c94b74de0e4 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), d870164263bb44b6b780a557daea3656 (127.18.100.67:39571)
I20260130 02:58:03.308676 20083 leader_election.cc:290] T 7d76046ee6c8420fa4964decbe8209c1 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), d870164263bb44b6b780a557daea3656 (127.18.100.67:39571)
I20260130 02:58:03.309757 19223 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5ca9919f80954e74a4d91c94b74de0e4" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152" is_pre_election: true
W20260130 02:58:03.316350 19314 leader_election.cc:343] T 5ca9919f80954e74a4d91c94b74de0e4 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:03.298295 19807 raft_consensus.cc:385] T b3f3177306914e3c8b253db54ee30fb6 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:03.316841 19377 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7d76046ee6c8420fa4964decbe8209c1" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74" is_pre_election: true
I20260130 02:58:03.317101 19807 raft_consensus.cc:740] T b3f3177306914e3c8b253db54ee30fb6 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 371a36d219164974abdf848841528152, State: Initialized, Role: FOLLOWER
I20260130 02:58:03.317396 19377 raft_consensus.cc:2468] T 7d76046ee6c8420fa4964decbe8209c1 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d870164263bb44b6b780a557daea3656 in term 0.
I20260130 02:58:03.320973 19807 consensus_queue.cc:260] T b3f3177306914e3c8b253db54ee30fb6 P 371a36d219164974abdf848841528152 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:03.310721 19221 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7d76046ee6c8420fa4964decbe8209c1" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152" is_pre_election: true
W20260130 02:58:03.323361 19314 leader_election.cc:343] T 7d76046ee6c8420fa4964decbe8209c1 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:03.324033 19462 leader_election.cc:304] T 7d76046ee6c8420fa4964decbe8209c1 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 78c2c46c7520492db35d51c244fb4f74, d870164263bb44b6b780a557daea3656; no voters: 371a36d219164974abdf848841528152
I20260130 02:58:03.325356 19971 raft_consensus.cc:2804] T 7d76046ee6c8420fa4964decbe8209c1 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260130 02:58:03.325731 19971 raft_consensus.cc:493] T 7d76046ee6c8420fa4964decbe8209c1 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260130 02:58:03.326027 19971 raft_consensus.cc:3060] T 7d76046ee6c8420fa4964decbe8209c1 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:03.328094 19525 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7d76046ee6c8420fa4964decbe8209c1" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
I20260130 02:58:03.336217 20083 consensus_queue.cc:1048] T a743a82540014c85aeab0e9d0994b0dc P 78c2c46c7520492db35d51c244fb4f74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260130 02:58:03.337438 19971 raft_consensus.cc:515] T 7d76046ee6c8420fa4964decbe8209c1 P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:03.338763 19525 raft_consensus.cc:2393] T 7d76046ee6c8420fa4964decbe8209c1 P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 78c2c46c7520492db35d51c244fb4f74 in current term 1: Already voted for candidate d870164263bb44b6b780a557daea3656 in this term.
I20260130 02:58:03.336220 19758 ts_tablet_manager.cc:1929] T 5b2976108584478a98fd90fe6594af66 P 78c2c46c7520492db35d51c244fb4f74: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260130 02:58:03.343227 19807 ts_tablet_manager.cc:1434] T b3f3177306914e3c8b253db54ee30fb6 P 371a36d219164974abdf848841528152: Time spent starting tablet: real 0.049s user 0.003s sys 0.005s
I20260130 02:58:03.347532 19758 log.cc:1199] T 5b2976108584478a98fd90fe6594af66 P 78c2c46c7520492db35d51c244fb4f74: Deleting WAL directory at /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-1/wal/wals/5b2976108584478a98fd90fe6594af66
I20260130 02:58:03.348287 19758 ts_tablet_manager.cc:1950] T 5b2976108584478a98fd90fe6594af66 P 78c2c46c7520492db35d51c244fb4f74: Deleting consensus metadata
I20260130 02:58:03.350370 20089 raft_consensus.cc:493] T 2ca0df71836145748cada4e4559a5656 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:03.350840 19527 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5ca9919f80954e74a4d91c94b74de0e4" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
I20260130 02:58:03.350832 20089 raft_consensus.cc:515] T 2ca0df71836145748cada4e4559a5656 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:03.353518 20089 leader_election.cc:290] T 2ca0df71836145748cada4e4559a5656 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), d870164263bb44b6b780a557daea3656 (127.18.100.67:39571)
I20260130 02:58:03.354218 19221 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ca0df71836145748cada4e4559a5656" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152" is_pre_election: true
W20260130 02:58:03.355441 19314 leader_election.cc:343] T 2ca0df71836145748cada4e4559a5656 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:03.357169 19947 ts_tablet_manager.cc:1929] T 073b5941850c4140b1751956dd7cf5ef P d870164263bb44b6b780a557daea3656: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260130 02:58:03.357595 19947 log.cc:1199] T 073b5941850c4140b1751956dd7cf5ef P d870164263bb44b6b780a557daea3656: Deleting WAL directory at /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-2/wal/wals/073b5941850c4140b1751956dd7cf5ef
I20260130 02:58:03.358042 19625 tablet_bootstrap.cc:654] T 5ca9919f80954e74a4d91c94b74de0e4 P d870164263bb44b6b780a557daea3656: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:03.358301 19947 ts_tablet_manager.cc:1950] T 073b5941850c4140b1751956dd7cf5ef P d870164263bb44b6b780a557daea3656: Deleting consensus metadata
I20260130 02:58:03.359324 19807 tablet_bootstrap.cc:492] T 7d76046ee6c8420fa4964decbe8209c1 P 371a36d219164974abdf848841528152: Bootstrap starting.
I20260130 02:58:03.363438 18860 catalog_manager.cc:4985] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767): tablet 5b2976108584478a98fd90fe6594af66 (table default.Table_a78ce7ab85924506b184fb74db3ed0c3 [id=5f4c327b216449a1a8b146d0f652de52]) successfully deleted
I20260130 02:58:03.363816 18874 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50284:
table { table_name: "default.Table_1c2b251034c0418cba750ae4d1308b9f" } modify_external_catalogs: true
I20260130 02:58:03.364480 18874 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50284:
table { table_name: "default.Table_1c2b251034c0418cba750ae4d1308b9f" } modify_external_catalogs: true
I20260130 02:58:03.365073 19971 leader_election.cc:290] T 7d76046ee6c8420fa4964decbe8209c1 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 election: Requested vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767)
I20260130 02:58:03.365744 19758 tablet_replica.cc:333] T d701152960b14396a6ac1bd10480d2b6 P 78c2c46c7520492db35d51c244fb4f74: stopping tablet replica
I20260130 02:58:03.365839 19221 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7d76046ee6c8420fa4964decbe8209c1" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152"
I20260130 02:58:03.366485 19758 raft_consensus.cc:2243] T d701152960b14396a6ac1bd10480d2b6 P 78c2c46c7520492db35d51c244fb4f74 [term 1 LEADER]: Raft consensus shutting down.
W20260130 02:58:03.366956 19463 leader_election.cc:343] T 7d76046ee6c8420fa4964decbe8209c1 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:03.367374 19758 raft_consensus.cc:2272] T d701152960b14396a6ac1bd10480d2b6 P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260130 02:58:03.369114 19758 ts_tablet_manager.cc:1916] T d701152960b14396a6ac1bd10480d2b6 P 78c2c46c7520492db35d51c244fb4f74: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260130 02:58:03.369518 19313 leader_election.cc:304] T 7d76046ee6c8420fa4964decbe8209c1 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78c2c46c7520492db35d51c244fb4f74; no voters: 371a36d219164974abdf848841528152, d870164263bb44b6b780a557daea3656
I20260130 02:58:03.371825 20083 raft_consensus.cc:3060] T 7d76046ee6c8420fa4964decbe8209c1 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:03.375267 19947 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59068 (request call id 95) took 1773 ms. Trace:
I20260130 02:58:03.375648 19947 rpcz_store.cc:276] 0130 02:58:01.601527 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:58:01.601646 (+ 119us) service_pool.cc:225] Handling call
0130 02:58:03.185213 (+1583567us) ts_tablet_manager.cc:1151] Deleting tablet 073b5941850c4140b1751956dd7cf5ef
0130 02:58:03.185263 (+ 50us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0130 02:58:03.293906 (+108643us) tablet_metadata.cc:641] Metadata flushed
0130 02:58:03.356927 (+ 63021us) tablet_metadata.cc:641] Metadata flushed
0130 02:58:03.375199 (+ 18272us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1511502}
I20260130 02:58:03.376953 18860 catalog_manager.cc:4985] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): tablet 073b5941850c4140b1751956dd7cf5ef (table default.Table_6eb0a92b1926487cbf0b978d26d6a7b1 [id=de7a420d40864f55b156e149308ba12f]) successfully deleted
I20260130 02:58:03.379211 20083 raft_consensus.cc:2749] T 7d76046ee6c8420fa4964decbe8209c1 P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260130 02:58:03.380185 19313 leader_election.cc:343] T 5ca9919f80954e74a4d91c94b74de0e4 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:03.380436 18869 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50284:
table { table_name: "default.Table_50c3bb8b3ede494ba9b9f826a2adba72" } new_table_name: "default.Table_3a858a3cb886427cb8268f42c953ad4c" modify_external_catalogs: true
I20260130 02:58:03.380525 19313 leader_election.cc:304] T 5ca9919f80954e74a4d91c94b74de0e4 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78c2c46c7520492db35d51c244fb4f74; no voters: 371a36d219164974abdf848841528152, d870164263bb44b6b780a557daea3656
I20260130 02:58:03.382709 19947 tablet_replica.cc:333] T 96584ca174b140f7ad1a2ca698baca71 P d870164263bb44b6b780a557daea3656: stopping tablet replica
I20260130 02:58:03.383610 19947 raft_consensus.cc:2243] T 96584ca174b140f7ad1a2ca698baca71 P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260130 02:58:03.383985 19840 tablet_bootstrap.cc:654] T 534dc31a9b7e492c82f3672c75742e6c P 78c2c46c7520492db35d51c244fb4f74: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:03.384172 19947 raft_consensus.cc:2272] T 96584ca174b140f7ad1a2ca698baca71 P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260130 02:58:03.387295 19861 raft_consensus.cc:493] T d3ba2df0b15c44f882070570f91b254d P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:03.387769 19861 raft_consensus.cc:515] T d3ba2df0b15c44f882070570f91b254d P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:03.389601 19861 leader_election.cc:290] T d3ba2df0b15c44f882070570f91b254d P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d870164263bb44b6b780a557daea3656 (127.18.100.67:39571), 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767)
I20260130 02:58:03.393812 19380 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d3ba2df0b15c44f882070570f91b254d" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74" is_pre_election: true
I20260130 02:58:03.395475 20083 raft_consensus.cc:2749] T 5ca9919f80954e74a4d91c94b74de0e4 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260130 02:58:03.397871 19861 raft_consensus.cc:493] T bd43c5d148514836869a8ac648c8ad11 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:03.398312 19861 raft_consensus.cc:515] T bd43c5d148514836869a8ac648c8ad11 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:03.400569 19861 leader_election.cc:290] T bd43c5d148514836869a8ac648c8ad11 P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767), d870164263bb44b6b780a557daea3656 (127.18.100.67:39571)
I20260130 02:58:03.401801 19525 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ca0df71836145748cada4e4559a5656" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
I20260130 02:58:03.407351 19525 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bd43c5d148514836869a8ac648c8ad11" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
W20260130 02:58:03.407689 18869 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 54be22352e404ff9b798a075693f7f21 consensus state has no leader
I20260130 02:58:03.408017 19525 raft_consensus.cc:2468] T bd43c5d148514836869a8ac648c8ad11 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 371a36d219164974abdf848841528152 in term 0.
I20260130 02:58:03.409077 19159 leader_election.cc:304] T bd43c5d148514836869a8ac648c8ad11 P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 371a36d219164974abdf848841528152, d870164263bb44b6b780a557daea3656; no voters:
W20260130 02:58:03.409874 19159 leader_election.cc:343] T bd43c5d148514836869a8ac648c8ad11 P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767): Not found: Tablet not found: bd43c5d148514836869a8ac648c8ad11
I20260130 02:58:03.410646 19861 raft_consensus.cc:2804] T bd43c5d148514836869a8ac648c8ad11 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260130 02:58:03.411005 19861 raft_consensus.cc:493] T bd43c5d148514836869a8ac648c8ad11 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260130 02:58:03.411307 19861 raft_consensus.cc:3060] T bd43c5d148514836869a8ac648c8ad11 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:03.414842 19527 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d3ba2df0b15c44f882070570f91b254d" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
I20260130 02:58:03.415620 19527 raft_consensus.cc:2393] T d3ba2df0b15c44f882070570f91b254d P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 371a36d219164974abdf848841528152 in current term 1: Already voted for candidate 78c2c46c7520492db35d51c244fb4f74 in this term.
I20260130 02:58:03.416738 19159 leader_election.cc:304] T d3ba2df0b15c44f882070570f91b254d P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 371a36d219164974abdf848841528152; no voters: 78c2c46c7520492db35d51c244fb4f74, d870164263bb44b6b780a557daea3656
I20260130 02:58:03.417702 19974 raft_consensus.cc:3060] T d3ba2df0b15c44f882070570f91b254d P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:03.386168 19947 ts_tablet_manager.cc:1916] T 96584ca174b140f7ad1a2ca698baca71 P d870164263bb44b6b780a557daea3656: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260130 02:58:03.420210 18870 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50284:
table { table_name: "default.Table_87dbbd65d92043b0b91c801517e56dfe" } modify_external_catalogs: true
I20260130 02:58:03.420717 18870 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50284:
table { table_name: "default.Table_87dbbd65d92043b0b91c801517e56dfe" } modify_external_catalogs: true
I20260130 02:58:03.422370 18869 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50284:
table { table_name: "default.Table_4d43592dc5b24ff28710e62ad4d87ce0" } new_table_name: "default.Table_f359d249f8274f319f5ff4e4ea89dd22" modify_external_catalogs: true
I20260130 02:58:03.425916 19974 raft_consensus.cc:2749] T d3ba2df0b15c44f882070570f91b254d P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260130 02:58:03.430832 19807 tablet_bootstrap.cc:654] T 7d76046ee6c8420fa4964decbe8209c1 P 371a36d219164974abdf848841528152: Neither blocks nor log segments found. Creating new log.
W20260130 02:58:03.432035 18862 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet 54be22352e404ff9b798a075693f7f21 (table default.Table_3a858a3cb886427cb8268f42c953ad4c [id=17150cb73e374ff9a1e92c43e0c3f2c1], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 54be22352e404ff9b798a075693f7f21 consensus state has no leader
I20260130 02:58:03.418658 19625 tablet_bootstrap.cc:492] T 5ca9919f80954e74a4d91c94b74de0e4 P d870164263bb44b6b780a557daea3656: No bootstrap required, opened a new log
I20260130 02:58:03.436030 19625 ts_tablet_manager.cc:1403] T 5ca9919f80954e74a4d91c94b74de0e4 P d870164263bb44b6b780a557daea3656: Time spent bootstrapping tablet: real 0.153s user 0.010s sys 0.001s
I20260130 02:58:03.422226 19861 raft_consensus.cc:515] T bd43c5d148514836869a8ac648c8ad11 P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
W20260130 02:58:03.442438 19159 leader_election.cc:343] T bd43c5d148514836869a8ac648c8ad11 P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767): Not found: Tablet not found: bd43c5d148514836869a8ac648c8ad11
I20260130 02:58:03.444401 19527 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bd43c5d148514836869a8ac648c8ad11" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656"
I20260130 02:58:03.444911 19527 raft_consensus.cc:3060] T bd43c5d148514836869a8ac648c8ad11 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:03.445792 19377 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7d76046ee6c8420fa4964decbe8209c1" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74"
I20260130 02:58:03.452535 19377 raft_consensus.cc:2468] T 7d76046ee6c8420fa4964decbe8209c1 P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d870164263bb44b6b780a557daea3656 in term 1.
I20260130 02:58:03.454802 19840 tablet_bootstrap.cc:492] T 534dc31a9b7e492c82f3672c75742e6c P 78c2c46c7520492db35d51c244fb4f74: No bootstrap required, opened a new log
I20260130 02:58:03.456054 19974 raft_consensus.cc:493] T a743a82540014c85aeab0e9d0994b0dc P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:03.456601 19974 raft_consensus.cc:515] T a743a82540014c85aeab0e9d0994b0dc P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:03.456866 19625 raft_consensus.cc:359] T 5ca9919f80954e74a4d91c94b74de0e4 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:03.457531 19625 raft_consensus.cc:385] T 5ca9919f80954e74a4d91c94b74de0e4 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:03.461608 19525 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a743a82540014c85aeab0e9d0994b0dc" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
I20260130 02:58:03.462740 19861 leader_election.cc:290] T bd43c5d148514836869a8ac648c8ad11 P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 election: Requested vote from peers 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767), d870164263bb44b6b780a557daea3656 (127.18.100.67:39571)
I20260130 02:58:03.463194 19861 raft_consensus.cc:493] T b3f3177306914e3c8b253db54ee30fb6 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:03.463541 19861 raft_consensus.cc:515] T b3f3177306914e3c8b253db54ee30fb6 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:03.465066 19861 leader_election.cc:290] T b3f3177306914e3c8b253db54ee30fb6 P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d870164263bb44b6b780a557daea3656 (127.18.100.67:39571), 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767)
I20260130 02:58:03.465996 19525 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b3f3177306914e3c8b253db54ee30fb6" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
I20260130 02:58:03.466776 19525 raft_consensus.cc:2393] T b3f3177306914e3c8b253db54ee30fb6 P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 371a36d219164974abdf848841528152 in current term 1: Already voted for candidate 78c2c46c7520492db35d51c244fb4f74 in this term.
I20260130 02:58:03.468643 19527 raft_consensus.cc:2468] T bd43c5d148514836869a8ac648c8ad11 P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 371a36d219164974abdf848841528152 in term 1.
I20260130 02:58:03.470111 19159 leader_election.cc:304] T bd43c5d148514836869a8ac648c8ad11 P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 371a36d219164974abdf848841528152, d870164263bb44b6b780a557daea3656; no voters: 78c2c46c7520492db35d51c244fb4f74
I20260130 02:58:03.472038 19807 tablet_bootstrap.cc:492] T 7d76046ee6c8420fa4964decbe8209c1 P 371a36d219164974abdf848841528152: No bootstrap required, opened a new log
I20260130 02:58:03.472430 19807 ts_tablet_manager.cc:1403] T 7d76046ee6c8420fa4964decbe8209c1 P 371a36d219164974abdf848841528152: Time spent bootstrapping tablet: real 0.116s user 0.003s sys 0.005s
I20260130 02:58:03.474540 19974 leader_election.cc:290] T a743a82540014c85aeab0e9d0994b0dc P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767), d870164263bb44b6b780a557daea3656 (127.18.100.67:39571)
I20260130 02:58:03.475664 19974 raft_consensus.cc:2804] T bd43c5d148514836869a8ac648c8ad11 P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Leader election won for term 1
I20260130 02:58:03.476568 19974 raft_consensus.cc:697] T bd43c5d148514836869a8ac648c8ad11 P 371a36d219164974abdf848841528152 [term 1 LEADER]: Becoming Leader. State: Replica: 371a36d219164974abdf848841528152, State: Running, Role: LEADER
I20260130 02:58:03.477811 19974 consensus_queue.cc:237] T bd43c5d148514836869a8ac648c8ad11 P 371a36d219164974abdf848841528152 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:03.480544 19807 raft_consensus.cc:359] T 7d76046ee6c8420fa4964decbe8209c1 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:03.481176 19807 raft_consensus.cc:385] T 7d76046ee6c8420fa4964decbe8209c1 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:03.481432 19807 raft_consensus.cc:740] T 7d76046ee6c8420fa4964decbe8209c1 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 371a36d219164974abdf848841528152, State: Initialized, Role: FOLLOWER
I20260130 02:58:03.482013 19807 consensus_queue.cc:260] T 7d76046ee6c8420fa4964decbe8209c1 P 371a36d219164974abdf848841528152 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:03.483892 19807 ts_tablet_manager.cc:1434] T 7d76046ee6c8420fa4964decbe8209c1 P 371a36d219164974abdf848841528152: Time spent starting tablet: real 0.011s user 0.006s sys 0.000s
I20260130 02:58:03.494722 19807 tablet_bootstrap.cc:492] T e47b22aafe5247c4b62c4193c24f427c P 371a36d219164974abdf848841528152: Bootstrap starting.
I20260130 02:58:03.494916 19505 tablet_service.cc:1505] Processing CreateTablet for tablet 8bd3e1c06b89426da57e2b259e7983af (DEFAULT_TABLE table=default.Table_1d9d1f3b2b6048f8949fdba26f5fd3d7 [id=1b409bcef4ca4d248418597c1aa69fdb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:03.495842 19505 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8bd3e1c06b89426da57e2b259e7983af. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:03.457782 19625 raft_consensus.cc:740] T 5ca9919f80954e74a4d91c94b74de0e4 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d870164263bb44b6b780a557daea3656, State: Initialized, Role: FOLLOWER
I20260130 02:58:03.499742 19625 consensus_queue.cc:260] T 5ca9919f80954e74a4d91c94b74de0e4 P d870164263bb44b6b780a557daea3656 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:03.510246 19625 ts_tablet_manager.cc:1434] T 5ca9919f80954e74a4d91c94b74de0e4 P d870164263bb44b6b780a557daea3656: Time spent starting tablet: real 0.067s user 0.005s sys 0.001s
I20260130 02:58:03.510443 19947 ts_tablet_manager.cc:1929] T 96584ca174b140f7ad1a2ca698baca71 P d870164263bb44b6b780a557daea3656: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.6
I20260130 02:58:03.510852 19947 log.cc:1199] T 96584ca174b140f7ad1a2ca698baca71 P d870164263bb44b6b780a557daea3656: Deleting WAL directory at /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-2/wal/wals/96584ca174b140f7ad1a2ca698baca71
I20260130 02:58:03.511142 19625 tablet_bootstrap.cc:492] T 7b80b22bd44c4233837c951aa97e6996 P d870164263bb44b6b780a557daea3656: Bootstrap starting.
I20260130 02:58:03.511572 19947 ts_tablet_manager.cc:1950] T 96584ca174b140f7ad1a2ca698baca71 P d870164263bb44b6b780a557daea3656: Deleting consensus metadata
I20260130 02:58:03.511972 19198 tablet_service.cc:1505] Processing CreateTablet for tablet 8bd3e1c06b89426da57e2b259e7983af (DEFAULT_TABLE table=default.Table_1d9d1f3b2b6048f8949fdba26f5fd3d7 [id=1b409bcef4ca4d248418597c1aa69fdb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:03.513217 19198 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8bd3e1c06b89426da57e2b259e7983af. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:03.516222 19204 tablet_service.cc:1505] Processing CreateTablet for tablet f54addb1a4304ce69c711ae0cc3b452c (DEFAULT_TABLE table=default.Table_2d345d0f0957486c86c2cf2e34b00941 [id=4c3e0d3ecd5642a3b7fc7d20e2521418]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:03.516979 19499 tablet_service.cc:1505] Processing CreateTablet for tablet f54addb1a4304ce69c711ae0cc3b452c (DEFAULT_TABLE table=default.Table_2d345d0f0957486c86c2cf2e34b00941 [id=4c3e0d3ecd5642a3b7fc7d20e2521418]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:03.517994 19499 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f54addb1a4304ce69c711ae0cc3b452c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:03.520006 19204 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f54addb1a4304ce69c711ae0cc3b452c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:03.539518 19498 tablet_service.cc:1505] Processing CreateTablet for tablet b2ac91da01954c2390b6b7dcc06c394f (DEFAULT_TABLE table=default.Table_f2f790e8e46840ecb199c34fae899ada [id=60b8e98e3f864716a6c1aa3816612211]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:03.540710 19498 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b2ac91da01954c2390b6b7dcc06c394f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:03.455298 19840 ts_tablet_manager.cc:1403] T 534dc31a9b7e492c82f3672c75742e6c P 78c2c46c7520492db35d51c244fb4f74: Time spent bootstrapping tablet: real 0.159s user 0.008s sys 0.001s
I20260130 02:58:03.468703 19378 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b3f3177306914e3c8b253db54ee30fb6" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74" is_pre_election: true
I20260130 02:58:03.474052 19377 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a743a82540014c85aeab0e9d0994b0dc" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74" is_pre_election: true
I20260130 02:58:03.475015 19758 ts_tablet_manager.cc:1929] T d701152960b14396a6ac1bd10480d2b6 P 78c2c46c7520492db35d51c244fb4f74: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260130 02:58:03.479414 20114 tablet.cc:1722] T 1bbc55bbf2254d8bb6b721d7e8f69d85 P 78c2c46c7520492db35d51c244fb4f74: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260130 02:58:03.543303 19758 log.cc:1199] T d701152960b14396a6ac1bd10480d2b6 P 78c2c46c7520492db35d51c244fb4f74: Deleting WAL directory at /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-1/wal/wals/d701152960b14396a6ac1bd10480d2b6
I20260130 02:58:03.493417 19360 tablet_service.cc:1505] Processing CreateTablet for tablet 8bd3e1c06b89426da57e2b259e7983af (DEFAULT_TABLE table=default.Table_1d9d1f3b2b6048f8949fdba26f5fd3d7 [id=1b409bcef4ca4d248418597c1aa69fdb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:03.544013 19758 ts_tablet_manager.cc:1950] T d701152960b14396a6ac1bd10480d2b6 P 78c2c46c7520492db35d51c244fb4f74: Deleting consensus metadata
I20260130 02:58:03.544549 19360 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8bd3e1c06b89426da57e2b259e7983af. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:03.546017 19840 raft_consensus.cc:359] T 534dc31a9b7e492c82f3672c75742e6c P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:03.546885 18860 catalog_manager.cc:4985] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767): tablet d701152960b14396a6ac1bd10480d2b6 (table default.Table_e3b63d7d5d874900aac850654d98d62e [id=bc8cc55916fa4a41b78019f59e6fbe39]) successfully deleted
W20260130 02:58:03.547514 18860 catalog_manager.cc:4952] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767): delete failed for tablet a46702bef0e6442d85e4ca49a5f7cfbb because the tablet was not found. No further retry: Not found: Tablet not found: a46702bef0e6442d85e4ca49a5f7cfbb
W20260130 02:58:03.548102 18860 catalog_manager.cc:4952] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767): delete failed for tablet 073b5941850c4140b1751956dd7cf5ef because the tablet was not found. No further retry: Not found: Tablet not found: 073b5941850c4140b1751956dd7cf5ef
I20260130 02:58:03.547608 19221 raft_consensus.cc:3060] T a743a82540014c85aeab0e9d0994b0dc P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Advancing to term 1
W20260130 02:58:03.548620 18860 catalog_manager.cc:4952] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767): delete failed for tablet 96584ca174b140f7ad1a2ca698baca71 because the tablet was not found. No further retry: Not found: Tablet not found: 96584ca174b140f7ad1a2ca698baca71
W20260130 02:58:03.549130 18860 catalog_manager.cc:4952] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767): delete failed for tablet 7a902d0d571a42bb82ebffd48f7eca3f because the tablet was not found. No further retry: Not found: Tablet not found: 7a902d0d571a42bb82ebffd48f7eca3f
I20260130 02:58:03.508358 20116 tablet.cc:1722] T f2d01bfdc9c4494386bd02bed655fe63 P 78c2c46c7520492db35d51c244fb4f74: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20260130 02:58:03.515017 19357 tablet_service.cc:1505] Processing CreateTablet for tablet f54addb1a4304ce69c711ae0cc3b452c (DEFAULT_TABLE table=default.Table_2d345d0f0957486c86c2cf2e34b00941 [id=4c3e0d3ecd5642a3b7fc7d20e2521418]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:03.555192 19462 leader_election.cc:304] T 7d76046ee6c8420fa4964decbe8209c1 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 78c2c46c7520492db35d51c244fb4f74, d870164263bb44b6b780a557daea3656; no voters: 371a36d219164974abdf848841528152
I20260130 02:58:03.555425 19357 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f54addb1a4304ce69c711ae0cc3b452c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:03.556475 19971 raft_consensus.cc:2804] T 7d76046ee6c8420fa4964decbe8209c1 P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Leader election won for term 1
I20260130 02:58:03.557121 19971 raft_consensus.cc:697] T 7d76046ee6c8420fa4964decbe8209c1 P d870164263bb44b6b780a557daea3656 [term 1 LEADER]: Becoming Leader. State: Replica: d870164263bb44b6b780a557daea3656, State: Running, Role: LEADER
I20260130 02:58:03.536186 19354 tablet_service.cc:1505] Processing CreateTablet for tablet b2ac91da01954c2390b6b7dcc06c394f (DEFAULT_TABLE table=default.Table_f2f790e8e46840ecb199c34fae899ada [id=60b8e98e3f864716a6c1aa3816612211]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:03.559226 19354 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b2ac91da01954c2390b6b7dcc06c394f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:03.559728 19971 consensus_queue.cc:237] T 7d76046ee6c8420fa4964decbe8209c1 P d870164263bb44b6b780a557daea3656 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
W20260130 02:58:03.568253 19159 outbound_call.cc:321] RPC callback for RPC call kudu.master.MasterService.TSHeartbeat -> {remote=127.18.100.125:39711, user_credentials={real_user=slave}} blocked reactor thread for 67966.8us
I20260130 02:58:03.571218 19159 leader_election.cc:304] T b3f3177306914e3c8b253db54ee30fb6 P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 371a36d219164974abdf848841528152; no voters: 78c2c46c7520492db35d51c244fb4f74, d870164263bb44b6b780a557daea3656
I20260130 02:58:03.572356 19159 leader_election.cc:304] T a743a82540014c85aeab0e9d0994b0dc P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 371a36d219164974abdf848841528152; no voters: 78c2c46c7520492db35d51c244fb4f74, d870164263bb44b6b780a557daea3656
I20260130 02:58:03.572527 20115 raft_consensus.cc:3060] T b3f3177306914e3c8b253db54ee30fb6 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:03.573654 19861 raft_consensus.cc:2749] T a743a82540014c85aeab0e9d0994b0dc P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260130 02:58:03.576471 20083 consensus_queue.cc:1048] T a743a82540014c85aeab0e9d0994b0dc P 78c2c46c7520492db35d51c244fb4f74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260130 02:58:03.576697 19201 tablet_service.cc:1505] Processing CreateTablet for tablet b2ac91da01954c2390b6b7dcc06c394f (DEFAULT_TABLE table=default.Table_f2f790e8e46840ecb199c34fae899ada [id=60b8e98e3f864716a6c1aa3816612211]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:03.577713 19201 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b2ac91da01954c2390b6b7dcc06c394f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:03.592746 18874 catalog_manager.cc:5941] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4: Sending DeleteTablet for 3 replicas of tablet 6573198830ad45628e71952a7d9c00a0
W20260130 02:58:03.592927 19462 consensus_peers.cc:597] T 5b2976108584478a98fd90fe6594af66 P d870164263bb44b6b780a557daea3656 -> Peer 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767): Couldn't send request to peer 78c2c46c7520492db35d51c244fb4f74. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5b2976108584478a98fd90fe6594af66. This is attempt 1: this message will repeat every 5th retry.
W20260130 02:58:03.594373 19313 leader_election.cc:343] T 2ca0df71836145748cada4e4559a5656 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:03.594758 19313 leader_election.cc:304] T 2ca0df71836145748cada4e4559a5656 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78c2c46c7520492db35d51c244fb4f74; no voters: 371a36d219164974abdf848841528152, d870164263bb44b6b780a557daea3656
I20260130 02:58:03.595361 20083 raft_consensus.cc:2749] T 2ca0df71836145748cada4e4559a5656 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260130 02:58:03.596021 19500 tablet_service.cc:1552] Processing DeleteTablet for tablet 6573198830ad45628e71952a7d9c00a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:58:03 UTC) from {username='slave'} at 127.0.0.1:59068
I20260130 02:58:03.596652 19352 tablet_service.cc:1552] Processing DeleteTablet for tablet 6573198830ad45628e71952a7d9c00a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:58:03 UTC) from {username='slave'} at 127.0.0.1:60110
I20260130 02:58:03.597476 19199 tablet_service.cc:1552] Processing DeleteTablet for tablet 6573198830ad45628e71952a7d9c00a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:58:03 UTC) from {username='slave'} at 127.0.0.1:42870
W20260130 02:58:03.598603 18860 catalog_manager.cc:4965] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767): delete failed for tablet 6573198830ad45628e71952a7d9c00a0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6573198830ad45628e71952a7d9c00a0 already in progress: creating tablet
I20260130 02:58:03.602808 19840 raft_consensus.cc:385] T 534dc31a9b7e492c82f3672c75742e6c P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:03.603111 19840 raft_consensus.cc:740] T 534dc31a9b7e492c82f3672c75742e6c P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78c2c46c7520492db35d51c244fb4f74, State: Initialized, Role: FOLLOWER
I20260130 02:58:03.603291 18866 catalog_manager.cc:5941] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4: Sending DeleteTablet for 3 replicas of tablet 730bd1d7d15b4cbe945b3ef03b2455f7
I20260130 02:58:03.603763 19840 consensus_queue.cc:260] T 534dc31a9b7e492c82f3672c75742e6c P 78c2c46c7520492db35d51c244fb4f74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:03.604463 18866 catalog_manager.cc:6724] ReplaceTablet: tablet 730bd1d7d15b4cbe945b3ef03b2455f7 deleted and replaced by tablet 3b02480d9c6f4328af83f7940716d698
I20260130 02:58:03.606632 19947 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59068 (request call id 94) took 1912 ms. Trace:
I20260130 02:58:03.607476 19199 tablet_service.cc:1552] Processing DeleteTablet for tablet 730bd1d7d15b4cbe945b3ef03b2455f7 with delete_type TABLET_DATA_DELETED (replaced by tablet 3b02480d9c6f4328af83f7940716d698) from {username='slave'} at 127.0.0.1:42870
I20260130 02:58:03.607131 19947 rpcz_store.cc:276] 0130 02:58:01.601193 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:58:01.602858 (+ 1665us) service_pool.cc:225] Handling call
0130 02:58:03.382518 (+1779660us) ts_tablet_manager.cc:1151] Deleting tablet 96584ca174b140f7ad1a2ca698baca71
0130 02:58:03.382587 (+ 69us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0130 02:58:03.437945 (+ 55358us) tablet_metadata.cc:641] Metadata flushed
0130 02:58:03.510208 (+ 72263us) tablet_metadata.cc:641] Metadata flushed
0130 02:58:03.513896 (+ 3688us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1678521}
I20260130 02:58:03.610160 18866 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50284:
name: "default.Table_caccb5141e014c558ebbda10e2415e16"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260130 02:58:03.610865 19754 ts_tablet_manager.cc:1929] T d701152960b14396a6ac1bd10480d2b6 P 371a36d219164974abdf848841528152: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260130 02:58:03.611227 19754 log.cc:1199] T d701152960b14396a6ac1bd10480d2b6 P 371a36d219164974abdf848841528152: Deleting WAL directory at /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-0/wal/wals/d701152960b14396a6ac1bd10480d2b6
W20260130 02:58:03.611644 18866 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_caccb5141e014c558ebbda10e2415e16 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260130 02:58:03.614094 20009 consensus_queue.cc:1048] T b3f3177306914e3c8b253db54ee30fb6 P 78c2c46c7520492db35d51c244fb4f74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260130 02:58:03.615203 18874 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50284:
name: "default.Table_8f632ff84d624b72a10cb90fb9e78636"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260130 02:58:03.616628 18874 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8f632ff84d624b72a10cb90fb9e78636 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260130 02:58:03.625195 18871 catalog_manager.cc:5941] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4: Sending DeleteTablet for 3 replicas of tablet b7b596f944be4daf953b82b65d5cec2a
I20260130 02:58:03.626400 18871 catalog_manager.cc:6724] ReplaceTablet: tablet b7b596f944be4daf953b82b65d5cec2a deleted and replaced by tablet 3389db58493448aea3cf00aca411047c
I20260130 02:58:03.626749 18871 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:50284 (request call id 148) took 1056 ms. Trace:
I20260130 02:58:03.627027 19947 tablet_replica.cc:333] T b078934be4bb4fd1915c691eeea9d00f P d870164263bb44b6b780a557daea3656: stopping tablet replica
I20260130 02:58:03.627812 19947 raft_consensus.cc:2243] T b078934be4bb4fd1915c691eeea9d00f P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260130 02:58:03.628288 19947 raft_consensus.cc:2272] T b078934be4bb4fd1915c691eeea9d00f P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260130 02:58:03.627014 18871 rpcz_store.cc:276] 0130 02:58:02.569880 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:58:03.050437 (+480557us) service_pool.cc:225] Handling call
0130 02:58:03.626686 (+576249us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0130 02:58:03.560556 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0130 02:58:03.560948 (+ 392us) write_op.cc:432] Acquiring schema lock in shared mode
0130 02:58:03.560971 (+ 23us) write_op.cc:435] Acquired schema lock
0130 02:58:03.560983 (+ 12us) tablet.cc:660] Decoding operations
0130 02:58:03.561269 (+ 286us) write_op.cc:620] Acquiring the partition lock for write op
0130 02:58:03.561304 (+ 35us) write_op.cc:641] Partition lock acquired for write op
0130 02:58:03.561325 (+ 21us) tablet.cc:683] Acquiring locks for 2 operations
0130 02:58:03.561512 (+ 187us) tablet.cc:699] Row locks acquired
0130 02:58:03.561525 (+ 13us) write_op.cc:260] PREPARE: finished
0130 02:58:03.561631 (+ 106us) write_op.cc:270] Start()
0130 02:58:03.561711 (+ 80us) write_op.cc:276] Timestamp: P: 1769741883561600 usec, L: 0
0130 02:58:03.561727 (+ 16us) op_driver.cc:348] REPLICATION: starting
0130 02:58:03.562187 (+ 460us) log.cc:844] Serialized 628 byte log entry
0130 02:58:03.620375 (+ 58188us) op_driver.cc:464] REPLICATION: finished
0130 02:58:03.622239 (+ 1864us) write_op.cc:301] APPLY: starting
0130 02:58:03.622300 (+ 61us) tablet.cc:1365] starting BulkCheckPresence
0130 02:58:03.622402 (+ 102us) tablet.cc:1368] finished BulkCheckPresence
0130 02:58:03.622492 (+ 90us) tablet.cc:1370] starting ApplyRowOperation cycle
0130 02:58:03.623963 (+ 1471us) tablet.cc:1381] finished ApplyRowOperation cycle
0130 02:58:03.624163 (+ 200us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0130 02:58:03.624212 (+ 49us) write_op.cc:312] APPLY: finished
0130 02:58:03.624392 (+ 180us) log.cc:844] Serialized 29 byte log entry
0130 02:58:03.624531 (+ 139us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:58:03.624600 (+ 69us) write_op.cc:454] Released schema lock
0130 02:58:03.624632 (+ 32us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":1750,"num_ops":2,"prepare.queue_time_us":72687,"replication_time_us":58579}]]}
I20260130 02:58:03.631716 19199 tablet_service.cc:1552] Processing DeleteTablet for tablet b7b596f944be4daf953b82b65d5cec2a with delete_type TABLET_DATA_DELETED (replaced by tablet 3389db58493448aea3cf00aca411047c) from {username='slave'} at 127.0.0.1:42870
I20260130 02:58:03.636575 18868 catalog_manager.cc:5746] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571) does not have the latest schema for tablet f2d01bfdc9c4494386bd02bed655fe63 (table default.Table_42cd032296a0408797e7af3a27a103dc [id=d128767e303145a885eec3c5f714a27f]). Expected version 2 got 1
I20260130 02:58:03.609920 19807 tablet_bootstrap.cc:654] T e47b22aafe5247c4b62c4193c24f427c P 371a36d219164974abdf848841528152: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:03.644434 18871 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50284:
name: "default.Table_5c724fbcf445490089992ba7fe40dc3d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260130 02:58:03.645427 19754 ts_tablet_manager.cc:1950] T d701152960b14396a6ac1bd10480d2b6 P 371a36d219164974abdf848841528152: Deleting consensus metadata
W20260130 02:58:03.646250 18871 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5c724fbcf445490089992ba7fe40dc3d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260130 02:58:03.648443 18861 catalog_manager.cc:4985] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021): tablet d701152960b14396a6ac1bd10480d2b6 (table default.Table_e3b63d7d5d874900aac850654d98d62e [id=bc8cc55916fa4a41b78019f59e6fbe39]) successfully deleted
W20260130 02:58:03.649115 18861 catalog_manager.cc:4965] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021): delete failed for tablet 481bd4cca6ee4c178873bf12392e3441 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 481bd4cca6ee4c178873bf12392e3441 already in progress: creating tablet
W20260130 02:58:03.649626 18861 catalog_manager.cc:4952] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021): delete failed for tablet 878f9346d3044d8d8b15c61f86869212 because the tablet was not found. No further retry: Not found: Tablet not found: 878f9346d3044d8d8b15c61f86869212
I20260130 02:58:03.650962 20089 raft_consensus.cc:493] T 7b80b22bd44c4233837c951aa97e6996 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:03.652452 18860 catalog_manager.cc:4985] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): tablet 96584ca174b140f7ad1a2ca698baca71 (table default.Table_4f2d6e4d9c8e465087d65efc9c2de442 [id=1515db389f284e2f9ed1a3cd0061d0f1]) successfully deleted
I20260130 02:58:03.652869 19352 tablet_service.cc:1552] Processing DeleteTablet for tablet 730bd1d7d15b4cbe945b3ef03b2455f7 with delete_type TABLET_DATA_DELETED (replaced by tablet 3b02480d9c6f4328af83f7940716d698) from {username='slave'} at 127.0.0.1:60110
I20260130 02:58:03.653306 19358 tablet_service.cc:1552] Processing DeleteTablet for tablet b7b596f944be4daf953b82b65d5cec2a with delete_type TABLET_DATA_DELETED (replaced by tablet 3389db58493448aea3cf00aca411047c) from {username='slave'} at 127.0.0.1:60110
I20260130 02:58:03.654191 19500 tablet_service.cc:1552] Processing DeleteTablet for tablet b7b596f944be4daf953b82b65d5cec2a with delete_type TABLET_DATA_DELETED (replaced by tablet 3389db58493448aea3cf00aca411047c) from {username='slave'} at 127.0.0.1:59068
I20260130 02:58:03.655491 19758 tablet_replica.cc:333] T b7b596f944be4daf953b82b65d5cec2a P 78c2c46c7520492db35d51c244fb4f74: stopping tablet replica
I20260130 02:58:03.656061 19505 tablet_service.cc:1552] Processing DeleteTablet for tablet 730bd1d7d15b4cbe945b3ef03b2455f7 with delete_type TABLET_DATA_DELETED (replaced by tablet 3b02480d9c6f4328af83f7940716d698) from {username='slave'} at 127.0.0.1:59068
I20260130 02:58:03.657733 20089 raft_consensus.cc:515] T 7b80b22bd44c4233837c951aa97e6996 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:03.659633 20089 leader_election.cc:290] T 7b80b22bd44c4233837c951aa97e6996 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), d870164263bb44b6b780a557daea3656 (127.18.100.67:39571)
I20260130 02:58:03.661664 19223 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7b80b22bd44c4233837c951aa97e6996" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152" is_pre_election: true
W20260130 02:58:03.662755 19314 leader_election.cc:343] T 7b80b22bd44c4233837c951aa97e6996 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:03.664966 19758 raft_consensus.cc:2243] T b7b596f944be4daf953b82b65d5cec2a P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260130 02:58:03.665494 19758 raft_consensus.cc:2272] T b7b596f944be4daf953b82b65d5cec2a P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260130 02:58:03.667297 19758 ts_tablet_manager.cc:1916] T b7b596f944be4daf953b82b65d5cec2a P 78c2c46c7520492db35d51c244fb4f74: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260130 02:58:03.670264 19199 tablet_service.cc:1505] Processing CreateTablet for tablet ee7c8ff92ca9465fa18be6eae63d78ca (DEFAULT_TABLE table=default.Table_112ffec66bd143398baeb18ddf231064 [id=77979d498aa040828de415129cb15a06]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:03.667265 19204 tablet_service.cc:1505] Processing CreateTablet for tablet 3593f8b89da24ff9b6cf549d5ac58bc6 (DEFAULT_TABLE table=default.Table_2c737f14bf854cfca58d4a9f83fb2e9e [id=fa4e8955bebb40b6adf6ed4ce0e43bfb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:03.671448 19199 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ee7c8ff92ca9465fa18be6eae63d78ca. 1 dirs total, 0 dirs full, 0 dirs failed
W20260130 02:58:03.674129 18860 catalog_manager.cc:4965] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767): delete failed for tablet 730bd1d7d15b4cbe945b3ef03b2455f7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 730bd1d7d15b4cbe945b3ef03b2455f7 already in progress: creating tablet
I20260130 02:58:03.675374 19505 tablet_service.cc:1505] Processing CreateTablet for tablet 3593f8b89da24ff9b6cf549d5ac58bc6 (DEFAULT_TABLE table=default.Table_2c737f14bf854cfca58d4a9f83fb2e9e [id=fa4e8955bebb40b6adf6ed4ce0e43bfb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:03.676585 19505 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3593f8b89da24ff9b6cf549d5ac58bc6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:03.683403 19360 tablet_service.cc:1505] Processing CreateTablet for tablet 3593f8b89da24ff9b6cf549d5ac58bc6 (DEFAULT_TABLE table=default.Table_2c737f14bf854cfca58d4a9f83fb2e9e [id=fa4e8955bebb40b6adf6ed4ce0e43bfb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:03.684468 19360 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3593f8b89da24ff9b6cf549d5ac58bc6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:03.685389 19625 tablet_bootstrap.cc:654] T 7b80b22bd44c4233837c951aa97e6996 P d870164263bb44b6b780a557daea3656: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:03.687723 19947 ts_tablet_manager.cc:1916] T b078934be4bb4fd1915c691eeea9d00f P d870164263bb44b6b780a557daea3656: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260130 02:58:03.695866 19204 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3593f8b89da24ff9b6cf549d5ac58bc6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:03.699460 19840 ts_tablet_manager.cc:1434] T 534dc31a9b7e492c82f3672c75742e6c P 78c2c46c7520492db35d51c244fb4f74: Time spent starting tablet: real 0.156s user 0.006s sys 0.000s
I20260130 02:58:03.708956 19807 tablet_bootstrap.cc:492] T e47b22aafe5247c4b62c4193c24f427c P 371a36d219164974abdf848841528152: No bootstrap required, opened a new log
I20260130 02:58:03.709393 19807 ts_tablet_manager.cc:1403] T e47b22aafe5247c4b62c4193c24f427c P 371a36d219164974abdf848841528152: Time spent bootstrapping tablet: real 0.215s user 0.017s sys 0.017s
I20260130 02:58:03.711895 18875 catalog_manager.cc:5746] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767) does not have the latest schema for tablet f2d01bfdc9c4494386bd02bed655fe63 (table default.Table_42cd032296a0408797e7af3a27a103dc [id=d128767e303145a885eec3c5f714a27f]). Expected version 2 got 1
I20260130 02:58:03.714555 18875 catalog_manager.cc:5654] T d3ba2df0b15c44f882070570f91b254d P 78c2c46c7520492db35d51c244fb4f74 reported cstate change: term changed from 0 to 1, leader changed from <none> to 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66). New cstate: current_term: 1 leader_uuid: "78c2c46c7520492db35d51c244fb4f74" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } health_report { overall_health: HEALTHY } } }
I20260130 02:58:03.720450 19807 raft_consensus.cc:359] T e47b22aafe5247c4b62c4193c24f427c P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:03.721082 19807 raft_consensus.cc:385] T e47b22aafe5247c4b62c4193c24f427c P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:03.721325 19807 raft_consensus.cc:740] T e47b22aafe5247c4b62c4193c24f427c P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 371a36d219164974abdf848841528152, State: Initialized, Role: FOLLOWER
I20260130 02:58:03.721910 19807 consensus_queue.cc:260] T e47b22aafe5247c4b62c4193c24f427c P 371a36d219164974abdf848841528152 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:03.728226 19527 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7b80b22bd44c4233837c951aa97e6996" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
I20260130 02:58:03.730463 19807 ts_tablet_manager.cc:1434] T e47b22aafe5247c4b62c4193c24f427c P 371a36d219164974abdf848841528152: Time spent starting tablet: real 0.021s user 0.007s sys 0.000s
I20260130 02:58:03.731793 19807 tablet_bootstrap.cc:492] T 481bd4cca6ee4c178873bf12392e3441 P 371a36d219164974abdf848841528152: Bootstrap starting.
I20260130 02:58:03.735980 18874 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50284:
name: "default.Table_84fbe659a98c4a2e9f805bb1efc7b4de"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260130 02:58:03.737771 18874 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_84fbe659a98c4a2e9f805bb1efc7b4de in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260130 02:58:03.738612 19807 tablet_bootstrap.cc:654] T 481bd4cca6ee4c178873bf12392e3441 P 371a36d219164974abdf848841528152: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:03.738608 19840 tablet_bootstrap.cc:492] T 730bd1d7d15b4cbe945b3ef03b2455f7 P 78c2c46c7520492db35d51c244fb4f74: Bootstrap starting.
I20260130 02:58:03.742812 20115 raft_consensus.cc:2749] T b3f3177306914e3c8b253db54ee30fb6 P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260130 02:58:03.745682 20089 consensus_queue.cc:1048] T b3f3177306914e3c8b253db54ee30fb6 P 78c2c46c7520492db35d51c244fb4f74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260130 02:58:03.753412 19807 tablet_bootstrap.cc:492] T 481bd4cca6ee4c178873bf12392e3441 P 371a36d219164974abdf848841528152: No bootstrap required, opened a new log
I20260130 02:58:03.753827 19807 ts_tablet_manager.cc:1403] T 481bd4cca6ee4c178873bf12392e3441 P 371a36d219164974abdf848841528152: Time spent bootstrapping tablet: real 0.022s user 0.005s sys 0.003s
I20260130 02:58:03.755955 19807 raft_consensus.cc:359] T 481bd4cca6ee4c178873bf12392e3441 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:03.756547 19807 raft_consensus.cc:385] T 481bd4cca6ee4c178873bf12392e3441 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:03.756798 19807 raft_consensus.cc:740] T 481bd4cca6ee4c178873bf12392e3441 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 371a36d219164974abdf848841528152, State: Initialized, Role: FOLLOWER
I20260130 02:58:03.760291 19201 tablet_service.cc:1505] Processing CreateTablet for tablet b89ed298e3dc49d789d166c613ad7fb8 (DEFAULT_TABLE table=default.Table_f8856f06ce97440bae4abdf34164716d [id=a3303ada20a94f99b2f37a02eec361ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:03.761271 19201 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b89ed298e3dc49d789d166c613ad7fb8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:03.758843 19807 consensus_queue.cc:260] T 481bd4cca6ee4c178873bf12392e3441 P 371a36d219164974abdf848841528152 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:03.763233 19357 tablet_service.cc:1505] Processing CreateTablet for tablet b89ed298e3dc49d789d166c613ad7fb8 (DEFAULT_TABLE table=default.Table_f8856f06ce97440bae4abdf34164716d [id=a3303ada20a94f99b2f37a02eec361ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:03.764226 19357 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b89ed298e3dc49d789d166c613ad7fb8. 1 dirs total, 0 dirs full, 0 dirs failed
W20260130 02:58:03.770243 18861 catalog_manager.cc:4952] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021): delete failed for tablet b078934be4bb4fd1915c691eeea9d00f because the tablet was not found. No further retry: Not found: Tablet not found: b078934be4bb4fd1915c691eeea9d00f
W20260130 02:58:03.771057 18861 catalog_manager.cc:4952] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021): delete failed for tablet 26560f746d244e58a182eb6bb24c8720 because the tablet was not found. No further retry: Not found: Tablet not found: 26560f746d244e58a182eb6bb24c8720
I20260130 02:58:03.771297 19754 tablet_replica.cc:333] T 5b2976108584478a98fd90fe6594af66 P 371a36d219164974abdf848841528152: stopping tablet replica
I20260130 02:58:03.771291 19861 raft_consensus.cc:493] T e47b22aafe5247c4b62c4193c24f427c P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:03.771844 19861 raft_consensus.cc:515] T e47b22aafe5247c4b62c4193c24f427c P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:03.772162 19754 raft_consensus.cc:2243] T 5b2976108584478a98fd90fe6594af66 P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260130 02:58:03.775555 19377 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e47b22aafe5247c4b62c4193c24f427c" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74" is_pre_election: true
I20260130 02:58:03.776186 19377 raft_consensus.cc:2468] T e47b22aafe5247c4b62c4193c24f427c P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 371a36d219164974abdf848841528152 in term 0.
I20260130 02:58:03.781296 19524 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e47b22aafe5247c4b62c4193c24f427c" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
I20260130 02:58:03.783257 19354 tablet_service.cc:1505] Processing CreateTablet for tablet ee7c8ff92ca9465fa18be6eae63d78ca (DEFAULT_TABLE table=default.Table_112ffec66bd143398baeb18ddf231064 [id=77979d498aa040828de415129cb15a06]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:03.784379 19354 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ee7c8ff92ca9465fa18be6eae63d78ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:03.787436 19499 tablet_service.cc:1505] Processing CreateTablet for tablet ee7c8ff92ca9465fa18be6eae63d78ca (DEFAULT_TABLE table=default.Table_112ffec66bd143398baeb18ddf231064 [id=77979d498aa040828de415129cb15a06]), partition=RANGE (key) PARTITION UNBOUNDED
W20260130 02:58:03.792881 19313 leader_election.cc:343] T 7b80b22bd44c4233837c951aa97e6996 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:03.793283 19313 leader_election.cc:304] T 7b80b22bd44c4233837c951aa97e6996 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78c2c46c7520492db35d51c244fb4f74; no voters: 371a36d219164974abdf848841528152, d870164263bb44b6b780a557daea3656
I20260130 02:58:03.773847 19861 leader_election.cc:290] T e47b22aafe5247c4b62c4193c24f427c P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d870164263bb44b6b780a557daea3656 (127.18.100.67:39571), 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767)
I20260130 02:58:03.777451 19159 leader_election.cc:304] T e47b22aafe5247c4b62c4193c24f427c P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 371a36d219164974abdf848841528152, 78c2c46c7520492db35d51c244fb4f74; no voters:
I20260130 02:58:03.778184 19807 ts_tablet_manager.cc:1434] T 481bd4cca6ee4c178873bf12392e3441 P 371a36d219164974abdf848841528152: Time spent starting tablet: real 0.024s user 0.005s sys 0.001s
W20260130 02:58:03.794507 19159 leader_election.cc:343] T e47b22aafe5247c4b62c4193c24f427c P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:03.795498 20115 raft_consensus.cc:2804] T e47b22aafe5247c4b62c4193c24f427c P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260130 02:58:03.795848 20115 raft_consensus.cc:493] T e47b22aafe5247c4b62c4193c24f427c P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260130 02:58:03.796136 20115 raft_consensus.cc:3060] T e47b22aafe5247c4b62c4193c24f427c P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:03.799386 19754 raft_consensus.cc:2272] T 5b2976108584478a98fd90fe6594af66 P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260130 02:58:03.805301 19754 ts_tablet_manager.cc:1916] T 5b2976108584478a98fd90fe6594af66 P 371a36d219164974abdf848841528152: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260130 02:58:03.718073 18875 catalog_manager.cc:5654] T a743a82540014c85aeab0e9d0994b0dc P 78c2c46c7520492db35d51c244fb4f74 reported cstate change: term changed from 0 to 1, leader changed from <none> to 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66). New cstate: current_term: 1 leader_uuid: "78c2c46c7520492db35d51c244fb4f74" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } health_report { overall_health: UNKNOWN } } }
I20260130 02:58:03.815346 19625 tablet_bootstrap.cc:492] T 7b80b22bd44c4233837c951aa97e6996 P d870164263bb44b6b780a557daea3656: No bootstrap required, opened a new log
I20260130 02:58:03.815281 20120 raft_consensus.cc:2749] T 7b80b22bd44c4233837c951aa97e6996 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260130 02:58:03.815768 19625 ts_tablet_manager.cc:1403] T 7b80b22bd44c4233837c951aa97e6996 P d870164263bb44b6b780a557daea3656: Time spent bootstrapping tablet: real 0.305s user 0.005s sys 0.005s
I20260130 02:58:03.818500 19625 raft_consensus.cc:359] T 7b80b22bd44c4233837c951aa97e6996 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:03.819293 19625 raft_consensus.cc:385] T 7b80b22bd44c4233837c951aa97e6996 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:03.819521 19625 raft_consensus.cc:740] T 7b80b22bd44c4233837c951aa97e6996 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d870164263bb44b6b780a557daea3656, State: Initialized, Role: FOLLOWER
I20260130 02:58:03.820842 18875 catalog_manager.cc:5746] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767) does not have the latest schema for tablet 1bbc55bbf2254d8bb6b721d7e8f69d85 (table default.Table_526677c8ef4144bd9de7a54a5e78e2be [id=b16908b75587459597a978c71756451c]). Expected version 1 got 0
W20260130 02:58:03.825865 19463 consensus_peers.cc:597] T 5b2976108584478a98fd90fe6594af66 P d870164263bb44b6b780a557daea3656 -> Peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Couldn't send request to peer 371a36d219164974abdf848841528152. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20260130 02:58:03.826150 19625 consensus_queue.cc:260] T 7b80b22bd44c4233837c951aa97e6996 P d870164263bb44b6b780a557daea3656 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:03.831431 19198 tablet_service.cc:1505] Processing CreateTablet for tablet 7d96e93d95f14af79cfb268626559b1e (DEFAULT_TABLE table=default.Table_6eb0a92b1926487cbf0b978d26d6a7b1 [id=de7a420d40864f55b156e149308ba12f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:03.832341 19625 ts_tablet_manager.cc:1434] T 7b80b22bd44c4233837c951aa97e6996 P d870164263bb44b6b780a557daea3656: Time spent starting tablet: real 0.016s user 0.006s sys 0.000s
I20260130 02:58:03.833185 19625 tablet_bootstrap.cc:492] T 2ca0df71836145748cada4e4559a5656 P d870164263bb44b6b780a557daea3656: Bootstrap starting.
I20260130 02:58:03.830771 19498 tablet_service.cc:1505] Processing CreateTablet for tablet b89ed298e3dc49d789d166c613ad7fb8 (DEFAULT_TABLE table=default.Table_f8856f06ce97440bae4abdf34164716d [id=a3303ada20a94f99b2f37a02eec361ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:03.837500 20115 raft_consensus.cc:515] T e47b22aafe5247c4b62c4193c24f427c P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
W20260130 02:58:03.842800 18860 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.18.100.124:34629: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20260130 02:58:03.843565 20115 leader_election.cc:290] T e47b22aafe5247c4b62c4193c24f427c P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 election: Requested vote from peers d870164263bb44b6b780a557daea3656 (127.18.100.67:39571), 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767)
I20260130 02:58:03.845005 19377 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e47b22aafe5247c4b62c4193c24f427c" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74"
I20260130 02:58:03.845679 19377 raft_consensus.cc:3060] T e47b22aafe5247c4b62c4193c24f427c P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:03.849306 20009 consensus_queue.cc:1048] T d3ba2df0b15c44f882070570f91b254d P 78c2c46c7520492db35d51c244fb4f74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260130 02:58:03.854897 19358 tablet_service.cc:1505] Processing CreateTablet for tablet 7d96e93d95f14af79cfb268626559b1e (DEFAULT_TABLE table=default.Table_6eb0a92b1926487cbf0b978d26d6a7b1 [id=de7a420d40864f55b156e149308ba12f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:03.855886 19358 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7d96e93d95f14af79cfb268626559b1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:03.822184 18875 catalog_manager.cc:5746] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767) does not have the latest schema for tablet 7d76046ee6c8420fa4964decbe8209c1 (table default.Table_55e0ba23f44146339c7c6928440d4948 [id=4c70005d1005479fa6c3a6cb6f5ce658]). Expected version 1 got 0
I20260130 02:58:03.862049 18873 catalog_manager.cc:6199] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4: default.Table_9e942a1ce3944e9aa028e75656ce066d [id=18ed268eb9ad4450abf9874310ace074] alter complete (version 1)
I20260130 02:58:03.842444 19499 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ee7c8ff92ca9465fa18be6eae63d78ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:03.857204 19971 consensus_queue.cc:1048] T b7b596f944be4daf953b82b65d5cec2a P d870164263bb44b6b780a557daea3656 [LEADER]: Connected to new peer: Peer: permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.009s
I20260130 02:58:03.878561 18870 catalog_manager.cc:5941] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4: Sending DeleteTablet for 3 replicas of tablet b3f3177306914e3c8b253db54ee30fb6
I20260130 02:58:03.880322 18870 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50284:
table { table_name: "default.Table_c04335826bdf4c2c94c4cf6e3bca1a7a" } new_table_name: "default.Table_9f83f87711d94e8394abe1925ac84ad1" modify_external_catalogs: true
I20260130 02:58:03.881639 19202 tablet_service.cc:1552] Processing DeleteTablet for tablet b3f3177306914e3c8b253db54ee30fb6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:58:03 UTC) from {username='slave'} at 127.0.0.1:42870
W20260130 02:58:03.882043 18860 consensus_peers.cc:597] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4 -> Peer 3749baa287ed4f249329a74071781edd (127.18.100.124:34629): Couldn't send request to peer 3749baa287ed4f249329a74071781edd. Status: Network error: Client connection negotiation failed: client connection to 127.18.100.124:34629: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20260130 02:58:03.855885 19377 raft_consensus.cc:2468] T e47b22aafe5247c4b62c4193c24f427c P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 371a36d219164974abdf848841528152 in term 1.
I20260130 02:58:03.887249 19159 leader_election.cc:304] T e47b22aafe5247c4b62c4193c24f427c P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 371a36d219164974abdf848841528152, 78c2c46c7520492db35d51c244fb4f74; no voters:
I20260130 02:58:03.888054 20115 raft_consensus.cc:2804] T e47b22aafe5247c4b62c4193c24f427c P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Leader election won for term 1
I20260130 02:58:03.888479 20115 raft_consensus.cc:697] T e47b22aafe5247c4b62c4193c24f427c P 371a36d219164974abdf848841528152 [term 1 LEADER]: Becoming Leader. State: Replica: 371a36d219164974abdf848841528152, State: Running, Role: LEADER
I20260130 02:58:03.889237 20115 consensus_queue.cc:237] T e47b22aafe5247c4b62c4193c24f427c P 371a36d219164974abdf848841528152 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
W20260130 02:58:03.902621 19313 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.100.67:39571, user_credentials={real_user=slave}} blocked reactor thread for 109764us
I20260130 02:58:03.902467 18874 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50284:
table { table_name: "default.Table_71ecfab2cca249b49070a8340586ce5e" } modify_external_catalogs: true
I20260130 02:58:03.906330 18874 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50284:
table { table_name: "default.Table_71ecfab2cca249b49070a8340586ce5e" } modify_external_catalogs: true
I20260130 02:58:03.910091 19840 tablet_bootstrap.cc:654] T 730bd1d7d15b4cbe945b3ef03b2455f7 P 78c2c46c7520492db35d51c244fb4f74: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:03.912289 19352 tablet_service.cc:1552] Processing DeleteTablet for tablet b3f3177306914e3c8b253db54ee30fb6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:58:03 UTC) from {username='slave'} at 127.0.0.1:60110
I20260130 02:58:03.914958 19202 tablet_service.cc:1505] Processing CreateTablet for tablet 04f4ac27800c4ac3a79ae12d35479757 (DEFAULT_TABLE table=default.Table_ddfb5df90fdb40a087ac434d088bbf0e [id=1e72b7dd669d4539b61884ba63ff7183]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:03.916265 19807 tablet_bootstrap.cc:492] T 5ca9919f80954e74a4d91c94b74de0e4 P 371a36d219164974abdf848841528152: Bootstrap starting.
I20260130 02:58:03.922228 19807 tablet_bootstrap.cc:654] T 5ca9919f80954e74a4d91c94b74de0e4 P 371a36d219164974abdf848841528152: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:03.928558 19947 ts_tablet_manager.cc:1929] T b078934be4bb4fd1915c691eeea9d00f P d870164263bb44b6b780a557daea3656: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20260130 02:58:03.929631 19947 log.cc:1199] T b078934be4bb4fd1915c691eeea9d00f P d870164263bb44b6b780a557daea3656: Deleting WAL directory at /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-2/wal/wals/b078934be4bb4fd1915c691eeea9d00f
I20260130 02:58:03.930465 19947 ts_tablet_manager.cc:1950] T b078934be4bb4fd1915c691eeea9d00f P d870164263bb44b6b780a557daea3656: Deleting consensus metadata
I20260130 02:58:03.937748 19199 tablet_service.cc:1505] Processing CreateTablet for tablet 127a84346b9f473d9e7f739b1e65feb8 (DEFAULT_TABLE table=default.Table_4f2d6e4d9c8e465087d65efc9c2de442 [id=1515db389f284e2f9ed1a3cd0061d0f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:03.938894 19625 tablet_bootstrap.cc:654] T 2ca0df71836145748cada4e4559a5656 P d870164263bb44b6b780a557daea3656: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:03.948246 19807 tablet_bootstrap.cc:492] T 5ca9919f80954e74a4d91c94b74de0e4 P 371a36d219164974abdf848841528152: No bootstrap required, opened a new log
I20260130 02:58:03.948654 19807 ts_tablet_manager.cc:1403] T 5ca9919f80954e74a4d91c94b74de0e4 P 371a36d219164974abdf848841528152: Time spent bootstrapping tablet: real 0.033s user 0.005s sys 0.007s
I20260130 02:58:03.938090 19524 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e47b22aafe5247c4b62c4193c24f427c" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656"
I20260130 02:58:03.951634 19807 raft_consensus.cc:359] T 5ca9919f80954e74a4d91c94b74de0e4 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:03.952797 19807 raft_consensus.cc:385] T 5ca9919f80954e74a4d91c94b74de0e4 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:03.953045 19807 raft_consensus.cc:740] T 5ca9919f80954e74a4d91c94b74de0e4 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 371a36d219164974abdf848841528152, State: Initialized, Role: FOLLOWER
I20260130 02:58:03.953634 19807 consensus_queue.cc:260] T 5ca9919f80954e74a4d91c94b74de0e4 P 371a36d219164974abdf848841528152 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:03.945976 19498 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b89ed298e3dc49d789d166c613ad7fb8. 1 dirs total, 0 dirs full, 0 dirs failed
W20260130 02:58:03.960548 19462 consensus_peers.cc:597] T b7b596f944be4daf953b82b65d5cec2a P d870164263bb44b6b780a557daea3656 -> Peer 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767): Couldn't send request to peer 78c2c46c7520492db35d51c244fb4f74. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20260130 02:58:03.961710 19807 ts_tablet_manager.cc:1434] T 5ca9919f80954e74a4d91c94b74de0e4 P 371a36d219164974abdf848841528152: Time spent starting tablet: real 0.013s user 0.005s sys 0.001s
I20260130 02:58:03.964280 19758 ts_tablet_manager.cc:1929] T b7b596f944be4daf953b82b65d5cec2a P 78c2c46c7520492db35d51c244fb4f74: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260130 02:58:03.965107 18866 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50284:
name: "default.Table_1cb6165b7dda4b659c1c379902ee4930"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260130 02:58:03.966504 19807 tablet_bootstrap.cc:492] T 7b80b22bd44c4233837c951aa97e6996 P 371a36d219164974abdf848841528152: Bootstrap starting.
W20260130 02:58:03.966714 18866 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1cb6165b7dda4b659c1c379902ee4930 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260130 02:58:03.967885 19758 log.cc:1199] T b7b596f944be4daf953b82b65d5cec2a P 78c2c46c7520492db35d51c244fb4f74: Deleting WAL directory at /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-1/wal/wals/b7b596f944be4daf953b82b65d5cec2a
I20260130 02:58:03.970532 19360 tablet_service.cc:1505] Processing CreateTablet for tablet 127a84346b9f473d9e7f739b1e65feb8 (DEFAULT_TABLE table=default.Table_4f2d6e4d9c8e465087d65efc9c2de442 [id=1515db389f284e2f9ed1a3cd0061d0f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:03.972255 19807 tablet_bootstrap.cc:654] T 7b80b22bd44c4233837c951aa97e6996 P 371a36d219164974abdf848841528152: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:03.972812 19360 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 127a84346b9f473d9e7f739b1e65feb8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:03.972990 19358 tablet_service.cc:1505] Processing CreateTablet for tablet 04f4ac27800c4ac3a79ae12d35479757 (DEFAULT_TABLE table=default.Table_ddfb5df90fdb40a087ac434d088bbf0e [id=1e72b7dd669d4539b61884ba63ff7183]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:03.975993 19358 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 04f4ac27800c4ac3a79ae12d35479757. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:03.981851 19200 tablet_service.cc:1505] Processing CreateTablet for tablet 3ae5472cba1e4613abe4cda42ccb2496 (DEFAULT_TABLE table=default.Table_69267d31df644f4396a44130fa3d499d [id=7efa53800a974e31a38a4c60c0a1b466]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:03.990939 19840 tablet_bootstrap.cc:492] T 730bd1d7d15b4cbe945b3ef03b2455f7 P 78c2c46c7520492db35d51c244fb4f74: No bootstrap required, opened a new log
I20260130 02:58:03.991374 19840 ts_tablet_manager.cc:1403] T 730bd1d7d15b4cbe945b3ef03b2455f7 P 78c2c46c7520492db35d51c244fb4f74: Time spent bootstrapping tablet: real 0.253s user 0.005s sys 0.008s
I20260130 02:58:03.993988 19506 tablet_service.cc:1505] Processing CreateTablet for tablet 7d96e93d95f14af79cfb268626559b1e (DEFAULT_TABLE table=default.Table_6eb0a92b1926487cbf0b978d26d6a7b1 [id=de7a420d40864f55b156e149308ba12f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:03.994494 19500 tablet_service.cc:1552] Processing DeleteTablet for tablet b3f3177306914e3c8b253db54ee30fb6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:58:03 UTC) from {username='slave'} at 127.0.0.1:59068
I20260130 02:58:03.995563 19502 tablet_service.cc:1505] Processing CreateTablet for tablet 04f4ac27800c4ac3a79ae12d35479757 (DEFAULT_TABLE table=default.Table_ddfb5df90fdb40a087ac434d088bbf0e [id=1e72b7dd669d4539b61884ba63ff7183]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:03.995738 19507 tablet_service.cc:1505] Processing CreateTablet for tablet 127a84346b9f473d9e7f739b1e65feb8 (DEFAULT_TABLE table=default.Table_4f2d6e4d9c8e465087d65efc9c2de442 [id=1515db389f284e2f9ed1a3cd0061d0f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:03.997431 19501 tablet_service.cc:1505] Processing CreateTablet for tablet 3ae5472cba1e4613abe4cda42ccb2496 (DEFAULT_TABLE table=default.Table_69267d31df644f4396a44130fa3d499d [id=7efa53800a974e31a38a4c60c0a1b466]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:03.997838 20083 raft_consensus.cc:493] T 534dc31a9b7e492c82f3672c75742e6c P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:03.998278 20083 raft_consensus.cc:515] T 534dc31a9b7e492c82f3672c75742e6c P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
W20260130 02:58:03.998863 19159 leader_election.cc:343] T e47b22aafe5247c4b62c4193c24f427c P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:04.000159 20083 leader_election.cc:290] T 534dc31a9b7e492c82f3672c75742e6c P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), d870164263bb44b6b780a557daea3656 (127.18.100.67:39571)
I20260130 02:58:04.001628 19840 raft_consensus.cc:359] T 730bd1d7d15b4cbe945b3ef03b2455f7 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:04.002259 19840 raft_consensus.cc:385] T 730bd1d7d15b4cbe945b3ef03b2455f7 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:04.003870 19224 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "534dc31a9b7e492c82f3672c75742e6c" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152" is_pre_election: true
I20260130 02:58:04.002507 19840 raft_consensus.cc:740] T 730bd1d7d15b4cbe945b3ef03b2455f7 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78c2c46c7520492db35d51c244fb4f74, State: Initialized, Role: FOLLOWER
W20260130 02:58:04.005280 19314 leader_election.cc:343] T 534dc31a9b7e492c82f3672c75742e6c P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:04.005724 19840 consensus_queue.cc:260] T 730bd1d7d15b4cbe945b3ef03b2455f7 P 78c2c46c7520492db35d51c244fb4f74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:04.004292 20083 consensus_queue.cc:1048] T d3ba2df0b15c44f882070570f91b254d P 78c2c46c7520492db35d51c244fb4f74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20260130 02:58:03.977299 19758 ts_tablet_manager.cc:1950] T b7b596f944be4daf953b82b65d5cec2a P 78c2c46c7520492db35d51c244fb4f74: Deleting consensus metadata
I20260130 02:58:04.008648 19807 tablet_bootstrap.cc:492] T 7b80b22bd44c4233837c951aa97e6996 P 371a36d219164974abdf848841528152: No bootstrap required, opened a new log
I20260130 02:58:04.009092 19807 ts_tablet_manager.cc:1403] T 7b80b22bd44c4233837c951aa97e6996 P 371a36d219164974abdf848841528152: Time spent bootstrapping tablet: real 0.043s user 0.006s sys 0.006s
I20260130 02:58:04.011287 19807 raft_consensus.cc:359] T 7b80b22bd44c4233837c951aa97e6996 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:04.013316 19527 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "534dc31a9b7e492c82f3672c75742e6c" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
I20260130 02:58:04.019595 19807 raft_consensus.cc:385] T 7b80b22bd44c4233837c951aa97e6996 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:04.019958 19807 raft_consensus.cc:740] T 7b80b22bd44c4233837c951aa97e6996 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 371a36d219164974abdf848841528152, State: Initialized, Role: FOLLOWER
I20260130 02:58:04.022070 19840 ts_tablet_manager.cc:1434] T 730bd1d7d15b4cbe945b3ef03b2455f7 P 78c2c46c7520492db35d51c244fb4f74: Time spent starting tablet: real 0.030s user 0.003s sys 0.003s
I20260130 02:58:04.022686 19807 consensus_queue.cc:260] T 7b80b22bd44c4233837c951aa97e6996 P 371a36d219164974abdf848841528152 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:04.024008 19198 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7d96e93d95f14af79cfb268626559b1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:04.024765 19807 ts_tablet_manager.cc:1434] T 7b80b22bd44c4233837c951aa97e6996 P 371a36d219164974abdf848841528152: Time spent starting tablet: real 0.015s user 0.006s sys 0.000s
I20260130 02:58:04.025312 19202 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 04f4ac27800c4ac3a79ae12d35479757. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:04.025969 19840 tablet_bootstrap.cc:492] T 111287b5557d4abda1c5cd560fb0d019 P 78c2c46c7520492db35d51c244fb4f74: Bootstrap starting.
I20260130 02:58:04.027486 19501 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3ae5472cba1e4613abe4cda42ccb2496. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:04.027649 19200 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3ae5472cba1e4613abe4cda42ccb2496. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:04.028949 19502 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 04f4ac27800c4ac3a79ae12d35479757. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:04.029534 19507 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 127a84346b9f473d9e7f739b1e65feb8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:04.030043 19199 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 127a84346b9f473d9e7f739b1e65feb8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:04.029804 18869 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50284:
name: "default.Table_ed6e2e20d89542199e5cbc7842df964d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260130 02:58:04.031373 18869 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ed6e2e20d89542199e5cbc7842df964d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260130 02:58:04.031752 19840 tablet_bootstrap.cc:654] T 111287b5557d4abda1c5cd560fb0d019 P 78c2c46c7520492db35d51c244fb4f74: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:04.035808 19807 tablet_bootstrap.cc:492] T 2ca0df71836145748cada4e4559a5656 P 371a36d219164974abdf848841528152: Bootstrap starting.
I20260130 02:58:04.036168 19506 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7d96e93d95f14af79cfb268626559b1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:04.054031 19807 tablet_bootstrap.cc:654] T 2ca0df71836145748cada4e4559a5656 P 371a36d219164974abdf848841528152: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:04.059303 20115 consensus_queue.cc:1048] T ec28ff12927d450ead16d26cf016138d P 371a36d219164974abdf848841528152 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260130 02:58:04.067865 19840 tablet_bootstrap.cc:492] T 111287b5557d4abda1c5cd560fb0d019 P 78c2c46c7520492db35d51c244fb4f74: No bootstrap required, opened a new log
I20260130 02:58:04.068290 19840 ts_tablet_manager.cc:1403] T 111287b5557d4abda1c5cd560fb0d019 P 78c2c46c7520492db35d51c244fb4f74: Time spent bootstrapping tablet: real 0.043s user 0.008s sys 0.000s
W20260130 02:58:04.069422 19313 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.100.67:39571, user_credentials={real_user=slave}} blocked reactor thread for 61219.5us
I20260130 02:58:04.070489 19840 raft_consensus.cc:359] T 111287b5557d4abda1c5cd560fb0d019 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:04.071151 19840 raft_consensus.cc:385] T 111287b5557d4abda1c5cd560fb0d019 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:04.071388 19840 raft_consensus.cc:740] T 111287b5557d4abda1c5cd560fb0d019 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78c2c46c7520492db35d51c244fb4f74, State: Initialized, Role: FOLLOWER
I20260130 02:58:04.071974 19840 consensus_queue.cc:260] T 111287b5557d4abda1c5cd560fb0d019 P 78c2c46c7520492db35d51c244fb4f74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:04.073523 19840 ts_tablet_manager.cc:1434] T 111287b5557d4abda1c5cd560fb0d019 P 78c2c46c7520492db35d51c244fb4f74: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
W20260130 02:58:04.075919 18870 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 91f954f3ef4244c9a575a7421a7e75eb consensus state has no leader
I20260130 02:58:04.080737 18870 master_service.cc:954] ReplaceTablet: received request to replace tablet f54addb1a4304ce69c711ae0cc3b452c from {username='slave'} at 127.0.0.1:50284
I20260130 02:58:04.084110 19754 ts_tablet_manager.cc:1929] T 5b2976108584478a98fd90fe6594af66 P 371a36d219164974abdf848841528152: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260130 02:58:04.084460 19754 log.cc:1199] T 5b2976108584478a98fd90fe6594af66 P 371a36d219164974abdf848841528152: Deleting WAL directory at /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-0/wal/wals/5b2976108584478a98fd90fe6594af66
I20260130 02:58:04.085132 19754 ts_tablet_manager.cc:1950] T 5b2976108584478a98fd90fe6594af66 P 371a36d219164974abdf848841528152: Deleting consensus metadata
I20260130 02:58:04.089232 19840 tablet_bootstrap.cc:492] T 91f954f3ef4244c9a575a7421a7e75eb P 78c2c46c7520492db35d51c244fb4f74: Bootstrap starting.
I20260130 02:58:04.090590 18861 catalog_manager.cc:4985] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021): tablet 5b2976108584478a98fd90fe6594af66 (table default.Table_a78ce7ab85924506b184fb74db3ed0c3 [id=5f4c327b216449a1a8b146d0f652de52]) successfully deleted
I20260130 02:58:04.091030 19754 tablet_replica.cc:333] T 7a902d0d571a42bb82ebffd48f7eca3f P 371a36d219164974abdf848841528152: stopping tablet replica
I20260130 02:58:04.091810 19754 raft_consensus.cc:2243] T 7a902d0d571a42bb82ebffd48f7eca3f P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260130 02:58:04.091905 18861 catalog_manager.cc:4952] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021): delete failed for tablet 96584ca174b140f7ad1a2ca698baca71 because the tablet was not found. No further retry: Not found: Tablet not found: 96584ca174b140f7ad1a2ca698baca71
I20260130 02:58:04.092262 19754 raft_consensus.cc:2272] T 7a902d0d571a42bb82ebffd48f7eca3f P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260130 02:58:04.092429 18861 catalog_manager.cc:4952] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021): delete failed for tablet 073b5941850c4140b1751956dd7cf5ef because the tablet was not found. No further retry: Not found: Tablet not found: 073b5941850c4140b1751956dd7cf5ef
I20260130 02:58:04.093917 19754 ts_tablet_manager.cc:1916] T 7a902d0d571a42bb82ebffd48f7eca3f P 371a36d219164974abdf848841528152: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260130 02:58:04.079051 19313 leader_election.cc:343] T 534dc31a9b7e492c82f3672c75742e6c P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:04.102878 19313 leader_election.cc:304] T 534dc31a9b7e492c82f3672c75742e6c P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78c2c46c7520492db35d51c244fb4f74; no voters: 371a36d219164974abdf848841528152, d870164263bb44b6b780a557daea3656
I20260130 02:58:04.117818 20115 consensus_queue.cc:1048] T ec28ff12927d450ead16d26cf016138d P 371a36d219164974abdf848841528152 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260130 02:58:04.119365 19840 tablet_bootstrap.cc:654] T 91f954f3ef4244c9a575a7421a7e75eb P 78c2c46c7520492db35d51c244fb4f74: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:04.122884 20089 raft_consensus.cc:2749] T 534dc31a9b7e492c82f3672c75742e6c P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260130 02:58:04.123050 19861 raft_consensus.cc:493] T 7d76046ee6c8420fa4964decbe8209c1 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:04.123409 19861 raft_consensus.cc:515] T 7d76046ee6c8420fa4964decbe8209c1 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:04.123768 20089 raft_consensus.cc:493] T 481bd4cca6ee4c178873bf12392e3441 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:04.124142 20089 raft_consensus.cc:515] T 481bd4cca6ee4c178873bf12392e3441 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:04.125849 20089 leader_election.cc:290] T 481bd4cca6ee4c178873bf12392e3441 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d870164263bb44b6b780a557daea3656 (127.18.100.67:39571), 371a36d219164974abdf848841528152 (127.18.100.65:42021)
W20260130 02:58:04.127301 19313 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.100.67:39571, user_credentials={real_user=slave}} blocked reactor thread for 48280.7us
I20260130 02:58:04.131409 19527 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "481bd4cca6ee4c178873bf12392e3441" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
I20260130 02:58:04.132315 19758 tablet_replica.cc:333] T b3f3177306914e3c8b253db54ee30fb6 P 78c2c46c7520492db35d51c244fb4f74: stopping tablet replica
I20260130 02:58:04.132380 18860 catalog_manager.cc:4985] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767): tablet b7b596f944be4daf953b82b65d5cec2a (table default.Table_22d10332a83f4700ac21892bab67f46b [id=da3a9cb1f6d5419abc1afc93659b5324]) successfully deleted
I20260130 02:58:04.133149 19758 raft_consensus.cc:2243] T b3f3177306914e3c8b253db54ee30fb6 P 78c2c46c7520492db35d51c244fb4f74 [term 1 LEADER]: Raft consensus shutting down.
I20260130 02:58:04.134030 19758 raft_consensus.cc:2272] T b3f3177306914e3c8b253db54ee30fb6 P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260130 02:58:04.145955 19861 leader_election.cc:290] T 7d76046ee6c8420fa4964decbe8209c1 P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d870164263bb44b6b780a557daea3656 (127.18.100.67:39571), 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767)
I20260130 02:58:04.147177 19224 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "481bd4cca6ee4c178873bf12392e3441" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152" is_pre_election: true
I20260130 02:58:04.147770 19224 raft_consensus.cc:2468] T 481bd4cca6ee4c178873bf12392e3441 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78c2c46c7520492db35d51c244fb4f74 in term 0.
I20260130 02:58:04.164986 19754 ts_tablet_manager.cc:1929] T 7a902d0d571a42bb82ebffd48f7eca3f P 371a36d219164974abdf848841528152: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260130 02:58:04.166309 19754 log.cc:1199] T 7a902d0d571a42bb82ebffd48f7eca3f P 371a36d219164974abdf848841528152: Deleting WAL directory at /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-0/wal/wals/7a902d0d571a42bb82ebffd48f7eca3f
I20260130 02:58:04.167075 19754 ts_tablet_manager.cc:1950] T 7a902d0d571a42bb82ebffd48f7eca3f P 371a36d219164974abdf848841528152: Deleting consensus metadata
I20260130 02:58:04.167564 19947 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59068 (request call id 101) took 2360 ms. Trace:
I20260130 02:58:04.168057 19947 rpcz_store.cc:276] 0130 02:58:01.806846 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:58:01.807303 (+ 457us) service_pool.cc:225] Handling call
0130 02:58:03.626858 (+1819555us) ts_tablet_manager.cc:1151] Deleting tablet b078934be4bb4fd1915c691eeea9d00f
0130 02:58:03.626908 (+ 50us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0130 02:58:03.806416 (+179508us) tablet_metadata.cc:641] Metadata flushed
0130 02:58:03.928315 (+121899us) tablet_metadata.cc:641] Metadata flushed
0130 02:58:04.109724 (+181409us) spinlock_profiling.cc:243] Waited 1.78 ms on lock 0x7b5800000098. stack: 00007f22d9409373 00007f22d9409270 00007f22d8f7e574 0000000000508744 00000000005086d9 00007f22d927b8b1 00007f22d93fddb1 00007f22d93fe3e2 00007f22e2258189 00007f22e14ab108 00007f22e1572462 00007f22e15a2f80 00007f22e158954d 00007f22e15894f9 00007f22e1589489 00007f22e1589451
0130 02:58:04.167496 (+ 57772us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4098816,"tablet-delete.queue_time_us":1792929}
I20260130 02:58:04.169111 19527 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7d76046ee6c8420fa4964decbe8209c1" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
I20260130 02:58:04.169211 18861 catalog_manager.cc:4985] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021): tablet 7a902d0d571a42bb82ebffd48f7eca3f (table default.Table_298b1af220674b6ab20d3652e8a637a1 [id=742a870a99554fc48fb4f369e68fc6d6]) successfully deleted
I20260130 02:58:04.169737 19947 tablet_replica.cc:333] T 5b2976108584478a98fd90fe6594af66 P d870164263bb44b6b780a557daea3656: stopping tablet replica
I20260130 02:58:04.169891 18860 catalog_manager.cc:4985] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): tablet b078934be4bb4fd1915c691eeea9d00f (table default.Table_b63ca4e4060f48e893b0099fb6433f04 [id=720decfb55c24f34983ef636f06e3169]) successfully deleted
I20260130 02:58:04.170619 19947 raft_consensus.cc:2243] T 5b2976108584478a98fd90fe6594af66 P d870164263bb44b6b780a557daea3656 [term 1 LEADER]: Raft consensus shutting down.
W20260130 02:58:04.179204 18861 catalog_manager.cc:4965] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021): delete failed for tablet 6573198830ad45628e71952a7d9c00a0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6573198830ad45628e71952a7d9c00a0 already in progress: creating tablet
I20260130 02:58:04.183161 19754 tablet_replica.cc:333] T b7b596f944be4daf953b82b65d5cec2a P 371a36d219164974abdf848841528152: stopping tablet replica
I20260130 02:58:04.182713 20115 raft_consensus.cc:493] T 7b80b22bd44c4233837c951aa97e6996 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:04.183949 19754 raft_consensus.cc:2243] T b7b596f944be4daf953b82b65d5cec2a P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260130 02:58:04.183720 20115 raft_consensus.cc:515] T 7b80b22bd44c4233837c951aa97e6996 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:04.184427 19754 raft_consensus.cc:2272] T b7b596f944be4daf953b82b65d5cec2a P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260130 02:58:04.185554 20115 leader_election.cc:290] T 7b80b22bd44c4233837c951aa97e6996 P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d870164263bb44b6b780a557daea3656 (127.18.100.67:39571), 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767)
I20260130 02:58:04.188406 18866 master_service.cc:954] ReplaceTablet: received request to replace tablet 5b2976108584478a98fd90fe6594af66 from {username='slave'} at 127.0.0.1:50284
W20260130 02:58:04.199347 18861 catalog_manager.cc:4965] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021): delete failed for tablet 730bd1d7d15b4cbe945b3ef03b2455f7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 730bd1d7d15b4cbe945b3ef03b2455f7 already in progress: creating tablet
I20260130 02:58:04.201113 19754 ts_tablet_manager.cc:1916] T b7b596f944be4daf953b82b65d5cec2a P 371a36d219164974abdf848841528152: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260130 02:58:04.203943 18871 catalog_manager.cc:5746] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021) does not have the latest schema for tablet 38a7a10e36874183bf9fe55a38d829f6 (table default.Table_f359d249f8274f319f5ff4e4ea89dd22 [id=27adf356fd004597a54c7a9887dd3ca8]). Expected version 4 got 3
I20260130 02:58:04.206559 18871 catalog_manager.cc:5654] T ec28ff12927d450ead16d26cf016138d P 371a36d219164974abdf848841528152 reported cstate change: term changed from 0 to 1, leader changed from <none> to 371a36d219164974abdf848841528152 (127.18.100.65). New cstate: current_term: 1 leader_uuid: "371a36d219164974abdf848841528152" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } health_report { overall_health: UNKNOWN } } }
I20260130 02:58:04.222177 19807 tablet_bootstrap.cc:492] T 2ca0df71836145748cada4e4559a5656 P 371a36d219164974abdf848841528152: No bootstrap required, opened a new log
I20260130 02:58:04.187186 19527 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7b80b22bd44c4233837c951aa97e6996" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
I20260130 02:58:04.222601 19807 ts_tablet_manager.cc:1403] T 2ca0df71836145748cada4e4559a5656 P 371a36d219164974abdf848841528152: Time spent bootstrapping tablet: real 0.188s user 0.020s sys 0.020s
I20260130 02:58:04.222909 19527 raft_consensus.cc:2468] T 7b80b22bd44c4233837c951aa97e6996 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 371a36d219164974abdf848841528152 in term 0.
I20260130 02:58:04.224578 18871 catalog_manager.cc:5746] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021) does not have the latest schema for tablet 7d76046ee6c8420fa4964decbe8209c1 (table default.Table_55e0ba23f44146339c7c6928440d4948 [id=4c70005d1005479fa6c3a6cb6f5ce658]). Expected version 1 got 0
I20260130 02:58:04.224745 19807 raft_consensus.cc:359] T 2ca0df71836145748cada4e4559a5656 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:04.225324 19807 raft_consensus.cc:385] T 2ca0df71836145748cada4e4559a5656 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:04.225570 19807 raft_consensus.cc:740] T 2ca0df71836145748cada4e4559a5656 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 371a36d219164974abdf848841528152, State: Initialized, Role: FOLLOWER
I20260130 02:58:04.226065 18874 catalog_manager.cc:5941] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4: Sending DeleteTablet for 3 replicas of tablet a743a82540014c85aeab0e9d0994b0dc
I20260130 02:58:04.226193 19807 consensus_queue.cc:260] T 2ca0df71836145748cada4e4559a5656 P 371a36d219164974abdf848841528152 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:04.227833 18874 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50284:
table { table_name: "default.Table_6eb0a92b1926487cbf0b978d26d6a7b1" } modify_external_catalogs: true
I20260130 02:58:04.228297 18874 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50284:
table { table_name: "default.Table_6eb0a92b1926487cbf0b978d26d6a7b1" } modify_external_catalogs: true
I20260130 02:58:04.144346 19357 tablet_service.cc:1505] Processing CreateTablet for tablet 3ae5472cba1e4613abe4cda42ccb2496 (DEFAULT_TABLE table=default.Table_69267d31df644f4396a44130fa3d499d [id=7efa53800a974e31a38a4c60c0a1b466]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:04.155809 19314 leader_election.cc:304] T 481bd4cca6ee4c178873bf12392e3441 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 371a36d219164974abdf848841528152, 78c2c46c7520492db35d51c244fb4f74; no voters:
I20260130 02:58:04.156800 19377 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7d76046ee6c8420fa4964decbe8209c1" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74" is_pre_election: true
I20260130 02:58:04.231011 19377 raft_consensus.cc:2393] T 7d76046ee6c8420fa4964decbe8209c1 P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 371a36d219164974abdf848841528152 in current term 1: Already voted for candidate d870164263bb44b6b780a557daea3656 in this term.
I20260130 02:58:04.231660 19357 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3ae5472cba1e4613abe4cda42ccb2496. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:04.232074 19159 leader_election.cc:304] T 7d76046ee6c8420fa4964decbe8209c1 P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 371a36d219164974abdf848841528152; no voters: 78c2c46c7520492db35d51c244fb4f74, d870164263bb44b6b780a557daea3656
I20260130 02:58:04.207226 19378 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7b80b22bd44c4233837c951aa97e6996" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74" is_pre_election: true
I20260130 02:58:04.232764 19378 raft_consensus.cc:2468] T 7b80b22bd44c4233837c951aa97e6996 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 371a36d219164974abdf848841528152 in term 0.
I20260130 02:58:04.233836 19202 tablet_service.cc:1552] Processing DeleteTablet for tablet a743a82540014c85aeab0e9d0994b0dc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:58:03 UTC) from {username='slave'} at 127.0.0.1:42870
I20260130 02:58:04.234772 19354 tablet_service.cc:1552] Processing DeleteTablet for tablet a743a82540014c85aeab0e9d0994b0dc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:58:03 UTC) from {username='slave'} at 127.0.0.1:60110
I20260130 02:58:04.236133 19159 leader_election.cc:304] T 7b80b22bd44c4233837c951aa97e6996 P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 371a36d219164974abdf848841528152, 78c2c46c7520492db35d51c244fb4f74; no voters:
I20260130 02:58:04.236749 19861 raft_consensus.cc:3060] T 7d76046ee6c8420fa4964decbe8209c1 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:04.238497 20115 raft_consensus.cc:2804] T 7b80b22bd44c4233837c951aa97e6996 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260130 02:58:04.238848 20115 raft_consensus.cc:493] T 7b80b22bd44c4233837c951aa97e6996 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260130 02:58:04.239135 20115 raft_consensus.cc:3060] T 7b80b22bd44c4233837c951aa97e6996 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:04.242025 19861 raft_consensus.cc:2749] T 7d76046ee6c8420fa4964decbe8209c1 P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260130 02:58:04.245731 20115 raft_consensus.cc:515] T 7b80b22bd44c4233837c951aa97e6996 P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
W20260130 02:58:04.247885 19314 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.100.65:42021, user_credentials={real_user=slave}} blocked reactor thread for 92164.8us
I20260130 02:58:04.247984 18869 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50284:
name: "default.Table_8c27a7d10b7847e78029f7e95321e9b6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260130 02:58:04.248744 20115 leader_election.cc:290] T 7b80b22bd44c4233837c951aa97e6996 P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 election: Requested vote from peers d870164263bb44b6b780a557daea3656 (127.18.100.67:39571), 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767)
I20260130 02:58:04.249562 20089 raft_consensus.cc:2804] T 481bd4cca6ee4c178873bf12392e3441 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Leader pre-election won for term 1
W20260130 02:58:04.249502 18869 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8c27a7d10b7847e78029f7e95321e9b6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260130 02:58:04.249897 20089 raft_consensus.cc:493] T 481bd4cca6ee4c178873bf12392e3441 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260130 02:58:04.250211 20089 raft_consensus.cc:3060] T 481bd4cca6ee4c178873bf12392e3441 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:04.251931 19807 ts_tablet_manager.cc:1434] T 2ca0df71836145748cada4e4559a5656 P 371a36d219164974abdf848841528152: Time spent starting tablet: real 0.029s user 0.006s sys 0.000s
I20260130 02:58:04.253466 19807 tablet_bootstrap.cc:492] T a90e92d569254bbf8a5fbf286baf5cce P 371a36d219164974abdf848841528152: Bootstrap starting.
I20260130 02:58:04.255096 18870 catalog_manager.cc:5941] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4: Sending DeleteTablet for 3 replicas of tablet f54addb1a4304ce69c711ae0cc3b452c
I20260130 02:58:04.255523 19527 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7b80b22bd44c4233837c951aa97e6996" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656"
I20260130 02:58:04.256017 19527 raft_consensus.cc:3060] T 7b80b22bd44c4233837c951aa97e6996 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:04.256340 18870 catalog_manager.cc:6724] ReplaceTablet: tablet f54addb1a4304ce69c711ae0cc3b452c deleted and replaced by tablet 7783db3568f4434e91e9e456799cc312
I20260130 02:58:04.257056 18870 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50284:
table { table_name: "default.Table_caccb5141e014c558ebbda10e2415e16" } new_table_name: "default.Table_5c1c447e85524ac5a61ffd8a48509d36" modify_external_catalogs: true
I20260130 02:58:04.258922 19377 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7b80b22bd44c4233837c951aa97e6996" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74"
I20260130 02:58:04.259474 19377 raft_consensus.cc:3060] T 7b80b22bd44c4233837c951aa97e6996 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:04.260906 19354 tablet_service.cc:1552] Processing DeleteTablet for tablet f54addb1a4304ce69c711ae0cc3b452c with delete_type TABLET_DATA_DELETED (replaced by tablet 7783db3568f4434e91e9e456799cc312) from {username='slave'} at 127.0.0.1:60110
I20260130 02:58:04.271646 19947 raft_consensus.cc:2272] T 5b2976108584478a98fd90fe6594af66 P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260130 02:58:04.278821 19505 tablet_service.cc:1552] Processing DeleteTablet for tablet a743a82540014c85aeab0e9d0994b0dc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:58:03 UTC) from {username='slave'} at 127.0.0.1:59068
I20260130 02:58:04.287331 18875 catalog_manager.cc:6199] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4: default.Table_333d2efbf068485a871f566d50e563ce [id=8b7db40afb894ae99625a756c75ef1de] alter complete (version 1)
I20260130 02:58:04.262512 19754 ts_tablet_manager.cc:1929] T b7b596f944be4daf953b82b65d5cec2a P 371a36d219164974abdf848841528152: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260130 02:58:04.288097 19754 log.cc:1199] T b7b596f944be4daf953b82b65d5cec2a P 371a36d219164974abdf848841528152: Deleting WAL directory at /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-0/wal/wals/b7b596f944be4daf953b82b65d5cec2a
I20260130 02:58:04.288805 19754 ts_tablet_manager.cc:1950] T b7b596f944be4daf953b82b65d5cec2a P 371a36d219164974abdf848841528152: Deleting consensus metadata
I20260130 02:58:04.278757 20115 consensus_queue.cc:1048] T bd43c5d148514836869a8ac648c8ad11 P 371a36d219164974abdf848841528152 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.012s
I20260130 02:58:04.278137 19807 tablet_bootstrap.cc:654] T a90e92d569254bbf8a5fbf286baf5cce P 371a36d219164974abdf848841528152: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:04.275310 20089 raft_consensus.cc:515] T 481bd4cca6ee4c178873bf12392e3441 P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:04.296190 18873 catalog_manager.cc:6199] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4: default.Table_da3d581b15a94840898777f74611994e [id=b7e241ced53e4160ac6fa1f65ef3674e] alter complete (version 1)
I20260130 02:58:04.298255 20089 leader_election.cc:290] T 481bd4cca6ee4c178873bf12392e3441 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 election: Requested vote from peers d870164263bb44b6b780a557daea3656 (127.18.100.67:39571), 371a36d219164974abdf848841528152 (127.18.100.65:42021)
I20260130 02:58:04.298930 18861 catalog_manager.cc:4985] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021): tablet b7b596f944be4daf953b82b65d5cec2a (table default.Table_22d10332a83f4700ac21892bab67f46b [id=da3a9cb1f6d5419abc1afc93659b5324]) successfully deleted
I20260130 02:58:04.269011 19377 raft_consensus.cc:2468] T 7b80b22bd44c4233837c951aa97e6996 P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 371a36d219164974abdf848841528152 in term 1.
I20260130 02:58:04.307606 19159 leader_election.cc:304] T 7b80b22bd44c4233837c951aa97e6996 P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 371a36d219164974abdf848841528152, 78c2c46c7520492db35d51c244fb4f74; no voters:
I20260130 02:58:04.308454 20115 raft_consensus.cc:2804] T 7b80b22bd44c4233837c951aa97e6996 P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Leader election won for term 1
I20260130 02:58:04.308869 20115 raft_consensus.cc:697] T 7b80b22bd44c4233837c951aa97e6996 P 371a36d219164974abdf848841528152 [term 1 LEADER]: Becoming Leader. State: Replica: 371a36d219164974abdf848841528152, State: Running, Role: LEADER
I20260130 02:58:04.310387 19354 tablet_service.cc:1552] Processing DeleteTablet for tablet ceda64471a92476f9ba183d2beab30d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:58:01 UTC) from {username='slave'} at 127.0.0.1:60110
I20260130 02:58:04.311184 20115 consensus_queue.cc:237] T 7b80b22bd44c4233837c951aa97e6996 P 371a36d219164974abdf848841528152 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:04.312150 19354 tablet_service.cc:1552] Processing DeleteTablet for tablet 7a902d0d571a42bb82ebffd48f7eca3f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:57:57 UTC) from {username='slave'} at 127.0.0.1:60110
I20260130 02:58:04.315232 19352 tablet_service.cc:1552] Processing DeleteTablet for tablet 37470ce4356f4542b621e21698551209 with delete_type TABLET_DATA_DELETED (replaced by tablet 7caf7cbad5054396b5617094e4d536a0) from {username='slave'} at 127.0.0.1:60110
I20260130 02:58:04.315599 19224 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "481bd4cca6ee4c178873bf12392e3441" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152"
I20260130 02:58:04.316120 19224 raft_consensus.cc:3060] T 481bd4cca6ee4c178873bf12392e3441 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:04.316576 19354 tablet_service.cc:1552] Processing DeleteTablet for tablet 5b2976108584478a98fd90fe6594af66 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:57:58 UTC) from {username='slave'} at 127.0.0.1:60110
I20260130 02:58:04.319190 19352 tablet_service.cc:1552] Processing DeleteTablet for tablet d701152960b14396a6ac1bd10480d2b6 with delete_type TABLET_DATA_DELETED (replaced by tablet 481bd4cca6ee4c178873bf12392e3441) from {username='slave'} at 127.0.0.1:60110
W20260130 02:58:04.320590 18873 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.18.100.66:59691 (request call id 21) took 2179 ms (client timeout 1000 ms). Trace:
W20260130 02:58:04.306432 19313 leader_election.cc:343] T 481bd4cca6ee4c178873bf12392e3441 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:04.321573 19354 tablet_service.cc:1552] Processing DeleteTablet for tablet b078934be4bb4fd1915c691eeea9d00f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:58:01 UTC) from {username='slave'} at 127.0.0.1:60110
I20260130 02:58:04.322285 18872 catalog_manager.cc:6199] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4: default.Table_bc11cb1bce0348ed9ea0912b3ee521b9 [id=df7d3d68e36142ff8564e63ffe0b8797] alter complete (version 1)
I20260130 02:58:04.322376 19947 ts_tablet_manager.cc:1916] T 5b2976108584478a98fd90fe6594af66 P d870164263bb44b6b780a557daea3656: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260130 02:58:04.325548 19758 ts_tablet_manager.cc:1916] T b3f3177306914e3c8b253db54ee30fb6 P 78c2c46c7520492db35d51c244fb4f74: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260130 02:58:04.327831 19754 tablet_replica.cc:333] T b3f3177306914e3c8b253db54ee30fb6 P 371a36d219164974abdf848841528152: stopping tablet replica
I20260130 02:58:04.328583 19754 raft_consensus.cc:2243] T b3f3177306914e3c8b253db54ee30fb6 P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260130 02:58:04.329080 19754 raft_consensus.cc:2272] T b3f3177306914e3c8b253db54ee30fb6 P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260130 02:58:04.343405 20151 tablet.cc:1722] T 38a7a10e36874183bf9fe55a38d829f6 P 371a36d219164974abdf848841528152: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20260130 02:58:04.349931 20157 raft_consensus.cc:493] T 5ca9919f80954e74a4d91c94b74de0e4 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260130 02:58:04.351330 19462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.100.66:40767, user_credentials={real_user=slave}} blocked reactor thread for 135715us
I20260130 02:58:04.358909 19527 raft_consensus.cc:2468] T 7b80b22bd44c4233837c951aa97e6996 P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 371a36d219164974abdf848841528152 in term 1.
I20260130 02:58:04.369547 18875 catalog_manager.cc:6199] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4: default.Table_e84c6277dfc944059f62cbe0d6b937ee [id=747af828147149efb7b7b8d33c49286d] alter complete (version 2)
I20260130 02:58:04.381263 18874 catalog_manager.cc:5941] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4: Sending DeleteTablet for 3 replicas of tablet 7d96e93d95f14af79cfb268626559b1e
I20260130 02:58:04.389500 19357 tablet_service.cc:1552] Processing DeleteTablet for tablet 7d96e93d95f14af79cfb268626559b1e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:58:04 UTC) from {username='slave'} at 127.0.0.1:60110
W20260130 02:58:04.391251 18870 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9fe0448bcc4a4346b3c67fc3f626ed3c has no consensus state
I20260130 02:58:04.402609 19758 ts_tablet_manager.cc:1929] T b3f3177306914e3c8b253db54ee30fb6 P 78c2c46c7520492db35d51c244fb4f74: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260130 02:58:04.402987 19758 log.cc:1199] T b3f3177306914e3c8b253db54ee30fb6 P 78c2c46c7520492db35d51c244fb4f74: Deleting WAL directory at /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-1/wal/wals/b3f3177306914e3c8b253db54ee30fb6
I20260130 02:58:04.403677 19758 ts_tablet_manager.cc:1950] T b3f3177306914e3c8b253db54ee30fb6 P 78c2c46c7520492db35d51c244fb4f74: Deleting consensus metadata
I20260130 02:58:04.408752 18874 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50284:
name: "default.Table_a3ecda6d64704b7fad065a5845683ad8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260130 02:58:04.410089 19758 tablet_replica.cc:333] T a743a82540014c85aeab0e9d0994b0dc P 78c2c46c7520492db35d51c244fb4f74: stopping tablet replica
W20260130 02:58:04.410545 18874 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a3ecda6d64704b7fad065a5845683ad8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260130 02:58:04.410845 19758 raft_consensus.cc:2243] T a743a82540014c85aeab0e9d0994b0dc P 78c2c46c7520492db35d51c244fb4f74 [term 1 LEADER]: Raft consensus shutting down.
I20260130 02:58:04.411566 19758 raft_consensus.cc:2272] T a743a82540014c85aeab0e9d0994b0dc P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260130 02:58:04.412452 18870 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50284:
table { table_name: "default.Table_112ffec66bd143398baeb18ddf231064" } new_table_name: "default.Table_dd22e9dc63f84d8688c0071f427f0b25" modify_external_catalogs: true
I20260130 02:58:04.415848 18860 catalog_manager.cc:4985] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767): tablet b3f3177306914e3c8b253db54ee30fb6 (table default.Table_87dbbd65d92043b0b91c801517e56dfe [id=adced31928ba4204b79e3b43f8b737ee]) successfully deleted
I20260130 02:58:04.416121 19758 ts_tablet_manager.cc:1916] T a743a82540014c85aeab0e9d0994b0dc P 78c2c46c7520492db35d51c244fb4f74: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260130 02:58:04.422160 18866 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50284:
name: "default.Table_0f3e44ae93bc4c028ed3929b94a49a24"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260130 02:58:04.423650 18866 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0f3e44ae93bc4c028ed3929b94a49a24 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260130 02:58:04.428885 19947 ts_tablet_manager.cc:1929] T 5b2976108584478a98fd90fe6594af66 P d870164263bb44b6b780a557daea3656: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260130 02:58:04.429328 19947 log.cc:1199] T 5b2976108584478a98fd90fe6594af66 P d870164263bb44b6b780a557daea3656: Deleting WAL directory at /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-2/wal/wals/5b2976108584478a98fd90fe6594af66
I20260130 02:58:04.430059 19947 ts_tablet_manager.cc:1950] T 5b2976108584478a98fd90fe6594af66 P d870164263bb44b6b780a557daea3656: Deleting consensus metadata
I20260130 02:58:04.430835 19947 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59068 (request call id 105) took 2555 ms. Trace:
I20260130 02:58:04.431143 19947 rpcz_store.cc:276] 0130 02:58:01.875002 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:58:01.875107 (+ 105us) service_pool.cc:225] Handling call
0130 02:58:04.169583 (+2294476us) ts_tablet_manager.cc:1151] Deleting tablet 5b2976108584478a98fd90fe6594af66
0130 02:58:04.169628 (+ 45us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0130 02:58:04.367165 (+197537us) tablet_metadata.cc:641] Metadata flushed
0130 02:58:04.410393 (+ 43228us) tablet_metadata.cc:641] Metadata flushed
0130 02:58:04.430776 (+ 20383us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":49665,"tablet-delete.queue_time_us":2293931}
I20260130 02:58:04.432235 19947 tablet_replica.cc:333] T d701152960b14396a6ac1bd10480d2b6 P d870164263bb44b6b780a557daea3656: stopping tablet replica
I20260130 02:58:04.432965 19947 raft_consensus.cc:2243] T d701152960b14396a6ac1bd10480d2b6 P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260130 02:58:04.446184 18859 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet 9fe0448bcc4a4346b3c67fc3f626ed3c (table default.Table_5c1c447e85524ac5a61ffd8a48509d36 [id=70aaf21ad9bd4d959f46939c6b4a4880], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 9fe0448bcc4a4346b3c67fc3f626ed3c has no consensus state
I20260130 02:58:04.452509 19758 ts_tablet_manager.cc:1929] T a743a82540014c85aeab0e9d0994b0dc P 78c2c46c7520492db35d51c244fb4f74: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260130 02:58:04.452858 19758 log.cc:1199] T a743a82540014c85aeab0e9d0994b0dc P 78c2c46c7520492db35d51c244fb4f74: Deleting WAL directory at /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-1/wal/wals/a743a82540014c85aeab0e9d0994b0dc
I20260130 02:58:04.453512 19758 ts_tablet_manager.cc:1950] T a743a82540014c85aeab0e9d0994b0dc P 78c2c46c7520492db35d51c244fb4f74: Deleting consensus metadata
I20260130 02:58:04.462920 19200 tablet_service.cc:1552] Processing DeleteTablet for tablet f54addb1a4304ce69c711ae0cc3b452c with delete_type TABLET_DATA_DELETED (replaced by tablet 7783db3568f4434e91e9e456799cc312) from {username='slave'} at 127.0.0.1:42870
I20260130 02:58:04.464319 19947 raft_consensus.cc:2272] T d701152960b14396a6ac1bd10480d2b6 P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260130 02:58:04.465006 19224 raft_consensus.cc:2468] T 481bd4cca6ee4c178873bf12392e3441 P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78c2c46c7520492db35d51c244fb4f74 in term 1.
I20260130 02:58:04.467235 19754 ts_tablet_manager.cc:1916] T b3f3177306914e3c8b253db54ee30fb6 P 371a36d219164974abdf848841528152: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260130 02:58:04.470790 19198 tablet_service.cc:1552] Processing DeleteTablet for tablet 7d96e93d95f14af79cfb268626559b1e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:58:04 UTC) from {username='slave'} at 127.0.0.1:42870
I20260130 02:58:04.480089 19807 tablet_bootstrap.cc:492] T a90e92d569254bbf8a5fbf286baf5cce P 371a36d219164974abdf848841528152: No bootstrap required, opened a new log
I20260130 02:58:04.480551 19861 raft_consensus.cc:493] T 5ca9919f80954e74a4d91c94b74de0e4 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260130 02:58:04.320971 18873 rpcz_store.cc:269] 0130 02:58:02.140758 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:58:02.140969 (+ 211us) service_pool.cc:225] Handling call
0130 02:58:04.320517 (+2179548us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0130 02:58:02.616710 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0130 02:58:02.617132 (+ 422us) write_op.cc:432] Acquiring schema lock in shared mode
0130 02:58:02.617158 (+ 26us) write_op.cc:435] Acquired schema lock
0130 02:58:02.617170 (+ 12us) tablet.cc:660] Decoding operations
0130 02:58:02.617580 (+ 410us) write_op.cc:620] Acquiring the partition lock for write op
0130 02:58:02.617618 (+ 38us) write_op.cc:641] Partition lock acquired for write op
0130 02:58:02.617640 (+ 22us) tablet.cc:683] Acquiring locks for 3 operations
0130 02:58:02.617877 (+ 237us) tablet.cc:699] Row locks acquired
0130 02:58:02.617899 (+ 22us) write_op.cc:260] PREPARE: finished
0130 02:58:02.618004 (+ 105us) write_op.cc:270] Start()
0130 02:58:02.618079 (+ 75us) write_op.cc:276] Timestamp: P: 1769741882617960 usec, L: 0
0130 02:58:02.618093 (+ 14us) op_driver.cc:348] REPLICATION: starting
0130 02:58:02.618595 (+ 502us) log.cc:844] Serialized 1293 byte log entry
0130 02:58:02.628782 (+ 10187us) op_driver.cc:464] REPLICATION: finished
0130 02:58:02.629401 (+ 619us) write_op.cc:301] APPLY: starting
0130 02:58:02.629454 (+ 53us) tablet.cc:1365] starting BulkCheckPresence
0130 02:58:02.629567 (+ 113us) tablet.cc:1368] finished BulkCheckPresence
0130 02:58:02.629657 (+ 90us) tablet.cc:1370] starting ApplyRowOperation cycle
0130 02:58:02.629978 (+ 321us) tablet.cc:1381] finished ApplyRowOperation cycle
0130 02:58:02.630116 (+ 138us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=3
0130 02:58:02.630155 (+ 39us) write_op.cc:312] APPLY: finished
0130 02:58:02.630345 (+ 190us) log.cc:844] Serialized 35 byte log entry
0130 02:58:02.631107 (+ 762us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:58:02.631177 (+ 70us) write_op.cc:454] Released schema lock
0130 02:58:02.631209 (+ 32us) write_op.cc:341] FINISH: Updating metrics
0130 02:58:02.631408 (+ 199us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:58:02.631429 (+ 21us) write_op.cc:454] Released schema lock
Related trace 'op':
0130 02:58:03.212017 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0130 02:58:03.212362 (+ 345us) write_op.cc:432] Acquiring schema lock in shared mode
0130 02:58:03.212383 (+ 21us) write_op.cc:435] Acquired schema lock
0130 02:58:03.212394 (+ 11us) tablet.cc:660] Decoding operations
0130 02:58:03.212627 (+ 233us) write_op.cc:620] Acquiring the partition lock for write op
0130 02:58:03.212661 (+ 34us) write_op.cc:641] Partition lock acquired for write op
0130 02:58:03.212682 (+ 21us) tablet.cc:683] Acquiring locks for 1 operations
0130 02:58:03.212811 (+ 129us) tablet.cc:699] Row locks acquired
0130 02:58:03.212823 (+ 12us) write_op.cc:260] PREPARE: finished
0130 02:58:03.212920 (+ 97us) write_op.cc:270] Start()
0130 02:58:03.213007 (+ 87us) write_op.cc:276] Timestamp: P: 1769741883212859 usec, L: 0
0130 02:58:03.213021 (+ 14us) op_driver.cc:348] REPLICATION: starting
0130 02:58:03.213448 (+ 427us) log.cc:844] Serialized 356 byte log entry
0130 02:58:03.339707 (+126259us) op_driver.cc:464] REPLICATION: finished
0130 02:58:03.343807 (+ 4100us) write_op.cc:301] APPLY: starting
0130 02:58:03.343860 (+ 53us) tablet.cc:1365] starting BulkCheckPresence
0130 02:58:03.343955 (+ 95us) tablet.cc:1368] finished BulkCheckPresence
0130 02:58:03.344057 (+ 102us) tablet.cc:1370] starting ApplyRowOperation cycle
0130 02:58:03.344206 (+ 149us) tablet.cc:1381] finished ApplyRowOperation cycle
0130 02:58:03.344333 (+ 127us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0130 02:58:03.344376 (+ 43us) write_op.cc:312] APPLY: finished
0130 02:58:03.344627 (+ 251us) log.cc:844] Serialized 23 byte log entry
0130 02:58:03.390486 (+ 45859us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:58:03.390549 (+ 63us) write_op.cc:454] Released schema lock
0130 02:58:03.390601 (+ 52us) write_op.cc:341] FINISH: Updating metrics
0130 02:58:03.390958 (+ 357us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:58:03.390984 (+ 26us) write_op.cc:454] Released schema lock
Related trace 'op':
0130 02:58:03.871133 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0130 02:58:03.871518 (+ 385us) write_op.cc:432] Acquiring schema lock in shared mode
0130 02:58:03.871541 (+ 23us) write_op.cc:435] Acquired schema lock
0130 02:58:03.871615 (+ 74us) tablet.cc:660] Decoding operations
0130 02:58:03.872005 (+ 390us) write_op.cc:620] Acquiring the partition lock for write op
0130 02:58:03.872042 (+ 37us) write_op.cc:641] Partition lock acquired for write op
0130 02:58:03.872065 (+ 23us) tablet.cc:683] Acquiring locks for 1 operations
0130 02:58:03.872201 (+ 136us) tablet.cc:699] Row locks acquired
0130 02:58:03.872213 (+ 12us) write_op.cc:260] PREPARE: finished
0130 02:58:03.872318 (+ 105us) write_op.cc:270] Start()
0130 02:58:03.872394 (+ 76us) write_op.cc:276] Timestamp: P: 1769741883872256 usec, L: 0
0130 02:58:03.872409 (+ 15us) op_driver.cc:348] REPLICATION: starting
0130 02:58:03.872953 (+ 544us) log.cc:844] Serialized 356 byte log entry
0130 02:58:03.945522 (+ 72569us) op_driver.cc:464] REPLICATION: finished
0130 02:58:03.946200 (+ 678us) write_op.cc:301] APPLY: starting
0130 02:58:03.946258 (+ 58us) tablet.cc:1365] starting BulkCheckPresence
0130 02:58:03.946345 (+ 87us) tablet.cc:1368] finished BulkCheckPresence
0130 02:58:03.946447 (+ 102us) tablet.cc:1370] starting ApplyRowOperation cycle
0130 02:58:03.946645 (+ 198us) tablet.cc:1381] finished ApplyRowOperation cycle
0130 02:58:03.946769 (+ 124us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0130 02:58:03.946812 (+ 43us) write_op.cc:312] APPLY: finished
0130 02:58:03.946985 (+ 173us) log.cc:844] Serialized 23 byte log entry
0130 02:58:03.947127 (+ 142us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:58:03.947186 (+ 59us) write_op.cc:454] Released schema lock
0130 02:58:03.947216 (+ 30us) write_op.cc:341] FINISH: Updating metrics
0130 02:58:03.947414 (+ 198us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:58:03.947438 (+ 24us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":22558,"reported_tablets":23,"child_traces":[["op",{"apply.queue_time_us":460,"apply.run_cpu_time_us":1370,"apply.run_wall_time_us":1981,"num_ops":3,"prepare.queue_time_us":12561,"prepare.run_cpu_time_us":2202,"prepare.run_wall_time_us":2203,"raft.queue_time_us":681,"raft.run_cpu_time_us":646,"raft.run_wall_time_us":647,"replication_time_us":10592}],["op",{"apply.queue_time_us":3956,"apply.run_cpu_time_us":1332,"apply.run_wall_time_us":47110,"mutex_wait_us":45018,"num_ops":1,"prepare.queue_time_us":74735,"prepare.run_cpu_time_us":1689,"prepare.run_wall_time_us":2639,"raft.queue_time_us":69,"raft.run_cpu_time_us":10,"raft.run_wall_time_us":11,"replication_time_us":126623,"thread_start_us":5456,"threads_started":1}],["op",{"apply.queue_time_us":526,"apply.run_cpu_time_us":1188,"apply.run_wall_time_us":1187,"num_ops":1,"prepare.queue_time_us":5407,"prepare.run_cpu_time_us":1912,"prepare.run_wall_time_us":2870,"replication_time_us":73022}]]}
I20260130 02:58:04.486217 19861 raft_consensus.cc:515] T 5ca9919f80954e74a4d91c94b74de0e4 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:04.487752 19861 leader_election.cc:290] T 5ca9919f80954e74a4d91c94b74de0e4 P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d870164263bb44b6b780a557daea3656 (127.18.100.67:39571), 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767)
I20260130 02:58:04.488198 19840 tablet_bootstrap.cc:492] T 91f954f3ef4244c9a575a7421a7e75eb P 78c2c46c7520492db35d51c244fb4f74: No bootstrap required, opened a new log
I20260130 02:58:04.488361 19807 ts_tablet_manager.cc:1403] T a90e92d569254bbf8a5fbf286baf5cce P 371a36d219164974abdf848841528152: Time spent bootstrapping tablet: real 0.235s user 0.008s sys 0.023s
I20260130 02:58:04.488597 19840 ts_tablet_manager.cc:1403] T 91f954f3ef4244c9a575a7421a7e75eb P 78c2c46c7520492db35d51c244fb4f74: Time spent bootstrapping tablet: real 0.400s user 0.028s sys 0.016s
I20260130 02:58:04.488880 19376 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5ca9919f80954e74a4d91c94b74de0e4" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74" is_pre_election: true
I20260130 02:58:04.489459 19376 raft_consensus.cc:2468] T 5ca9919f80954e74a4d91c94b74de0e4 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 371a36d219164974abdf848841528152 in term 0.
I20260130 02:58:04.490787 19527 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5ca9919f80954e74a4d91c94b74de0e4" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
I20260130 02:58:04.495209 19807 raft_consensus.cc:359] T a90e92d569254bbf8a5fbf286baf5cce P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:04.495807 19807 raft_consensus.cc:385] T a90e92d569254bbf8a5fbf286baf5cce P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:04.496034 19807 raft_consensus.cc:740] T a90e92d569254bbf8a5fbf286baf5cce P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 371a36d219164974abdf848841528152, State: Initialized, Role: FOLLOWER
I20260130 02:58:04.496547 19807 consensus_queue.cc:260] T a90e92d569254bbf8a5fbf286baf5cce P 371a36d219164974abdf848841528152 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:04.350406 20157 raft_consensus.cc:515] T 5ca9919f80954e74a4d91c94b74de0e4 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:04.499779 20157 leader_election.cc:290] T 5ca9919f80954e74a4d91c94b74de0e4 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767)
I20260130 02:58:04.500443 19223 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5ca9919f80954e74a4d91c94b74de0e4" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152" is_pre_election: true
I20260130 02:58:04.500666 19376 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5ca9919f80954e74a4d91c94b74de0e4" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74" is_pre_election: true
I20260130 02:58:04.501029 19223 raft_consensus.cc:2468] T 5ca9919f80954e74a4d91c94b74de0e4 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d870164263bb44b6b780a557daea3656 in term 0.
I20260130 02:58:04.501219 19376 raft_consensus.cc:2468] T 5ca9919f80954e74a4d91c94b74de0e4 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d870164263bb44b6b780a557daea3656 in term 0.
I20260130 02:58:04.502252 19462 leader_election.cc:304] T 5ca9919f80954e74a4d91c94b74de0e4 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78c2c46c7520492db35d51c244fb4f74, d870164263bb44b6b780a557daea3656; no voters:
I20260130 02:58:04.503005 19742 raft_consensus.cc:2804] T 5ca9919f80954e74a4d91c94b74de0e4 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260130 02:58:04.503288 19742 raft_consensus.cc:493] T 5ca9919f80954e74a4d91c94b74de0e4 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260130 02:58:04.503533 19742 raft_consensus.cc:3060] T 5ca9919f80954e74a4d91c94b74de0e4 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:04.511241 19314 leader_election.cc:304] T 481bd4cca6ee4c178873bf12392e3441 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 371a36d219164974abdf848841528152, 78c2c46c7520492db35d51c244fb4f74; no voters:
W20260130 02:58:04.512444 19159 consensus_peers.cc:597] T bd43c5d148514836869a8ac648c8ad11 P 371a36d219164974abdf848841528152 -> Peer 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767): Couldn't send request to peer 78c2c46c7520492db35d51c244fb4f74. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: bd43c5d148514836869a8ac648c8ad11. This is attempt 1: this message will repeat every 5th retry.
I20260130 02:58:04.512939 19159 leader_election.cc:304] T 5ca9919f80954e74a4d91c94b74de0e4 P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 371a36d219164974abdf848841528152, 78c2c46c7520492db35d51c244fb4f74; no voters:
I20260130 02:58:04.514016 19861 raft_consensus.cc:2804] T 5ca9919f80954e74a4d91c94b74de0e4 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260130 02:58:04.514293 19861 raft_consensus.cc:493] T 5ca9919f80954e74a4d91c94b74de0e4 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260130 02:58:04.514551 19861 raft_consensus.cc:3060] T 5ca9919f80954e74a4d91c94b74de0e4 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:04.518283 19807 ts_tablet_manager.cc:1434] T a90e92d569254bbf8a5fbf286baf5cce P 371a36d219164974abdf848841528152: Time spent starting tablet: real 0.030s user 0.002s sys 0.002s
I20260130 02:58:04.527803 19840 raft_consensus.cc:359] T 91f954f3ef4244c9a575a7421a7e75eb P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:04.528416 19840 raft_consensus.cc:385] T 91f954f3ef4244c9a575a7421a7e75eb P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:04.528662 19840 raft_consensus.cc:740] T 91f954f3ef4244c9a575a7421a7e75eb P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78c2c46c7520492db35d51c244fb4f74, State: Initialized, Role: FOLLOWER
I20260130 02:58:04.528867 19861 raft_consensus.cc:515] T 5ca9919f80954e74a4d91c94b74de0e4 P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:04.530615 19861 leader_election.cc:290] T 5ca9919f80954e74a4d91c94b74de0e4 P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 election: Requested vote from peers d870164263bb44b6b780a557daea3656 (127.18.100.67:39571), 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767)
I20260130 02:58:04.544096 20009 raft_consensus.cc:2804] T 481bd4cca6ee4c178873bf12392e3441 P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Leader election won for term 1
I20260130 02:58:04.544626 20009 raft_consensus.cc:697] T 481bd4cca6ee4c178873bf12392e3441 P 78c2c46c7520492db35d51c244fb4f74 [term 1 LEADER]: Becoming Leader. State: Replica: 78c2c46c7520492db35d51c244fb4f74, State: Running, Role: LEADER
I20260130 02:58:04.545410 20009 consensus_queue.cc:237] T 481bd4cca6ee4c178873bf12392e3441 P 78c2c46c7520492db35d51c244fb4f74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:04.547544 19742 raft_consensus.cc:515] T 5ca9919f80954e74a4d91c94b74de0e4 P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:04.555433 18860 catalog_manager.cc:4985] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): tablet 5b2976108584478a98fd90fe6594af66 (table default.Table_a78ce7ab85924506b184fb74db3ed0c3 [id=5f4c327b216449a1a8b146d0f652de52]) successfully deleted
I20260130 02:58:04.565514 19377 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5ca9919f80954e74a4d91c94b74de0e4" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74"
I20260130 02:58:04.566080 19377 raft_consensus.cc:3060] T 5ca9919f80954e74a4d91c94b74de0e4 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:04.566661 19378 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5ca9919f80954e74a4d91c94b74de0e4" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74"
I20260130 02:58:04.549124 19742 leader_election.cc:290] T 5ca9919f80954e74a4d91c94b74de0e4 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 election: Requested vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767)
I20260130 02:58:04.573235 19224 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5ca9919f80954e74a4d91c94b74de0e4" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152"
I20260130 02:58:04.574442 19224 raft_consensus.cc:2393] T 5ca9919f80954e74a4d91c94b74de0e4 P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate d870164263bb44b6b780a557daea3656 in current term 1: Already voted for candidate 371a36d219164974abdf848841528152 in this term.
W20260130 02:58:04.581866 18870 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ee7c8ff92ca9465fa18be6eae63d78ca consensus state has no leader
I20260130 02:58:04.582337 19840 consensus_queue.cc:260] T 91f954f3ef4244c9a575a7421a7e75eb P 78c2c46c7520492db35d51c244fb4f74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:04.590781 20009 raft_consensus.cc:493] T 111287b5557d4abda1c5cd560fb0d019 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:04.591482 19527 raft_consensus.cc:2393] T 5ca9919f80954e74a4d91c94b74de0e4 P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 371a36d219164974abdf848841528152 in current term 1: Already voted for candidate d870164263bb44b6b780a557daea3656 in this term.
I20260130 02:58:04.591245 20009 raft_consensus.cc:515] T 111287b5557d4abda1c5cd560fb0d019 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:04.593779 20009 leader_election.cc:290] T 111287b5557d4abda1c5cd560fb0d019 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), d870164263bb44b6b780a557daea3656 (127.18.100.67:39571)
I20260130 02:58:04.594449 19224 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "111287b5557d4abda1c5cd560fb0d019" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152" is_pre_election: true
W20260130 02:58:04.595515 19314 leader_election.cc:343] T 111287b5557d4abda1c5cd560fb0d019 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:04.595907 19377 raft_consensus.cc:2468] T 5ca9919f80954e74a4d91c94b74de0e4 P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 371a36d219164974abdf848841528152 in term 1.
I20260130 02:58:04.597465 20153 tablet.cc:1722] T 38a7a10e36874183bf9fe55a38d829f6 P d870164263bb44b6b780a557daea3656: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20260130 02:58:04.599869 19840 ts_tablet_manager.cc:1434] T 91f954f3ef4244c9a575a7421a7e75eb P 78c2c46c7520492db35d51c244fb4f74: Time spent starting tablet: real 0.111s user 0.005s sys 0.001s
I20260130 02:58:04.600745 19840 tablet_bootstrap.cc:492] T 0a763e48583c4e748a46a325746ce69a P 78c2c46c7520492db35d51c244fb4f74: Bootstrap starting.
I20260130 02:58:04.602620 19625 tablet_bootstrap.cc:492] T 2ca0df71836145748cada4e4559a5656 P d870164263bb44b6b780a557daea3656: No bootstrap required, opened a new log
I20260130 02:58:04.603067 19625 ts_tablet_manager.cc:1403] T 2ca0df71836145748cada4e4559a5656 P d870164263bb44b6b780a557daea3656: Time spent bootstrapping tablet: real 0.770s user 0.018s sys 0.012s
I20260130 02:58:04.603925 18866 master_service.cc:954] ReplaceTablet: received request to replace tablet c3fce8cc99fb451ebd532fa37390e862 from {username='slave'} at 127.0.0.1:50284
I20260130 02:58:04.604923 19159 leader_election.cc:304] T 5ca9919f80954e74a4d91c94b74de0e4 P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 371a36d219164974abdf848841528152, 78c2c46c7520492db35d51c244fb4f74; no voters:
I20260130 02:58:04.608721 19462 leader_election.cc:304] T 5ca9919f80954e74a4d91c94b74de0e4 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d870164263bb44b6b780a557daea3656; no voters: 371a36d219164974abdf848841528152, 78c2c46c7520492db35d51c244fb4f74
I20260130 02:58:04.549823 19499 tablet_service.cc:1552] Processing DeleteTablet for tablet 7d96e93d95f14af79cfb268626559b1e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:58:04 UTC) from {username='slave'} at 127.0.0.1:59068
I20260130 02:58:04.548475 19502 tablet_service.cc:1552] Processing DeleteTablet for tablet f54addb1a4304ce69c711ae0cc3b452c with delete_type TABLET_DATA_DELETED (replaced by tablet 7783db3568f4434e91e9e456799cc312) from {username='slave'} at 127.0.0.1:59068
I20260130 02:58:04.613757 19523 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5ca9919f80954e74a4d91c94b74de0e4" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656"
I20260130 02:58:04.614600 19523 raft_consensus.cc:2393] T 5ca9919f80954e74a4d91c94b74de0e4 P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 371a36d219164974abdf848841528152 in current term 1: Already voted for candidate d870164263bb44b6b780a557daea3656 in this term.
I20260130 02:58:04.616797 18860 catalog_manager.cc:4985] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767): tablet a743a82540014c85aeab0e9d0994b0dc (table default.Table_71ecfab2cca249b49070a8340586ce5e [id=19941bcff5454783adb47df8dd77f352]) successfully deleted
W20260130 02:58:04.617436 18860 catalog_manager.cc:4965] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767): delete failed for tablet f54addb1a4304ce69c711ae0cc3b452c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f54addb1a4304ce69c711ae0cc3b452c already in progress: creating tablet
I20260130 02:58:04.618525 18873 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50284:
table { table_name: "default.Table_081113b62fb743189a6e04bf85dcc5fa" } modify_external_catalogs: true
I20260130 02:58:04.619072 18873 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50284:
table { table_name: "default.Table_081113b62fb743189a6e04bf85dcc5fa" } modify_external_catalogs: true
I20260130 02:58:04.619333 20089 raft_consensus.cc:493] T 730bd1d7d15b4cbe945b3ef03b2455f7 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:04.619761 20089 raft_consensus.cc:515] T 730bd1d7d15b4cbe945b3ef03b2455f7 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
W20260130 02:58:04.620357 18860 catalog_manager.cc:4952] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767): delete failed for tablet ceda64471a92476f9ba183d2beab30d1 because the tablet was not found. No further retry: Not found: Tablet not found: ceda64471a92476f9ba183d2beab30d1
W20260130 02:58:04.620916 18860 catalog_manager.cc:4952] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767): delete failed for tablet 7a902d0d571a42bb82ebffd48f7eca3f because the tablet was not found. No further retry: Not found: Tablet not found: 7a902d0d571a42bb82ebffd48f7eca3f
W20260130 02:58:04.621415 18860 catalog_manager.cc:4952] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767): delete failed for tablet 37470ce4356f4542b621e21698551209 because the tablet was not found. No further retry: Not found: Tablet not found: 37470ce4356f4542b621e21698551209
I20260130 02:58:04.551719 19526 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "481bd4cca6ee4c178873bf12392e3441" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656"
W20260130 02:58:04.621928 18860 catalog_manager.cc:4952] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767): delete failed for tablet 5b2976108584478a98fd90fe6594af66 because the tablet was not found. No further retry: Not found: Tablet not found: 5b2976108584478a98fd90fe6594af66
I20260130 02:58:04.622614 20152 raft_consensus.cc:2749] T 5ca9919f80954e74a4d91c94b74de0e4 P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260130 02:58:04.623042 20089 leader_election.cc:290] T 730bd1d7d15b4cbe945b3ef03b2455f7 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), d870164263bb44b6b780a557daea3656 (127.18.100.67:39571)
I20260130 02:58:04.625466 18870 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50284:
name: "default.Table_6b60df181f10452ba3f2e20e847bf502"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260130 02:58:04.629470 18870 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6b60df181f10452ba3f2e20e847bf502 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260130 02:58:04.631502 19526 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "111287b5557d4abda1c5cd560fb0d019" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
I20260130 02:58:04.632109 19840 tablet_bootstrap.cc:654] T 0a763e48583c4e748a46a325746ce69a P 78c2c46c7520492db35d51c244fb4f74: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:04.632606 19528 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "730bd1d7d15b4cbe945b3ef03b2455f7" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
I20260130 02:58:04.636940 18874 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50284:
name: "default.Table_21178f657dca43a1912ce596834d1618"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260130 02:58:04.639053 19313 leader_election.cc:343] T 481bd4cca6ee4c178873bf12392e3441 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): Illegal state: must be running to vote when last-logged opid is not known
W20260130 02:58:04.642283 19313 leader_election.cc:343] T 111287b5557d4abda1c5cd560fb0d019 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:04.643280 19313 leader_election.cc:304] T 111287b5557d4abda1c5cd560fb0d019 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78c2c46c7520492db35d51c244fb4f74; no voters: 371a36d219164974abdf848841528152, d870164263bb44b6b780a557daea3656
I20260130 02:58:04.644013 20120 raft_consensus.cc:2749] T 111287b5557d4abda1c5cd560fb0d019 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260130 02:58:04.645295 19313 leader_election.cc:343] T 730bd1d7d15b4cbe945b3ef03b2455f7 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): Illegal state: must be running to vote when last-logged opid is not known
W20260130 02:58:04.638370 18874 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_21178f657dca43a1912ce596834d1618 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260130 02:58:04.622426 18860 catalog_manager.cc:4952] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767): delete failed for tablet d701152960b14396a6ac1bd10480d2b6 because the tablet was not found. No further retry: Not found: Tablet not found: d701152960b14396a6ac1bd10480d2b6
W20260130 02:58:04.653394 18860 catalog_manager.cc:4952] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767): delete failed for tablet b078934be4bb4fd1915c691eeea9d00f because the tablet was not found. No further retry: Not found: Tablet not found: b078934be4bb4fd1915c691eeea9d00f
W20260130 02:58:04.654225 18860 catalog_manager.cc:4965] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767): delete failed for tablet 7d96e93d95f14af79cfb268626559b1e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7d96e93d95f14af79cfb268626559b1e already in progress: creating tablet
I20260130 02:58:04.657819 19840 tablet_bootstrap.cc:492] T 0a763e48583c4e748a46a325746ce69a P 78c2c46c7520492db35d51c244fb4f74: No bootstrap required, opened a new log
I20260130 02:58:04.658226 19840 ts_tablet_manager.cc:1403] T 0a763e48583c4e748a46a325746ce69a P 78c2c46c7520492db35d51c244fb4f74: Time spent bootstrapping tablet: real 0.058s user 0.008s sys 0.000s
I20260130 02:58:04.662210 20157 consensus_queue.cc:1048] T 7d76046ee6c8420fa4964decbe8209c1 P d870164263bb44b6b780a557daea3656 [LEADER]: Connected to new peer: Peer: permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260130 02:58:04.663386 20164 tablet.cc:1722] T 38a7a10e36874183bf9fe55a38d829f6 P 78c2c46c7520492db35d51c244fb4f74: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20260130 02:58:04.669165 18866 catalog_manager.cc:5941] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4: Sending DeleteTablet for 3 replicas of tablet c3fce8cc99fb451ebd532fa37390e862
I20260130 02:58:04.674005 18866 catalog_manager.cc:6724] ReplaceTablet: tablet c3fce8cc99fb451ebd532fa37390e862 deleted and replaced by tablet 78e9bc3e55fe4e4a98e8eb01da59d365
I20260130 02:58:04.674639 19840 raft_consensus.cc:359] T 0a763e48583c4e748a46a325746ce69a P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:04.674995 18866 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50284:
table { table_name: "default.Table_f2f790e8e46840ecb199c34fae899ada" } modify_external_catalogs: true
I20260130 02:58:04.675295 19840 raft_consensus.cc:385] T 0a763e48583c4e748a46a325746ce69a P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:04.675452 18866 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50284:
table { table_name: "default.Table_f2f790e8e46840ecb199c34fae899ada" } modify_external_catalogs: true
I20260130 02:58:04.675542 19840 raft_consensus.cc:740] T 0a763e48583c4e748a46a325746ce69a P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78c2c46c7520492db35d51c244fb4f74, State: Initialized, Role: FOLLOWER
I20260130 02:58:04.677565 19502 tablet_service.cc:1552] Processing DeleteTablet for tablet c3fce8cc99fb451ebd532fa37390e862 with delete_type TABLET_DATA_DELETED (replaced by tablet 78e9bc3e55fe4e4a98e8eb01da59d365) from {username='slave'} at 127.0.0.1:59068
I20260130 02:58:04.687463 19358 tablet_service.cc:1552] Processing DeleteTablet for tablet c3fce8cc99fb451ebd532fa37390e862 with delete_type TABLET_DATA_DELETED (replaced by tablet 78e9bc3e55fe4e4a98e8eb01da59d365) from {username='slave'} at 127.0.0.1:60110
I20260130 02:58:04.689559 19758 tablet_replica.cc:333] T c3fce8cc99fb451ebd532fa37390e862 P 78c2c46c7520492db35d51c244fb4f74: stopping tablet replica
I20260130 02:58:04.690402 19758 raft_consensus.cc:2243] T c3fce8cc99fb451ebd532fa37390e862 P 78c2c46c7520492db35d51c244fb4f74 [term 1 LEADER]: Raft consensus shutting down.
I20260130 02:58:04.691236 19758 raft_consensus.cc:2272] T c3fce8cc99fb451ebd532fa37390e862 P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260130 02:58:04.693030 19758 ts_tablet_manager.cc:1916] T c3fce8cc99fb451ebd532fa37390e862 P 78c2c46c7520492db35d51c244fb4f74: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260130 02:58:04.701529 18874 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50284:
table { table_name: "default.Table_5588d44386964492b3b8a4f7bd19b520" } new_table_name: "default.Table_dee034aec0b24982b8dc2a3fa1b326f1" modify_external_catalogs: true
I20260130 02:58:04.716799 19840 consensus_queue.cc:260] T 0a763e48583c4e748a46a325746ce69a P 78c2c46c7520492db35d51c244fb4f74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:04.720520 19840 ts_tablet_manager.cc:1434] T 0a763e48583c4e748a46a325746ce69a P 78c2c46c7520492db35d51c244fb4f74: Time spent starting tablet: real 0.062s user 0.007s sys 0.000s
I20260130 02:58:04.721374 19840 tablet_bootstrap.cc:492] T 341041cb0be2456e88403367c0814704 P 78c2c46c7520492db35d51c244fb4f74: Bootstrap starting.
I20260130 02:58:04.727227 18870 master_service.cc:954] ReplaceTablet: received request to replace tablet df2c8989d6c1480c97baa23db01141eb from {username='slave'} at 127.0.0.1:50284
I20260130 02:58:04.519138 19807 tablet_bootstrap.cc:492] T 534dc31a9b7e492c82f3672c75742e6c P 371a36d219164974abdf848841528152: Bootstrap starting.
I20260130 02:58:04.608964 19754 ts_tablet_manager.cc:1929] T b3f3177306914e3c8b253db54ee30fb6 P 371a36d219164974abdf848841528152: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260130 02:58:04.622287 19224 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "730bd1d7d15b4cbe945b3ef03b2455f7" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152" is_pre_election: true
I20260130 02:58:04.741130 19754 log.cc:1199] T b3f3177306914e3c8b253db54ee30fb6 P 371a36d219164974abdf848841528152: Deleting WAL directory at /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-0/wal/wals/b3f3177306914e3c8b253db54ee30fb6
I20260130 02:58:04.673199 19199 tablet_service.cc:1552] Processing DeleteTablet for tablet c3fce8cc99fb451ebd532fa37390e862 with delete_type TABLET_DATA_DELETED (replaced by tablet 78e9bc3e55fe4e4a98e8eb01da59d365) from {username='slave'} at 127.0.0.1:42870
I20260130 02:58:04.698953 19861 raft_consensus.cc:493] T a90e92d569254bbf8a5fbf286baf5cce P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:04.741995 19754 ts_tablet_manager.cc:1950] T b3f3177306914e3c8b253db54ee30fb6 P 371a36d219164974abdf848841528152: Deleting consensus metadata
I20260130 02:58:04.742210 19861 raft_consensus.cc:515] T a90e92d569254bbf8a5fbf286baf5cce P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:04.743794 19861 leader_election.cc:290] T a90e92d569254bbf8a5fbf286baf5cce P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d870164263bb44b6b780a557daea3656 (127.18.100.67:39571), 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767)
W20260130 02:58:04.744410 19314 leader_election.cc:343] T 730bd1d7d15b4cbe945b3ef03b2455f7 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:04.744799 19314 leader_election.cc:304] T 730bd1d7d15b4cbe945b3ef03b2455f7 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78c2c46c7520492db35d51c244fb4f74; no voters: 371a36d219164974abdf848841528152, d870164263bb44b6b780a557daea3656
I20260130 02:58:04.746387 20009 raft_consensus.cc:2749] T 730bd1d7d15b4cbe945b3ef03b2455f7 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260130 02:58:04.750128 18861 catalog_manager.cc:4985] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021): tablet b3f3177306914e3c8b253db54ee30fb6 (table default.Table_87dbbd65d92043b0b91c801517e56dfe [id=adced31928ba4204b79e3b43f8b737ee]) successfully deleted
I20260130 02:58:04.751793 19758 ts_tablet_manager.cc:1929] T c3fce8cc99fb451ebd532fa37390e862 P 78c2c46c7520492db35d51c244fb4f74: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260130 02:58:04.752449 19754 tablet_replica.cc:333] T a743a82540014c85aeab0e9d0994b0dc P 371a36d219164974abdf848841528152: stopping tablet replica
I20260130 02:58:04.753141 19754 raft_consensus.cc:2243] T a743a82540014c85aeab0e9d0994b0dc P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260130 02:58:04.753572 19754 raft_consensus.cc:2272] T a743a82540014c85aeab0e9d0994b0dc P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260130 02:58:04.755993 19807 tablet_bootstrap.cc:654] T 534dc31a9b7e492c82f3672c75742e6c P 371a36d219164974abdf848841528152: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:04.760834 19754 ts_tablet_manager.cc:1916] T a743a82540014c85aeab0e9d0994b0dc P 371a36d219164974abdf848841528152: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260130 02:58:04.761968 19758 log.cc:1199] T c3fce8cc99fb451ebd532fa37390e862 P 78c2c46c7520492db35d51c244fb4f74: Deleting WAL directory at /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-1/wal/wals/c3fce8cc99fb451ebd532fa37390e862
I20260130 02:58:04.766191 19758 ts_tablet_manager.cc:1950] T c3fce8cc99fb451ebd532fa37390e862 P 78c2c46c7520492db35d51c244fb4f74: Deleting consensus metadata
I20260130 02:58:04.768602 20149 raft_consensus.cc:493] T 37470ce4356f4542b621e21698551209 P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 371a36d219164974abdf848841528152)
I20260130 02:58:04.765414 19840 tablet_bootstrap.cc:654] T 341041cb0be2456e88403367c0814704 P 78c2c46c7520492db35d51c244fb4f74: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:04.769161 20149 raft_consensus.cc:515] T 37470ce4356f4542b621e21698551209 P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
W20260130 02:58:04.772796 19159 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.100.66:40767, user_credentials={real_user=slave}} blocked reactor thread for 167936us
I20260130 02:58:04.774171 19947 ts_tablet_manager.cc:1916] T d701152960b14396a6ac1bd10480d2b6 P d870164263bb44b6b780a557daea3656: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260130 02:58:04.775092 19861 raft_consensus.cc:2804] T 5ca9919f80954e74a4d91c94b74de0e4 P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Leader election won for term 1
I20260130 02:58:04.775504 19861 raft_consensus.cc:697] T 5ca9919f80954e74a4d91c94b74de0e4 P 371a36d219164974abdf848841528152 [term 1 LEADER]: Becoming Leader. State: Replica: 371a36d219164974abdf848841528152, State: Running, Role: LEADER
I20260130 02:58:04.776141 19861 consensus_queue.cc:237] T 5ca9919f80954e74a4d91c94b74de0e4 P 371a36d219164974abdf848841528152 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
W20260130 02:58:04.781637 18874 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a8a5038e9def44eb8b3b1e4aeae25652 has no consensus state
I20260130 02:58:04.783440 18870 catalog_manager.cc:5941] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4: Sending DeleteTablet for 3 replicas of tablet df2c8989d6c1480c97baa23db01141eb
I20260130 02:58:04.784608 18870 catalog_manager.cc:6724] ReplaceTablet: tablet df2c8989d6c1480c97baa23db01141eb deleted and replaced by tablet e2aaaf00f53d418b87d5c7af286e4d84
I20260130 02:58:04.784834 20115 raft_consensus.cc:493] T 2ca0df71836145748cada4e4559a5656 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:04.785254 20115 raft_consensus.cc:515] T 2ca0df71836145748cada4e4559a5656 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:04.787042 20115 leader_election.cc:290] T 2ca0df71836145748cada4e4559a5656 P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d870164263bb44b6b780a557daea3656 (127.18.100.67:39571), 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767)
I20260130 02:58:04.787556 19199 tablet_service.cc:1552] Processing DeleteTablet for tablet df2c8989d6c1480c97baa23db01141eb with delete_type TABLET_DATA_DELETED (replaced by tablet e2aaaf00f53d418b87d5c7af286e4d84) from {username='slave'} at 127.0.0.1:42870
I20260130 02:58:04.791407 19358 tablet_service.cc:1552] Processing DeleteTablet for tablet df2c8989d6c1480c97baa23db01141eb with delete_type TABLET_DATA_DELETED (replaced by tablet e2aaaf00f53d418b87d5c7af286e4d84) from {username='slave'} at 127.0.0.1:60110
I20260130 02:58:04.791736 19501 tablet_service.cc:1552] Processing DeleteTablet for tablet df2c8989d6c1480c97baa23db01141eb with delete_type TABLET_DATA_DELETED (replaced by tablet e2aaaf00f53d418b87d5c7af286e4d84) from {username='slave'} at 127.0.0.1:59068
I20260130 02:58:04.793524 18870 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50284:
table { table_name: "default.Table_42cd032296a0408797e7af3a27a103dc" } new_table_name: "default.Table_89d4117da99842719125411e30d10169" modify_external_catalogs: true
I20260130 02:58:04.797902 19758 tablet_replica.cc:333] T df2c8989d6c1480c97baa23db01141eb P 78c2c46c7520492db35d51c244fb4f74: stopping tablet replica
I20260130 02:58:04.798606 19758 raft_consensus.cc:2243] T df2c8989d6c1480c97baa23db01141eb P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260130 02:58:04.799078 19758 raft_consensus.cc:2272] T df2c8989d6c1480c97baa23db01141eb P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260130 02:58:04.799548 19625 raft_consensus.cc:359] T 2ca0df71836145748cada4e4559a5656 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:04.800346 19625 raft_consensus.cc:385] T 2ca0df71836145748cada4e4559a5656 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:04.800611 19625 raft_consensus.cc:740] T 2ca0df71836145748cada4e4559a5656 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d870164263bb44b6b780a557daea3656, State: Initialized, Role: FOLLOWER
I20260130 02:58:04.800745 19758 ts_tablet_manager.cc:1916] T df2c8989d6c1480c97baa23db01141eb P 78c2c46c7520492db35d51c244fb4f74: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260130 02:58:04.806056 20150 consensus_queue.cc:1048] T 7d76046ee6c8420fa4964decbe8209c1 P d870164263bb44b6b780a557daea3656 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260130 02:58:04.815388 19840 tablet_bootstrap.cc:492] T 341041cb0be2456e88403367c0814704 P 78c2c46c7520492db35d51c244fb4f74: No bootstrap required, opened a new log
I20260130 02:58:04.815793 19840 ts_tablet_manager.cc:1403] T 341041cb0be2456e88403367c0814704 P 78c2c46c7520492db35d51c244fb4f74: Time spent bootstrapping tablet: real 0.095s user 0.006s sys 0.005s
I20260130 02:58:04.817972 19840 raft_consensus.cc:359] T 341041cb0be2456e88403367c0814704 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:04.821121 19528 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a90e92d569254bbf8a5fbf286baf5cce" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
I20260130 02:58:04.823936 20149 leader_election.cc:290] T 37470ce4356f4542b621e21698551209 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767)
I20260130 02:58:04.818564 19840 raft_consensus.cc:385] T 341041cb0be2456e88403367c0814704 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:04.827651 19840 raft_consensus.cc:740] T 341041cb0be2456e88403367c0814704 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78c2c46c7520492db35d51c244fb4f74, State: Initialized, Role: FOLLOWER
I20260130 02:58:04.829026 19840 consensus_queue.cc:260] T 341041cb0be2456e88403367c0814704 P 78c2c46c7520492db35d51c244fb4f74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
W20260130 02:58:04.831925 19159 leader_election.cc:343] T a90e92d569254bbf8a5fbf286baf5cce P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:04.832685 19377 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a90e92d569254bbf8a5fbf286baf5cce" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74" is_pre_election: true
I20260130 02:58:04.840118 18860 catalog_manager.cc:4985] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767): tablet c3fce8cc99fb451ebd532fa37390e862 (table default.Table_5d29239ce1984d2fad02607a2bdd4dd0 [id=7e271b81bb064a949685b2411b485e3e]) successfully deleted
W20260130 02:58:04.825913 19463 leader_election.cc:343] T 37470ce4356f4542b621e21698551209 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Not found: Tablet not found: 37470ce4356f4542b621e21698551209
I20260130 02:58:04.860272 19358 tablet_service.cc:1505] Processing CreateTablet for tablet 43cec19c51ec4f05b63800c07f9534ef (DEFAULT_TABLE table=default.Table_8f632ff84d624b72a10cb90fb9e78636 [id=3cf79022ea954f6397f0a0497e6e6a6c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:04.861239 19358 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 43cec19c51ec4f05b63800c07f9534ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:04.864753 19625 consensus_queue.cc:260] T 2ca0df71836145748cada4e4559a5656 P d870164263bb44b6b780a557daea3656 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:04.866952 19625 ts_tablet_manager.cc:1434] T 2ca0df71836145748cada4e4559a5656 P d870164263bb44b6b780a557daea3656: Time spent starting tablet: real 0.264s user 0.004s sys 0.001s
W20260130 02:58:04.875545 19462 leader_election.cc:343] T 37470ce4356f4542b621e21698551209 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767): Not found: Tablet not found: 37470ce4356f4542b621e21698551209
I20260130 02:58:04.875923 19462 leader_election.cc:304] T 37470ce4356f4542b621e21698551209 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d870164263bb44b6b780a557daea3656; no voters: 371a36d219164974abdf848841528152, 78c2c46c7520492db35d51c244fb4f74
I20260130 02:58:04.876783 20149 raft_consensus.cc:2749] T 37470ce4356f4542b621e21698551209 P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260130 02:58:04.850615 19199 tablet_service.cc:1505] Processing CreateTablet for tablet 43cec19c51ec4f05b63800c07f9534ef (DEFAULT_TABLE table=default.Table_8f632ff84d624b72a10cb90fb9e78636 [id=3cf79022ea954f6397f0a0497e6e6a6c]), partition=RANGE (key) PARTITION UNBOUNDED
W20260130 02:58:04.877466 19159 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.100.67:39571, user_credentials={real_user=slave}} blocked reactor thread for 45575.7us
I20260130 02:58:04.877954 19199 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 43cec19c51ec4f05b63800c07f9534ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:04.859833 19198 tablet_service.cc:1505] Processing CreateTablet for tablet 8e57ae31ab3941f789ac69627384e3b0 (DEFAULT_TABLE table=default.Table_09baf25d18fa489e9af0d7e289070082 [id=8a9182f49e18489fafec8dca190c1b2f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:04.860445 19200 tablet_service.cc:1505] Processing CreateTablet for tablet dc09a0f0cecf49738bab04596944c0f3 (DEFAULT_TABLE table=default.Table_6a7faee4c75c4eb7862fb4c05afc17cb [id=676ece9dc36b418a8b1400e8406d6b98]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:04.864431 19202 tablet_service.cc:1505] Processing CreateTablet for tablet d55291d7cea64a0d9a345ce712fbfa80 (DEFAULT_TABLE table=default.Table_b85cb876f25740bfaaab01949ac26b38 [id=839e292e040c4aeb832f3fc0258c32ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:04.865365 19201 tablet_service.cc:1505] Processing CreateTablet for tablet d27ca7dd87f44102b7680e1f4a88954a (DEFAULT_TABLE table=default.Table_109c8b6c17d4442b939bde9441d0476f [id=dcf82ea15d6044c2b023f81f0a579ade]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:04.881345 19198 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8e57ae31ab3941f789ac69627384e3b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:04.887216 19201 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d27ca7dd87f44102b7680e1f4a88954a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:04.889102 19807 tablet_bootstrap.cc:492] T 534dc31a9b7e492c82f3672c75742e6c P 371a36d219164974abdf848841528152: No bootstrap required, opened a new log
I20260130 02:58:04.889532 19807 ts_tablet_manager.cc:1403] T 534dc31a9b7e492c82f3672c75742e6c P 371a36d219164974abdf848841528152: Time spent bootstrapping tablet: real 0.371s user 0.012s sys 0.000s
I20260130 02:58:04.891984 19807 raft_consensus.cc:359] T 534dc31a9b7e492c82f3672c75742e6c P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:04.892645 19807 raft_consensus.cc:385] T 534dc31a9b7e492c82f3672c75742e6c P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:04.892885 19807 raft_consensus.cc:740] T 534dc31a9b7e492c82f3672c75742e6c P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 371a36d219164974abdf848841528152, State: Initialized, Role: FOLLOWER
I20260130 02:58:04.892156 19200 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dc09a0f0cecf49738bab04596944c0f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:04.893461 19807 consensus_queue.cc:260] T 534dc31a9b7e492c82f3672c75742e6c P 371a36d219164974abdf848841528152 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:04.899075 19202 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d55291d7cea64a0d9a345ce712fbfa80. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:04.907950 19377 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ca0df71836145748cada4e4559a5656" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74" is_pre_election: true
I20260130 02:58:04.908587 19377 raft_consensus.cc:2468] T 2ca0df71836145748cada4e4559a5656 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 371a36d219164974abdf848841528152 in term 0.
I20260130 02:58:04.909545 19528 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ca0df71836145748cada4e4559a5656" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
I20260130 02:58:04.910094 19528 raft_consensus.cc:2468] T 2ca0df71836145748cada4e4559a5656 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 371a36d219164974abdf848841528152 in term 0.
I20260130 02:58:04.922935 19357 tablet_service.cc:1505] Processing CreateTablet for tablet 8e57ae31ab3941f789ac69627384e3b0 (DEFAULT_TABLE table=default.Table_09baf25d18fa489e9af0d7e289070082 [id=8a9182f49e18489fafec8dca190c1b2f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:04.923873 19357 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8e57ae31ab3941f789ac69627384e3b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:04.924983 19840 ts_tablet_manager.cc:1434] T 341041cb0be2456e88403367c0814704 P 78c2c46c7520492db35d51c244fb4f74: Time spent starting tablet: real 0.109s user 0.005s sys 0.001s
I20260130 02:58:04.925874 19840 tablet_bootstrap.cc:492] T a90e92d569254bbf8a5fbf286baf5cce P 78c2c46c7520492db35d51c244fb4f74: Bootstrap starting.
I20260130 02:58:04.927145 19501 tablet_service.cc:1505] Processing CreateTablet for tablet 43cec19c51ec4f05b63800c07f9534ef (DEFAULT_TABLE table=default.Table_8f632ff84d624b72a10cb90fb9e78636 [id=3cf79022ea954f6397f0a0497e6e6a6c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:04.928839 19498 tablet_service.cc:1505] Processing CreateTablet for tablet 8e57ae31ab3941f789ac69627384e3b0 (DEFAULT_TABLE table=default.Table_09baf25d18fa489e9af0d7e289070082 [id=8a9182f49e18489fafec8dca190c1b2f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:04.929881 19360 tablet_service.cc:1505] Processing CreateTablet for tablet dc09a0f0cecf49738bab04596944c0f3 (DEFAULT_TABLE table=default.Table_6a7faee4c75c4eb7862fb4c05afc17cb [id=676ece9dc36b418a8b1400e8406d6b98]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:04.930195 19501 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 43cec19c51ec4f05b63800c07f9534ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:04.930786 19360 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dc09a0f0cecf49738bab04596944c0f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:04.930989 19354 tablet_service.cc:1505] Processing CreateTablet for tablet d55291d7cea64a0d9a345ce712fbfa80 (DEFAULT_TABLE table=default.Table_b85cb876f25740bfaaab01949ac26b38 [id=839e292e040c4aeb832f3fc0258c32ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:04.931854 19354 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d55291d7cea64a0d9a345ce712fbfa80. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:04.932147 19502 tablet_service.cc:1505] Processing CreateTablet for tablet dc09a0f0cecf49738bab04596944c0f3 (DEFAULT_TABLE table=default.Table_6a7faee4c75c4eb7862fb4c05afc17cb [id=676ece9dc36b418a8b1400e8406d6b98]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:04.935729 19499 tablet_service.cc:1505] Processing CreateTablet for tablet d55291d7cea64a0d9a345ce712fbfa80 (DEFAULT_TABLE table=default.Table_b85cb876f25740bfaaab01949ac26b38 [id=839e292e040c4aeb832f3fc0258c32ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:04.936091 19352 tablet_service.cc:1505] Processing CreateTablet for tablet d27ca7dd87f44102b7680e1f4a88954a (DEFAULT_TABLE table=default.Table_109c8b6c17d4442b939bde9441d0476f [id=dcf82ea15d6044c2b023f81f0a579ade]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:04.936901 19502 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dc09a0f0cecf49738bab04596944c0f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:04.937549 19352 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d27ca7dd87f44102b7680e1f4a88954a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:04.945183 19204 tablet_service.cc:1505] Processing CreateTablet for tablet a8a5038e9def44eb8b3b1e4aeae25652 (DEFAULT_TABLE table=default.Table_dee034aec0b24982b8dc2a3fa1b326f1 [id=66033e75b3814a8584311cf7e4a07ad3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:04.946167 19204 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a8a5038e9def44eb8b3b1e4aeae25652. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:04.952800 19499 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d55291d7cea64a0d9a345ce712fbfa80. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:04.953086 19358 tablet_service.cc:1505] Processing CreateTablet for tablet a8a5038e9def44eb8b3b1e4aeae25652 (DEFAULT_TABLE table=default.Table_dee034aec0b24982b8dc2a3fa1b326f1 [id=66033e75b3814a8584311cf7e4a07ad3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:04.954066 19358 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a8a5038e9def44eb8b3b1e4aeae25652. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:04.954783 19498 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8e57ae31ab3941f789ac69627384e3b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:04.959687 19507 tablet_service.cc:1505] Processing CreateTablet for tablet d27ca7dd87f44102b7680e1f4a88954a (DEFAULT_TABLE table=default.Table_109c8b6c17d4442b939bde9441d0476f [id=dcf82ea15d6044c2b023f81f0a579ade]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:04.960593 19507 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d27ca7dd87f44102b7680e1f4a88954a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:04.965934 19758 ts_tablet_manager.cc:1929] T df2c8989d6c1480c97baa23db01141eb P 78c2c46c7520492db35d51c244fb4f74: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20260130 02:58:04.966291 19758 log.cc:1199] T df2c8989d6c1480c97baa23db01141eb P 78c2c46c7520492db35d51c244fb4f74: Deleting WAL directory at /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-1/wal/wals/df2c8989d6c1480c97baa23db01141eb
I20260130 02:58:04.972254 19505 tablet_service.cc:1505] Processing CreateTablet for tablet a8a5038e9def44eb8b3b1e4aeae25652 (DEFAULT_TABLE table=default.Table_dee034aec0b24982b8dc2a3fa1b326f1 [id=66033e75b3814a8584311cf7e4a07ad3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:04.973177 19505 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a8a5038e9def44eb8b3b1e4aeae25652. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:04.973747 19840 tablet_bootstrap.cc:654] T a90e92d569254bbf8a5fbf286baf5cce P 78c2c46c7520492db35d51c244fb4f74: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:04.975296 19506 tablet_service.cc:1505] Processing CreateTablet for tablet 3b02480d9c6f4328af83f7940716d698 (DEFAULT_TABLE table=default.Table_1e9e20a1bfec4bcf81f9eb6219c33d59 [id=f675b7a3ec744533903d8834eeed51d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:04.975999 19199 tablet_service.cc:1505] Processing CreateTablet for tablet 3b02480d9c6f4328af83f7940716d698 (DEFAULT_TABLE table=default.Table_1e9e20a1bfec4bcf81f9eb6219c33d59 [id=f675b7a3ec744533903d8834eeed51d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:04.976310 19506 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3b02480d9c6f4328af83f7940716d698. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:04.976434 18866 catalog_manager.cc:5941] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4: Sending DeleteTablet for 3 replicas of tablet b2ac91da01954c2390b6b7dcc06c394f
I20260130 02:58:04.976935 19199 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3b02480d9c6f4328af83f7940716d698. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:04.979095 19356 tablet_service.cc:1505] Processing CreateTablet for tablet 3b02480d9c6f4328af83f7940716d698 (DEFAULT_TABLE table=default.Table_1e9e20a1bfec4bcf81f9eb6219c33d59 [id=f675b7a3ec744533903d8834eeed51d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:04.980106 19356 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3b02480d9c6f4328af83f7940716d698. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:04.981048 19198 tablet_service.cc:1552] Processing DeleteTablet for tablet b2ac91da01954c2390b6b7dcc06c394f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:58:04 UTC) from {username='slave'} at 127.0.0.1:42870
I20260130 02:58:04.988379 18869 catalog_manager.cc:5746] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767) does not have the latest schema for tablet 38a7a10e36874183bf9fe55a38d829f6 (table default.Table_f359d249f8274f319f5ff4e4ea89dd22 [id=27adf356fd004597a54c7a9887dd3ca8]). Expected version 4 got 3
I20260130 02:58:04.989625 19758 ts_tablet_manager.cc:1950] T df2c8989d6c1480c97baa23db01141eb P 78c2c46c7520492db35d51c244fb4f74: Deleting consensus metadata
I20260130 02:58:04.992516 19198 tablet_service.cc:1505] Processing CreateTablet for tablet 16691c9315e94253bb9b3fb5058e5636 (DEFAULT_TABLE table=default.Table_4c2511f2cf3f40ee9c134faafc7c8383 [id=4f499357303d464aafc17679ffcaaa07]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:04.999435 18869 catalog_manager.cc:5746] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767) does not have the latest schema for tablet 7d76046ee6c8420fa4964decbe8209c1 (table default.Table_55e0ba23f44146339c7c6928440d4948 [id=4c70005d1005479fa6c3a6cb6f5ce658]). Expected version 1 got 0
I20260130 02:58:05.004284 19198 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 16691c9315e94253bb9b3fb5058e5636. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:05.004679 19205 tablet_service.cc:1505] Processing CreateTablet for tablet 753344737b5941d9a93b6779e72f63d4 (DEFAULT_TABLE table=default.Table_8896365e792f4d24b2c16c2d269b0a93 [id=44816e07033849479a7fea0f51d00fad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:05.005548 19205 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 753344737b5941d9a93b6779e72f63d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:05.030887 19357 tablet_service.cc:1552] Processing DeleteTablet for tablet b2ac91da01954c2390b6b7dcc06c394f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:58:04 UTC) from {username='slave'} at 127.0.0.1:60110
I20260130 02:58:05.031629 19500 tablet_service.cc:1552] Processing DeleteTablet for tablet b2ac91da01954c2390b6b7dcc06c394f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:58:04 UTC) from {username='slave'} at 127.0.0.1:59068
I20260130 02:58:05.031622 19197 tablet_service.cc:1505] Processing CreateTablet for tablet 4c139aa8cd184f488cec4e05ef7dcada (DEFAULT_TABLE table=default.Table_5c724fbcf445490089992ba7fe40dc3d [id=fcbd7e3a7fc64d89aa740494d7e1a0f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:05.032793 19197 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4c139aa8cd184f488cec4e05ef7dcada. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:05.044402 18860 catalog_manager.cc:4985] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767): tablet df2c8989d6c1480c97baa23db01141eb (table default.Table_e84c6277dfc944059f62cbe0d6b937ee [id=747af828147149efb7b7b8d33c49286d]) successfully deleted
I20260130 02:58:05.045027 19357 tablet_service.cc:1505] Processing CreateTablet for tablet 16691c9315e94253bb9b3fb5058e5636 (DEFAULT_TABLE table=default.Table_4c2511f2cf3f40ee9c134faafc7c8383 [id=4f499357303d464aafc17679ffcaaa07]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:05.045017 19500 tablet_service.cc:1505] Processing CreateTablet for tablet 16691c9315e94253bb9b3fb5058e5636 (DEFAULT_TABLE table=default.Table_4c2511f2cf3f40ee9c134faafc7c8383 [id=4f499357303d464aafc17679ffcaaa07]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:05.046027 19357 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 16691c9315e94253bb9b3fb5058e5636. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:05.046085 19500 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 16691c9315e94253bb9b3fb5058e5636. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:05.047719 19503 tablet_service.cc:1505] Processing CreateTablet for tablet 753344737b5941d9a93b6779e72f63d4 (DEFAULT_TABLE table=default.Table_8896365e792f4d24b2c16c2d269b0a93 [id=44816e07033849479a7fea0f51d00fad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:05.048717 19503 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 753344737b5941d9a93b6779e72f63d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:05.051199 19360 tablet_service.cc:1505] Processing CreateTablet for tablet 753344737b5941d9a93b6779e72f63d4 (DEFAULT_TABLE table=default.Table_8896365e792f4d24b2c16c2d269b0a93 [id=44816e07033849479a7fea0f51d00fad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:05.052139 19360 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 753344737b5941d9a93b6779e72f63d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:05.060771 18873 catalog_manager.cc:5941] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4: Sending DeleteTablet for 3 replicas of tablet 534dc31a9b7e492c82f3672c75742e6c
I20260130 02:58:05.070209 18873 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50284:
name: "default.Table_baaf376733384fdbbac49ec8f2357432"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260130 02:58:05.071784 18873 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_baaf376733384fdbbac49ec8f2357432 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260130 02:58:05.073930 19159 leader_election.cc:343] T a90e92d569254bbf8a5fbf286baf5cce P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:05.074333 19159 leader_election.cc:304] T a90e92d569254bbf8a5fbf286baf5cce P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 371a36d219164974abdf848841528152; no voters: 78c2c46c7520492db35d51c244fb4f74, d870164263bb44b6b780a557daea3656
W20260130 02:58:05.075500 18860 catalog_manager.cc:4965] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767): delete failed for tablet b2ac91da01954c2390b6b7dcc06c394f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b2ac91da01954c2390b6b7dcc06c394f already in progress: creating tablet
I20260130 02:58:05.103092 19807 ts_tablet_manager.cc:1434] T 534dc31a9b7e492c82f3672c75742e6c P 371a36d219164974abdf848841528152: Time spent starting tablet: real 0.213s user 0.005s sys 0.000s
I20260130 02:58:05.104151 19861 raft_consensus.cc:2749] T a90e92d569254bbf8a5fbf286baf5cce P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260130 02:58:05.104596 19159 leader_election.cc:304] T 2ca0df71836145748cada4e4559a5656 P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 371a36d219164974abdf848841528152, 78c2c46c7520492db35d51c244fb4f74; no voters:
I20260130 02:58:05.107218 19861 raft_consensus.cc:2804] T 2ca0df71836145748cada4e4559a5656 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260130 02:58:05.107512 19861 raft_consensus.cc:493] T 2ca0df71836145748cada4e4559a5656 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260130 02:58:05.107788 19861 raft_consensus.cc:3060] T 2ca0df71836145748cada4e4559a5656 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:05.113732 19861 raft_consensus.cc:515] T 2ca0df71836145748cada4e4559a5656 P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:05.115458 19861 leader_election.cc:290] T 2ca0df71836145748cada4e4559a5656 P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 election: Requested vote from peers d870164263bb44b6b780a557daea3656 (127.18.100.67:39571), 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767)
I20260130 02:58:05.118963 19807 tablet_bootstrap.cc:492] T 0a763e48583c4e748a46a325746ce69a P 371a36d219164974abdf848841528152: Bootstrap starting.
I20260130 02:58:05.122063 19504 tablet_service.cc:1505] Processing CreateTablet for tablet 4c139aa8cd184f488cec4e05ef7dcada (DEFAULT_TABLE table=default.Table_5c724fbcf445490089992ba7fe40dc3d [id=fcbd7e3a7fc64d89aa740494d7e1a0f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:05.122340 19355 tablet_service.cc:1552] Processing DeleteTablet for tablet 534dc31a9b7e492c82f3672c75742e6c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:58:04 UTC) from {username='slave'} at 127.0.0.1:60110
I20260130 02:58:05.123090 19758 tablet_replica.cc:333] T 534dc31a9b7e492c82f3672c75742e6c P 78c2c46c7520492db35d51c244fb4f74: stopping tablet replica
I20260130 02:58:05.123202 19504 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4c139aa8cd184f488cec4e05ef7dcada. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:05.123467 19358 tablet_service.cc:1505] Processing CreateTablet for tablet 0e8018c094024c4788c484e8f5466e00 (DEFAULT_TABLE table=default.Table_f4b80e13bbe24163a77efb7b60f5dc2c [id=43254e466e7b4a2fb59fbcec68cfde36]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:05.123795 19758 raft_consensus.cc:2243] T 534dc31a9b7e492c82f3672c75742e6c P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260130 02:58:05.124270 19758 raft_consensus.cc:2272] T 534dc31a9b7e492c82f3672c75742e6c P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260130 02:58:05.124377 19358 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0e8018c094024c4788c484e8f5466e00. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:05.125027 18874 catalog_manager.cc:5654] T 7d76046ee6c8420fa4964decbe8209c1 P d870164263bb44b6b780a557daea3656 reported cstate change: term changed from 0 to 1, leader changed from <none> to d870164263bb44b6b780a557daea3656 (127.18.100.67). New cstate: current_term: 1 leader_uuid: "d870164263bb44b6b780a557daea3656" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } health_report { overall_health: UNKNOWN } } }
I20260130 02:58:05.125972 19758 ts_tablet_manager.cc:1916] T 534dc31a9b7e492c82f3672c75742e6c P 78c2c46c7520492db35d51c244fb4f74: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260130 02:58:05.127316 19353 tablet_service.cc:1505] Processing CreateTablet for tablet 7caf7cbad5054396b5617094e4d536a0 (DEFAULT_TABLE table=default.Table_007e969f70d345a7bcb3dfbaabbc025d [id=77d4fa8705a3419cb0a84d3ca1bb4ce3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:05.128306 19353 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7caf7cbad5054396b5617094e4d536a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:05.129947 19508 tablet_service.cc:1505] Processing CreateTablet for tablet 0e8018c094024c4788c484e8f5466e00 (DEFAULT_TABLE table=default.Table_f4b80e13bbe24163a77efb7b60f5dc2c [id=43254e466e7b4a2fb59fbcec68cfde36]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:05.130935 19508 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0e8018c094024c4788c484e8f5466e00. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:05.132148 19497 tablet_service.cc:1552] Processing DeleteTablet for tablet 534dc31a9b7e492c82f3672c75742e6c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:58:04 UTC) from {username='slave'} at 127.0.0.1:59068
I20260130 02:58:05.134729 19754 ts_tablet_manager.cc:1929] T a743a82540014c85aeab0e9d0994b0dc P 371a36d219164974abdf848841528152: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260130 02:58:05.135169 19754 log.cc:1199] T a743a82540014c85aeab0e9d0994b0dc P 371a36d219164974abdf848841528152: Deleting WAL directory at /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-0/wal/wals/a743a82540014c85aeab0e9d0994b0dc
I20260130 02:58:05.135895 19754 ts_tablet_manager.cc:1950] T a743a82540014c85aeab0e9d0994b0dc P 371a36d219164974abdf848841528152: Deleting consensus metadata
I20260130 02:58:05.137156 19354 tablet_service.cc:1505] Processing CreateTablet for tablet 4c139aa8cd184f488cec4e05ef7dcada (DEFAULT_TABLE table=default.Table_5c724fbcf445490089992ba7fe40dc3d [id=fcbd7e3a7fc64d89aa740494d7e1a0f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:05.138123 19354 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4c139aa8cd184f488cec4e05ef7dcada. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:05.141688 20176 raft_consensus.cc:493] T 534dc31a9b7e492c82f3672c75742e6c P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:05.142168 20176 raft_consensus.cc:515] T 534dc31a9b7e492c82f3672c75742e6c P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:05.144076 20176 leader_election.cc:290] T 534dc31a9b7e492c82f3672c75742e6c P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d870164263bb44b6b780a557daea3656 (127.18.100.67:39571), 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767)
I20260130 02:58:05.145923 19198 tablet_service.cc:1552] Processing DeleteTablet for tablet 534dc31a9b7e492c82f3672c75742e6c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:58:04 UTC) from {username='slave'} at 127.0.0.1:42870
I20260130 02:58:05.146997 19200 tablet_service.cc:1505] Processing CreateTablet for tablet 7caf7cbad5054396b5617094e4d536a0 (DEFAULT_TABLE table=default.Table_007e969f70d345a7bcb3dfbaabbc025d [id=77d4fa8705a3419cb0a84d3ca1bb4ce3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:05.148097 19200 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7caf7cbad5054396b5617094e4d536a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:05.151683 19357 tablet_service.cc:1505] Processing CreateTablet for tablet ba91a508b1cb4430a4d8ca26946aecc8 (DEFAULT_TABLE table=default.Table_84fbe659a98c4a2e9f805bb1efc7b4de [id=780188e2335f43fbb11de8212eb38d56]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:05.151690 19497 tablet_service.cc:1505] Processing CreateTablet for tablet 7caf7cbad5054396b5617094e4d536a0 (DEFAULT_TABLE table=default.Table_007e969f70d345a7bcb3dfbaabbc025d [id=77d4fa8705a3419cb0a84d3ca1bb4ce3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:05.152843 19497 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7caf7cbad5054396b5617094e4d536a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:05.153175 19355 tablet_service.cc:1505] Processing CreateTablet for tablet 3389db58493448aea3cf00aca411047c (DEFAULT_TABLE table=default.Table_22d10332a83f4700ac21892bab67f46b [id=da3a9cb1f6d5419abc1afc93659b5324]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:05.154690 19351 tablet_service.cc:1505] Processing CreateTablet for tablet 4907294442c145c6b03eb1de057c66a8 (DEFAULT_TABLE table=default.Table_ea2e0aa925a0486bb292304ca21bb819 [id=18580b9c54e448d1bdaae84f697f7b0b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:05.156205 19350 tablet_service.cc:1505] Processing CreateTablet for tablet 9fe0448bcc4a4346b3c67fc3f626ed3c (DEFAULT_TABLE table=default.Table_5c1c447e85524ac5a61ffd8a48509d36 [id=70aaf21ad9bd4d959f46939c6b4a4880]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:05.157506 19496 tablet_service.cc:1505] Processing CreateTablet for tablet ba91a508b1cb4430a4d8ca26946aecc8 (DEFAULT_TABLE table=default.Table_84fbe659a98c4a2e9f805bb1efc7b4de [id=780188e2335f43fbb11de8212eb38d56]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:05.159080 20009 raft_consensus.cc:493] T 0a763e48583c4e748a46a325746ce69a P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:05.159333 19496 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ba91a508b1cb4430a4d8ca26946aecc8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:05.159469 20009 raft_consensus.cc:515] T 0a763e48583c4e748a46a325746ce69a P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:05.161223 20009 leader_election.cc:290] T 0a763e48583c4e748a46a325746ce69a P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), d870164263bb44b6b780a557daea3656 (127.18.100.67:39571)
I20260130 02:58:05.163378 19495 tablet_service.cc:1505] Processing CreateTablet for tablet 345eaaa5625c48d099280b7fbb2e514b (DEFAULT_TABLE table=default.Table_003c5fa197e648259fe978a5ee4b44c8 [id=7a2d5367224b492b844199088b01b149]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:05.164412 19807 tablet_bootstrap.cc:654] T 0a763e48583c4e748a46a325746ce69a P 371a36d219164974abdf848841528152: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:05.165019 19495 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 345eaaa5625c48d099280b7fbb2e514b. 1 dirs total, 0 dirs full, 0 dirs failed
W20260130 02:58:05.167212 19159 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.100.67:39571, user_credentials={real_user=slave}} blocked reactor thread for 55838.2us
I20260130 02:58:05.173933 19376 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ca0df71836145748cada4e4559a5656" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74"
I20260130 02:58:05.174189 19352 tablet_service.cc:1505] Processing CreateTablet for tablet 345eaaa5625c48d099280b7fbb2e514b (DEFAULT_TABLE table=default.Table_003c5fa197e648259fe978a5ee4b44c8 [id=7a2d5367224b492b844199088b01b149]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:05.174467 19376 raft_consensus.cc:3060] T 2ca0df71836145748cada4e4559a5656 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:05.174850 19754 tablet_replica.cc:333] T c3fce8cc99fb451ebd532fa37390e862 P 371a36d219164974abdf848841528152: stopping tablet replica
I20260130 02:58:05.175180 19197 tablet_service.cc:1505] Processing CreateTablet for tablet 0e8018c094024c4788c484e8f5466e00 (DEFAULT_TABLE table=default.Table_f4b80e13bbe24163a77efb7b60f5dc2c [id=43254e466e7b4a2fb59fbcec68cfde36]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:05.175259 19198 tablet_service.cc:1505] Processing CreateTablet for tablet ba91a508b1cb4430a4d8ca26946aecc8 (DEFAULT_TABLE table=default.Table_84fbe659a98c4a2e9f805bb1efc7b4de [id=780188e2335f43fbb11de8212eb38d56]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:05.175596 19754 raft_consensus.cc:2243] T c3fce8cc99fb451ebd532fa37390e862 P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260130 02:58:05.176628 19204 tablet_service.cc:1505] Processing CreateTablet for tablet 345eaaa5625c48d099280b7fbb2e514b (DEFAULT_TABLE table=default.Table_003c5fa197e648259fe978a5ee4b44c8 [id=7a2d5367224b492b844199088b01b149]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:05.178386 19199 tablet_service.cc:1505] Processing CreateTablet for tablet 9fe0448bcc4a4346b3c67fc3f626ed3c (DEFAULT_TABLE table=default.Table_5c1c447e85524ac5a61ffd8a48509d36 [id=70aaf21ad9bd4d959f46939c6b4a4880]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:05.178701 19380 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "534dc31a9b7e492c82f3672c75742e6c" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74" is_pre_election: true
I20260130 02:58:05.178416 19201 tablet_service.cc:1505] Processing CreateTablet for tablet 4907294442c145c6b03eb1de057c66a8 (DEFAULT_TABLE table=default.Table_ea2e0aa925a0486bb292304ca21bb819 [id=18580b9c54e448d1bdaae84f697f7b0b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:05.179018 19205 tablet_service.cc:1505] Processing CreateTablet for tablet 3389db58493448aea3cf00aca411047c (DEFAULT_TABLE table=default.Table_22d10332a83f4700ac21892bab67f46b [id=da3a9cb1f6d5419abc1afc93659b5324]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:05.126332 18874 catalog_manager.cc:5746] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571) does not have the latest schema for tablet 7d76046ee6c8420fa4964decbe8209c1 (table default.Table_55e0ba23f44146339c7c6928440d4948 [id=4c70005d1005479fa6c3a6cb6f5ce658]). Expected version 1 got 0
I20260130 02:58:05.183095 18861 catalog_manager.cc:4985] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021): tablet a743a82540014c85aeab0e9d0994b0dc (table default.Table_71ecfab2cca249b49070a8340586ce5e [id=19941bcff5454783adb47df8dd77f352]) successfully deleted
W20260130 02:58:05.183775 18861 catalog_manager.cc:4965] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021): delete failed for tablet f54addb1a4304ce69c711ae0cc3b452c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f54addb1a4304ce69c711ae0cc3b452c already in progress: creating tablet
W20260130 02:58:05.184338 18861 catalog_manager.cc:4965] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021): delete failed for tablet 7d96e93d95f14af79cfb268626559b1e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7d96e93d95f14af79cfb268626559b1e already in progress: creating tablet
I20260130 02:58:05.183624 19494 tablet_service.cc:1505] Processing CreateTablet for tablet 3389db58493448aea3cf00aca411047c (DEFAULT_TABLE table=default.Table_22d10332a83f4700ac21892bab67f46b [id=da3a9cb1f6d5419abc1afc93659b5324]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:05.186332 19494 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3389db58493448aea3cf00aca411047c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:05.190455 19198 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ba91a508b1cb4430a4d8ca26946aecc8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:05.196280 19199 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9fe0448bcc4a4346b3c67fc3f626ed3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:05.196831 19205 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3389db58493448aea3cf00aca411047c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:05.198596 19201 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4907294442c145c6b03eb1de057c66a8. 1 dirs total, 0 dirs full, 0 dirs failed
W20260130 02:58:05.203426 19159 leader_election.cc:343] T 534dc31a9b7e492c82f3672c75742e6c P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767): Illegal state: Tablet not RUNNING: STOPPED
I20260130 02:58:05.207823 19754 raft_consensus.cc:2272] T c3fce8cc99fb451ebd532fa37390e862 P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260130 02:58:05.212203 19754 ts_tablet_manager.cc:1916] T c3fce8cc99fb451ebd532fa37390e862 P 371a36d219164974abdf848841528152: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260130 02:58:05.217478 19376 raft_consensus.cc:2468] T 2ca0df71836145748cada4e4559a5656 P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 371a36d219164974abdf848841528152 in term 1.
I20260130 02:58:05.220285 19223 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0a763e48583c4e748a46a325746ce69a" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152" is_pre_election: true
W20260130 02:58:05.221376 19314 leader_election.cc:343] T 0a763e48583c4e748a46a325746ce69a P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:05.226912 19204 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 345eaaa5625c48d099280b7fbb2e514b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:05.227231 19357 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ba91a508b1cb4430a4d8ca26946aecc8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:05.228636 19355 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3389db58493448aea3cf00aca411047c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:05.230002 19351 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4907294442c145c6b03eb1de057c66a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:05.230885 19350 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9fe0448bcc4a4346b3c67fc3f626ed3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:05.240427 20180 raft_consensus.cc:493] T 91f954f3ef4244c9a575a7421a7e75eb P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:05.240939 20180 raft_consensus.cc:515] T 91f954f3ef4244c9a575a7421a7e75eb P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:05.244334 20180 leader_election.cc:290] T 91f954f3ef4244c9a575a7421a7e75eb P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), d870164263bb44b6b780a557daea3656 (127.18.100.67:39571)
I20260130 02:58:05.245762 19223 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "91f954f3ef4244c9a575a7421a7e75eb" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152" is_pre_election: true
W20260130 02:58:05.247990 19314 leader_election.cc:343] T 91f954f3ef4244c9a575a7421a7e75eb P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:05.249656 19197 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0e8018c094024c4788c484e8f5466e00. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:05.251366 19159 leader_election.cc:304] T 2ca0df71836145748cada4e4559a5656 P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 371a36d219164974abdf848841528152, 78c2c46c7520492db35d51c244fb4f74; no voters:
I20260130 02:58:05.260365 20179 consensus_queue.cc:1048] T e47b22aafe5247c4b62c4193c24f427c P 371a36d219164974abdf848841528152 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260130 02:58:05.261623 20179 raft_consensus.cc:2804] T 2ca0df71836145748cada4e4559a5656 P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Leader election won for term 1
I20260130 02:58:05.262279 20179 raft_consensus.cc:697] T 2ca0df71836145748cada4e4559a5656 P 371a36d219164974abdf848841528152 [term 1 LEADER]: Becoming Leader. State: Replica: 371a36d219164974abdf848841528152, State: Running, Role: LEADER
I20260130 02:58:05.263055 20179 consensus_queue.cc:237] T 2ca0df71836145748cada4e4559a5656 P 371a36d219164974abdf848841528152 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:05.264281 18873 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50284:
name: "default.Table_12b505700b3d44e28353172e1786e5c8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260130 02:58:05.265803 18873 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_12b505700b3d44e28353172e1786e5c8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260130 02:58:05.306171 19754 ts_tablet_manager.cc:1929] T c3fce8cc99fb451ebd532fa37390e862 P 371a36d219164974abdf848841528152: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260130 02:58:05.306535 19754 log.cc:1199] T c3fce8cc99fb451ebd532fa37390e862 P 371a36d219164974abdf848841528152: Deleting WAL directory at /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-0/wal/wals/c3fce8cc99fb451ebd532fa37390e862
I20260130 02:58:05.327584 19754 ts_tablet_manager.cc:1950] T c3fce8cc99fb451ebd532fa37390e862 P 371a36d219164974abdf848841528152: Deleting consensus metadata
I20260130 02:58:05.330869 18873 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50284:
name: "default.Table_95f0bff8e17d4ca995c0af0d0c24e84d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260130 02:58:05.332358 18873 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_95f0bff8e17d4ca995c0af0d0c24e84d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260130 02:58:05.340288 19807 tablet_bootstrap.cc:492] T 0a763e48583c4e748a46a325746ce69a P 371a36d219164974abdf848841528152: No bootstrap required, opened a new log
I20260130 02:58:05.340744 19807 ts_tablet_manager.cc:1403] T 0a763e48583c4e748a46a325746ce69a P 371a36d219164974abdf848841528152: Time spent bootstrapping tablet: real 0.222s user 0.029s sys 0.004s
I20260130 02:58:05.342365 18861 catalog_manager.cc:4985] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021): tablet c3fce8cc99fb451ebd532fa37390e862 (table default.Table_5d29239ce1984d2fad02607a2bdd4dd0 [id=7e271b81bb064a949685b2411b485e3e]) successfully deleted
I20260130 02:58:05.344079 19754 tablet_replica.cc:333] T df2c8989d6c1480c97baa23db01141eb P 371a36d219164974abdf848841528152: stopping tablet replica
I20260130 02:58:05.346390 19807 raft_consensus.cc:359] T 0a763e48583c4e748a46a325746ce69a P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:05.347541 19807 raft_consensus.cc:385] T 0a763e48583c4e748a46a325746ce69a P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:05.348749 19807 raft_consensus.cc:740] T 0a763e48583c4e748a46a325746ce69a P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 371a36d219164974abdf848841528152, State: Initialized, Role: FOLLOWER
I20260130 02:58:05.310959 19352 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 345eaaa5625c48d099280b7fbb2e514b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:05.354807 19754 raft_consensus.cc:2243] T df2c8989d6c1480c97baa23db01141eb P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260130 02:58:05.355309 19754 raft_consensus.cc:2272] T df2c8989d6c1480c97baa23db01141eb P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260130 02:58:05.356952 19754 ts_tablet_manager.cc:1916] T df2c8989d6c1480c97baa23db01141eb P 371a36d219164974abdf848841528152: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260130 02:58:05.358963 18873 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50284:
name: "default.Table_d24e400f7451443a82d23140a90adc57"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260130 02:58:05.360574 18873 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d24e400f7451443a82d23140a90adc57 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260130 02:58:05.372020 19758 ts_tablet_manager.cc:1929] T 534dc31a9b7e492c82f3672c75742e6c P 78c2c46c7520492db35d51c244fb4f74: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260130 02:58:05.372359 19758 log.cc:1199] T 534dc31a9b7e492c82f3672c75742e6c P 78c2c46c7520492db35d51c244fb4f74: Deleting WAL directory at /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-1/wal/wals/534dc31a9b7e492c82f3672c75742e6c
I20260130 02:58:05.372948 19758 ts_tablet_manager.cc:1950] T 534dc31a9b7e492c82f3672c75742e6c P 78c2c46c7520492db35d51c244fb4f74: Deleting consensus metadata
I20260130 02:58:05.372526 19807 consensus_queue.cc:260] T 0a763e48583c4e748a46a325746ce69a P 371a36d219164974abdf848841528152 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:05.377755 19807 ts_tablet_manager.cc:1434] T 0a763e48583c4e748a46a325746ce69a P 371a36d219164974abdf848841528152: Time spent starting tablet: real 0.037s user 0.006s sys 0.000s
I20260130 02:58:05.385542 19840 tablet_bootstrap.cc:492] T a90e92d569254bbf8a5fbf286baf5cce P 78c2c46c7520492db35d51c244fb4f74: No bootstrap required, opened a new log
I20260130 02:58:05.386008 19840 ts_tablet_manager.cc:1403] T a90e92d569254bbf8a5fbf286baf5cce P 78c2c46c7520492db35d51c244fb4f74: Time spent bootstrapping tablet: real 0.460s user 0.023s sys 0.022s
I20260130 02:58:05.396701 19840 raft_consensus.cc:359] T a90e92d569254bbf8a5fbf286baf5cce P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:05.397342 19840 raft_consensus.cc:385] T a90e92d569254bbf8a5fbf286baf5cce P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:05.397589 19840 raft_consensus.cc:740] T a90e92d569254bbf8a5fbf286baf5cce P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78c2c46c7520492db35d51c244fb4f74, State: Initialized, Role: FOLLOWER
I20260130 02:58:05.402009 19807 tablet_bootstrap.cc:492] T 730bd1d7d15b4cbe945b3ef03b2455f7 P 371a36d219164974abdf848841528152: Bootstrap starting.
I20260130 02:58:05.398151 19840 consensus_queue.cc:260] T a90e92d569254bbf8a5fbf286baf5cce P 78c2c46c7520492db35d51c244fb4f74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:05.412722 19523 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "534dc31a9b7e492c82f3672c75742e6c" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
I20260130 02:58:05.413846 19525 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ca0df71836145748cada4e4559a5656" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656"
I20260130 02:58:05.414451 19525 raft_consensus.cc:3060] T 2ca0df71836145748cada4e4559a5656 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:05.410459 19840 ts_tablet_manager.cc:1434] T a90e92d569254bbf8a5fbf286baf5cce P 78c2c46c7520492db35d51c244fb4f74: Time spent starting tablet: real 0.016s user 0.002s sys 0.003s
I20260130 02:58:05.417514 19840 tablet_bootstrap.cc:492] T 6573198830ad45628e71952a7d9c00a0 P 78c2c46c7520492db35d51c244fb4f74: Bootstrap starting.
I20260130 02:58:05.425269 18860 catalog_manager.cc:4985] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767): tablet 534dc31a9b7e492c82f3672c75742e6c (table default.Table_081113b62fb743189a6e04bf85dcc5fa [id=66e19e6011bc4d86bf579a4a1b3db867]) successfully deleted
W20260130 02:58:05.432648 19159 consensus_peers.cc:597] T e47b22aafe5247c4b62c4193c24f427c P 371a36d219164974abdf848841528152 -> Peer d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): Couldn't send request to peer d870164263bb44b6b780a557daea3656. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20260130 02:58:05.433200 19159 leader_election.cc:343] T 534dc31a9b7e492c82f3672c75742e6c P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:05.433545 19159 leader_election.cc:304] T 534dc31a9b7e492c82f3672c75742e6c P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 371a36d219164974abdf848841528152; no voters: 78c2c46c7520492db35d51c244fb4f74, d870164263bb44b6b780a557daea3656
I20260130 02:58:05.434643 20176 raft_consensus.cc:2749] T 534dc31a9b7e492c82f3672c75742e6c P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260130 02:58:05.435415 19625 tablet_bootstrap.cc:492] T 481bd4cca6ee4c178873bf12392e3441 P d870164263bb44b6b780a557daea3656: Bootstrap starting.
I20260130 02:58:05.422561 19840 tablet_bootstrap.cc:654] T 6573198830ad45628e71952a7d9c00a0 P 78c2c46c7520492db35d51c244fb4f74: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:05.443043 19508 tablet_service.cc:1505] Processing CreateTablet for tablet 4907294442c145c6b03eb1de057c66a8 (DEFAULT_TABLE table=default.Table_ea2e0aa925a0486bb292304ca21bb819 [id=18580b9c54e448d1bdaae84f697f7b0b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:05.444024 19508 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4907294442c145c6b03eb1de057c66a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:05.450479 19947 ts_tablet_manager.cc:1929] T d701152960b14396a6ac1bd10480d2b6 P d870164263bb44b6b780a557daea3656: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260130 02:58:05.450876 19947 log.cc:1199] T d701152960b14396a6ac1bd10480d2b6 P d870164263bb44b6b780a557daea3656: Deleting WAL directory at /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-2/wal/wals/d701152960b14396a6ac1bd10480d2b6
I20260130 02:58:05.451558 19947 ts_tablet_manager.cc:1950] T d701152960b14396a6ac1bd10480d2b6 P d870164263bb44b6b780a557daea3656: Deleting consensus metadata
I20260130 02:58:05.458452 19527 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "91f954f3ef4244c9a575a7421a7e75eb" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
I20260130 02:58:05.460323 20180 raft_consensus.cc:493] T 341041cb0be2456e88403367c0814704 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:05.460786 20180 raft_consensus.cc:515] T 341041cb0be2456e88403367c0814704 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:05.463721 19223 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "341041cb0be2456e88403367c0814704" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152" is_pre_election: true
W20260130 02:58:05.465083 19314 leader_election.cc:343] T 341041cb0be2456e88403367c0814704 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:05.462534 20180 leader_election.cc:290] T 341041cb0be2456e88403367c0814704 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), d870164263bb44b6b780a557daea3656 (127.18.100.67:39571)
I20260130 02:58:05.466729 19947 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59068 (request call id 106) took 3590 ms. Trace:
I20260130 02:58:05.467053 19947 rpcz_store.cc:276] 0130 02:58:01.876009 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:58:01.876180 (+ 171us) service_pool.cc:225] Handling call
0130 02:58:04.432091 (+2555911us) ts_tablet_manager.cc:1151] Deleting tablet d701152960b14396a6ac1bd10480d2b6
0130 02:58:04.432131 (+ 40us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0130 02:58:05.042055 (+609924us) tablet_metadata.cc:641] Metadata flushed
0130 02:58:05.326253 (+284198us) tablet_metadata.cc:641] Metadata flushed
0130 02:58:05.466664 (+140411us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":37241,"tablet-delete.queue_time_us":2555429}
I20260130 02:58:05.468317 19947 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59068 (request call id 103) took 3597 ms. Trace:
I20260130 02:58:05.468601 19947 rpcz_store.cc:276] 0130 02:58:01.871406 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:58:01.876773 (+ 5367us) service_pool.cc:225] Handling call
0130 02:58:05.468106 (+3591333us) ts_tablet_manager.cc:1151] Deleting tablet d4f2b54930cc4f5c825562e3e0a79f4d
0130 02:58:05.468144 (+ 38us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0130 02:58:05.468272 (+ 128us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3590732}
I20260130 02:58:05.472591 19840 tablet_bootstrap.cc:492] T 6573198830ad45628e71952a7d9c00a0 P 78c2c46c7520492db35d51c244fb4f74: No bootstrap required, opened a new log
I20260130 02:58:05.473021 19840 ts_tablet_manager.cc:1403] T 6573198830ad45628e71952a7d9c00a0 P 78c2c46c7520492db35d51c244fb4f74: Time spent bootstrapping tablet: real 0.057s user 0.009s sys 0.000s
I20260130 02:58:05.477869 19840 raft_consensus.cc:359] T 6573198830ad45628e71952a7d9c00a0 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:05.479804 19840 raft_consensus.cc:385] T 6573198830ad45628e71952a7d9c00a0 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:05.480060 19840 raft_consensus.cc:740] T 6573198830ad45628e71952a7d9c00a0 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78c2c46c7520492db35d51c244fb4f74, State: Initialized, Role: FOLLOWER
I20260130 02:58:05.469462 19947 tablet_replica.cc:333] T 37470ce4356f4542b621e21698551209 P d870164263bb44b6b780a557daea3656: stopping tablet replica
I20260130 02:58:05.481966 19947 raft_consensus.cc:2243] T 37470ce4356f4542b621e21698551209 P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260130 02:58:05.483040 20179 raft_consensus.cc:493] T 0a763e48583c4e748a46a325746ce69a P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:05.483532 20179 raft_consensus.cc:515] T 0a763e48583c4e748a46a325746ce69a P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:05.485549 20179 leader_election.cc:290] T 0a763e48583c4e748a46a325746ce69a P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d870164263bb44b6b780a557daea3656 (127.18.100.67:39571), 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767)
I20260130 02:58:05.487064 19754 ts_tablet_manager.cc:1929] T df2c8989d6c1480c97baa23db01141eb P 371a36d219164974abdf848841528152: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20260130 02:58:05.487579 19377 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0a763e48583c4e748a46a325746ce69a" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74" is_pre_election: true
I20260130 02:58:05.488204 19377 raft_consensus.cc:2468] T 0a763e48583c4e748a46a325746ce69a P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 371a36d219164974abdf848841528152 in term 0.
I20260130 02:58:05.489152 19159 leader_election.cc:304] T 0a763e48583c4e748a46a325746ce69a P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 371a36d219164974abdf848841528152, 78c2c46c7520492db35d51c244fb4f74; no voters:
I20260130 02:58:05.489997 20179 raft_consensus.cc:2804] T 0a763e48583c4e748a46a325746ce69a P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260130 02:58:05.490315 20179 raft_consensus.cc:493] T 0a763e48583c4e748a46a325746ce69a P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260130 02:58:05.490634 20179 raft_consensus.cc:3060] T 0a763e48583c4e748a46a325746ce69a P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:05.490664 19947 raft_consensus.cc:2272] T 37470ce4356f4542b621e21698551209 P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260130 02:58:05.491252 19840 consensus_queue.cc:260] T 6573198830ad45628e71952a7d9c00a0 P 78c2c46c7520492db35d51c244fb4f74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:05.497308 20179 raft_consensus.cc:515] T 0a763e48583c4e748a46a325746ce69a P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:05.498768 19505 tablet_service.cc:1505] Processing CreateTablet for tablet 9fe0448bcc4a4346b3c67fc3f626ed3c (DEFAULT_TABLE table=default.Table_5c1c447e85524ac5a61ffd8a48509d36 [id=70aaf21ad9bd4d959f46939c6b4a4880]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:05.499774 19754 log.cc:1199] T df2c8989d6c1480c97baa23db01141eb P 371a36d219164974abdf848841528152: Deleting WAL directory at /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-0/wal/wals/df2c8989d6c1480c97baa23db01141eb
I20260130 02:58:05.500228 19840 ts_tablet_manager.cc:1434] T 6573198830ad45628e71952a7d9c00a0 P 78c2c46c7520492db35d51c244fb4f74: Time spent starting tablet: real 0.027s user 0.006s sys 0.000s
I20260130 02:58:05.500487 19754 ts_tablet_manager.cc:1950] T df2c8989d6c1480c97baa23db01141eb P 371a36d219164974abdf848841528152: Deleting consensus metadata
I20260130 02:58:05.502779 18861 catalog_manager.cc:4985] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021): tablet df2c8989d6c1480c97baa23db01141eb (table default.Table_e84c6277dfc944059f62cbe0d6b937ee [id=747af828147149efb7b7b8d33c49286d]) successfully deleted
I20260130 02:58:05.502995 19840 tablet_bootstrap.cc:492] T 3b148691090e4ee581d46f9845a6e49a P 78c2c46c7520492db35d51c244fb4f74: Bootstrap starting.
I20260130 02:58:05.494376 20153 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.AlterSchema from 127.0.0.1:59068 (request call id 128) took 1549 ms. Trace:
I20260130 02:58:05.512585 19377 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0a763e48583c4e748a46a325746ce69a" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74"
I20260130 02:58:05.513134 19377 raft_consensus.cc:3060] T 0a763e48583c4e748a46a325746ce69a P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:05.513743 20179 leader_election.cc:290] T 0a763e48583c4e748a46a325746ce69a P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 election: Requested vote from peers d870164263bb44b6b780a557daea3656 (127.18.100.67:39571), 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767)
W20260130 02:58:05.517222 18861 catalog_manager.cc:4965] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021): delete failed for tablet b2ac91da01954c2390b6b7dcc06c394f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b2ac91da01954c2390b6b7dcc06c394f already in progress: creating tablet
I20260130 02:58:05.518520 20180 raft_consensus.cc:493] T 6573198830ad45628e71952a7d9c00a0 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:05.519037 19754 tablet_replica.cc:333] T 534dc31a9b7e492c82f3672c75742e6c P 371a36d219164974abdf848841528152: stopping tablet replica
I20260130 02:58:05.518989 20180 raft_consensus.cc:515] T 6573198830ad45628e71952a7d9c00a0 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:05.519775 19754 raft_consensus.cc:2243] T 534dc31a9b7e492c82f3672c75742e6c P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260130 02:58:05.520197 19754 raft_consensus.cc:2272] T 534dc31a9b7e492c82f3672c75742e6c P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260130 02:58:05.520540 19526 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0a763e48583c4e748a46a325746ce69a" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
I20260130 02:58:05.520751 20180 leader_election.cc:290] T 6573198830ad45628e71952a7d9c00a0 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), d870164263bb44b6b780a557daea3656 (127.18.100.67:39571)
I20260130 02:58:05.521847 19754 ts_tablet_manager.cc:1916] T 534dc31a9b7e492c82f3672c75742e6c P 371a36d219164974abdf848841528152: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260130 02:58:05.522114 19223 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6573198830ad45628e71952a7d9c00a0" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152" is_pre_election: true
W20260130 02:58:05.523766 19314 leader_election.cc:343] T 6573198830ad45628e71952a7d9c00a0 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:05.525179 19377 raft_consensus.cc:2468] T 0a763e48583c4e748a46a325746ce69a P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 371a36d219164974abdf848841528152 in term 1.
I20260130 02:58:05.526401 19159 leader_election.cc:304] T 0a763e48583c4e748a46a325746ce69a P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 371a36d219164974abdf848841528152, 78c2c46c7520492db35d51c244fb4f74; no voters:
I20260130 02:58:05.527257 20179 raft_consensus.cc:2804] T 0a763e48583c4e748a46a325746ce69a P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Leader election won for term 1
W20260130 02:58:05.528059 18859 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet 9fe0448bcc4a4346b3c67fc3f626ed3c (table default.Table_5c1c447e85524ac5a61ffd8a48509d36 [id=70aaf21ad9bd4d959f46939c6b4a4880], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 9fe0448bcc4a4346b3c67fc3f626ed3c consensus state has no leader
I20260130 02:58:05.529013 20179 raft_consensus.cc:697] T 0a763e48583c4e748a46a325746ce69a P 371a36d219164974abdf848841528152 [term 1 LEADER]: Becoming Leader. State: Replica: 371a36d219164974abdf848841528152, State: Running, Role: LEADER
I20260130 02:58:05.530181 20179 consensus_queue.cc:237] T 0a763e48583c4e748a46a325746ce69a P 371a36d219164974abdf848841528152 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:05.535473 20180 consensus_queue.cc:1048] T 481bd4cca6ee4c178873bf12392e3441 P 78c2c46c7520492db35d51c244fb4f74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260130 02:58:05.536087 19840 tablet_bootstrap.cc:654] T 3b148691090e4ee581d46f9845a6e49a P 78c2c46c7520492db35d51c244fb4f74: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:05.541672 20115 consensus_queue.cc:1048] T 7b80b22bd44c4233837c951aa97e6996 P 371a36d219164974abdf848841528152 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260130 02:58:05.505905 20153 rpcz_store.cc:276] 0130 02:58:03.946053 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:58:04.153058 (+207005us) service_pool.cc:225] Handling call
0130 02:58:05.487444 (+1334386us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0130 02:58:04.153913 (+ 0us) alter_schema_op.cc:104] PREPARE ALTER-SCHEMA: Starting
0130 02:58:04.154238 (+ 325us) alter_schema_op.cc:63] Acquiring schema lock in exclusive mode
0130 02:58:04.154267 (+ 29us) alter_schema_op.cc:66] Acquired schema lock
0130 02:58:04.154319 (+ 52us) alter_schema_op.cc:119] PREPARE ALTER-SCHEMA: finished
0130 02:58:04.154429 (+ 110us) alter_schema_op.cc:128] START. Timestamp: 1769741884154368
0130 02:58:04.154449 (+ 20us) op_driver.cc:348] REPLICATION: starting
0130 02:58:04.154986 (+ 537us) log.cc:844] Serialized 173 byte log entry
0130 02:58:04.286481 (+131495us) op_driver.cc:464] REPLICATION: finished
0130 02:58:04.597356 (+310875us) alter_schema_op.cc:133] APPLY ALTER-SCHEMA: Starting
0130 02:58:04.599025 (+ 1669us) log.cc:844] Serialized 5 byte log entry
0130 02:58:04.901525 (+302500us) tablet_metadata.cc:641] Metadata flushed
0130 02:58:05.463090 (+561565us) log.cc:844] Serialized 14 byte log entry
0130 02:58:05.465873 (+ 2783us) alter_schema_op.cc:73] Released schema lock
0130 02:58:05.470443 (+ 4570us) alter_schema_op.cc:186] AlterSchemaCommitCallback: making alter schema visible
Metrics: {"child_traces":[["op",{"apply.queue_time_us":309237,"prepare.queue_time_us":113,"prepare.run_cpu_time_us":2054,"prepare.run_wall_time_us":2064,"raft.queue_time_us":1448,"raft.run_cpu_time_us":1527,"raft.run_wall_time_us":1526,"replication_time_us":118217,"thread_start_us":237929,"threads_started":2,"wal-append.queue_time_us":819}]]}
I20260130 02:58:05.554984 19505 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9fe0448bcc4a4346b3c67fc3f626ed3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:05.585490 19840 tablet_bootstrap.cc:492] T 3b148691090e4ee581d46f9845a6e49a P 78c2c46c7520492db35d51c244fb4f74: No bootstrap required, opened a new log
I20260130 02:58:05.585915 19840 ts_tablet_manager.cc:1403] T 3b148691090e4ee581d46f9845a6e49a P 78c2c46c7520492db35d51c244fb4f74: Time spent bootstrapping tablet: real 0.083s user 0.006s sys 0.002s
I20260130 02:58:05.592090 19625 tablet_bootstrap.cc:654] T 481bd4cca6ee4c178873bf12392e3441 P d870164263bb44b6b780a557daea3656: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:05.594252 19840 raft_consensus.cc:359] T 3b148691090e4ee581d46f9845a6e49a P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:05.594918 19840 raft_consensus.cc:385] T 3b148691090e4ee581d46f9845a6e49a P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:05.595151 19840 raft_consensus.cc:740] T 3b148691090e4ee581d46f9845a6e49a P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78c2c46c7520492db35d51c244fb4f74, State: Initialized, Role: FOLLOWER
I20260130 02:58:05.599697 19840 consensus_queue.cc:260] T 3b148691090e4ee581d46f9845a6e49a P 78c2c46c7520492db35d51c244fb4f74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:05.600004 19525 raft_consensus.cc:2468] T 2ca0df71836145748cada4e4559a5656 P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 371a36d219164974abdf848841528152 in term 1.
I20260130 02:58:05.601629 19840 ts_tablet_manager.cc:1434] T 3b148691090e4ee581d46f9845a6e49a P 78c2c46c7520492db35d51c244fb4f74: Time spent starting tablet: real 0.015s user 0.005s sys 0.000s
I20260130 02:58:05.602428 19840 tablet_bootstrap.cc:492] T 3cf5154d8cc44f2bb12978202699c193 P 78c2c46c7520492db35d51c244fb4f74: Bootstrap starting.
I20260130 02:58:05.619361 19840 tablet_bootstrap.cc:654] T 3cf5154d8cc44f2bb12978202699c193 P 78c2c46c7520492db35d51c244fb4f74: Neither blocks nor log segments found. Creating new log.
W20260130 02:58:05.623319 19498 consensus_meta.cc:287] T 8e57ae31ab3941f789ac69627384e3b0 P d870164263bb44b6b780a557daea3656: Time spent flushing consensus metadata: real 0.516s user 0.005s sys 0.002s
W20260130 02:58:05.634174 19313 leader_election.cc:343] T 91f954f3ef4244c9a575a7421a7e75eb P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:05.634618 19313 leader_election.cc:304] T 91f954f3ef4244c9a575a7421a7e75eb P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78c2c46c7520492db35d51c244fb4f74; no voters: 371a36d219164974abdf848841528152, d870164263bb44b6b780a557daea3656
I20260130 02:58:05.635270 20194 raft_consensus.cc:2749] T 91f954f3ef4244c9a575a7421a7e75eb P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260130 02:58:05.636842 19313 leader_election.cc:343] T 0a763e48583c4e748a46a325746ce69a P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:05.637192 19313 leader_election.cc:304] T 0a763e48583c4e748a46a325746ce69a P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78c2c46c7520492db35d51c244fb4f74; no voters: 371a36d219164974abdf848841528152, d870164263bb44b6b780a557daea3656
W20260130 02:58:05.640527 19499 consensus_meta.cc:287] T d55291d7cea64a0d9a345ce712fbfa80 P d870164263bb44b6b780a557daea3656: Time spent flushing consensus metadata: real 0.562s user 0.007s sys 0.000s
I20260130 02:58:05.642452 19947 ts_tablet_manager.cc:1916] T 37470ce4356f4542b621e21698551209 P d870164263bb44b6b780a557daea3656: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260130 02:58:05.644129 19840 tablet_bootstrap.cc:492] T 3cf5154d8cc44f2bb12978202699c193 P 78c2c46c7520492db35d51c244fb4f74: No bootstrap required, opened a new log
I20260130 02:58:05.646567 19840 ts_tablet_manager.cc:1403] T 3cf5154d8cc44f2bb12978202699c193 P 78c2c46c7520492db35d51c244fb4f74: Time spent bootstrapping tablet: real 0.044s user 0.007s sys 0.003s
I20260130 02:58:05.648579 19522 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "341041cb0be2456e88403367c0814704" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
I20260130 02:58:05.650826 19528 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6573198830ad45628e71952a7d9c00a0" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
I20260130 02:58:05.658365 19625 tablet_bootstrap.cc:492] T 481bd4cca6ee4c178873bf12392e3441 P d870164263bb44b6b780a557daea3656: No bootstrap required, opened a new log
I20260130 02:58:05.658776 19625 ts_tablet_manager.cc:1403] T 481bd4cca6ee4c178873bf12392e3441 P d870164263bb44b6b780a557daea3656: Time spent bootstrapping tablet: real 0.224s user 0.010s sys 0.001s
I20260130 02:58:05.658591 20179 consensus_queue.cc:1048] T 7b80b22bd44c4233837c951aa97e6996 P 371a36d219164974abdf848841528152 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260130 02:58:05.660952 19625 raft_consensus.cc:359] T 481bd4cca6ee4c178873bf12392e3441 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:05.661556 19625 raft_consensus.cc:385] T 481bd4cca6ee4c178873bf12392e3441 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:05.661799 19625 raft_consensus.cc:740] T 481bd4cca6ee4c178873bf12392e3441 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d870164263bb44b6b780a557daea3656, State: Initialized, Role: FOLLOWER
I20260130 02:58:05.662418 19625 consensus_queue.cc:260] T 481bd4cca6ee4c178873bf12392e3441 P d870164263bb44b6b780a557daea3656 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
W20260130 02:58:05.657806 19462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.100.66:40767, user_credentials={real_user=slave}} blocked reactor thread for 301322us
W20260130 02:58:05.669384 18860 catalog_manager.cc:4952] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): delete failed for tablet d4f2b54930cc4f5c825562e3e0a79f4d because the tablet was not found. No further retry: Not found: Tablet not found: d4f2b54930cc4f5c825562e3e0a79f4d
I20260130 02:58:05.672996 19525 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0a763e48583c4e748a46a325746ce69a" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
I20260130 02:58:05.674046 19754 ts_tablet_manager.cc:1929] T 534dc31a9b7e492c82f3672c75742e6c P 371a36d219164974abdf848841528152: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260130 02:58:05.674402 19754 log.cc:1199] T 534dc31a9b7e492c82f3672c75742e6c P 371a36d219164974abdf848841528152: Deleting WAL directory at /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-0/wal/wals/534dc31a9b7e492c82f3672c75742e6c
I20260130 02:58:05.675117 19754 ts_tablet_manager.cc:1950] T 534dc31a9b7e492c82f3672c75742e6c P 371a36d219164974abdf848841528152: Deleting consensus metadata
I20260130 02:58:05.675341 19525 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0a763e48583c4e748a46a325746ce69a" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656"
I20260130 02:58:05.677373 18861 catalog_manager.cc:4985] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021): tablet 534dc31a9b7e492c82f3672c75742e6c (table default.Table_081113b62fb743189a6e04bf85dcc5fa [id=66e19e6011bc4d86bf579a4a1b3db867]) successfully deleted
I20260130 02:58:05.678131 20194 raft_consensus.cc:2749] T 0a763e48583c4e748a46a325746ce69a P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260130 02:58:05.679437 19313 leader_election.cc:343] T 341041cb0be2456e88403367c0814704 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:05.679857 19313 leader_election.cc:304] T 341041cb0be2456e88403367c0814704 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78c2c46c7520492db35d51c244fb4f74; no voters: 371a36d219164974abdf848841528152, d870164263bb44b6b780a557daea3656
I20260130 02:58:05.680546 20194 raft_consensus.cc:2749] T 341041cb0be2456e88403367c0814704 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260130 02:58:05.681823 19807 tablet_bootstrap.cc:654] T 730bd1d7d15b4cbe945b3ef03b2455f7 P 371a36d219164974abdf848841528152: Neither blocks nor log segments found. Creating new log.
W20260130 02:58:05.682116 19313 leader_election.cc:343] T 6573198830ad45628e71952a7d9c00a0 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): Illegal state: must be running to vote when last-logged opid is not known
W20260130 02:58:05.693359 19463 consensus_peers.cc:597] T b7b596f944be4daf953b82b65d5cec2a P d870164263bb44b6b780a557daea3656 -> Peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Couldn't send request to peer 371a36d219164974abdf848841528152. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b7b596f944be4daf953b82b65d5cec2a. This is attempt 1: this message will repeat every 5th retry.
W20260130 02:58:05.694293 19159 leader_election.cc:343] T 0a763e48583c4e748a46a325746ce69a P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:05.703773 19197 tablet_service.cc:1505] Processing CreateTablet for tablet 274530359b174ff79f046c2d4a6afa76 (DEFAULT_TABLE table=default.Table_6b60df181f10452ba3f2e20e847bf502 [id=c456c6418c93499a9d08f652720854ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:05.704977 19197 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 274530359b174ff79f046c2d4a6afa76. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:05.713653 20115 consensus_queue.cc:1048] T 5ca9919f80954e74a4d91c94b74de0e4 P 371a36d219164974abdf848841528152 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260130 02:58:05.714792 19313 leader_election.cc:304] T 6573198830ad45628e71952a7d9c00a0 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78c2c46c7520492db35d51c244fb4f74; no voters: 371a36d219164974abdf848841528152, d870164263bb44b6b780a557daea3656
I20260130 02:58:05.715523 20180 raft_consensus.cc:2749] T 6573198830ad45628e71952a7d9c00a0 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260130 02:58:05.717377 19200 tablet_service.cc:1505] Processing CreateTablet for tablet 03025d13f0e545cfa6288677a2715343 (DEFAULT_TABLE table=default.Table_21178f657dca43a1912ce596834d1618 [id=53b79067a6034189aac5f1554df542c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:05.717782 19352 tablet_service.cc:1505] Processing CreateTablet for tablet 274530359b174ff79f046c2d4a6afa76 (DEFAULT_TABLE table=default.Table_6b60df181f10452ba3f2e20e847bf502 [id=c456c6418c93499a9d08f652720854ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:05.718359 19200 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 03025d13f0e545cfa6288677a2715343. 1 dirs total, 0 dirs full, 0 dirs failed
W20260130 02:58:05.718283 19313 consensus_peers.cc:597] T 481bd4cca6ee4c178873bf12392e3441 P 78c2c46c7520492db35d51c244fb4f74 -> Peer d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): Couldn't send request to peer d870164263bb44b6b780a557daea3656. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20260130 02:58:05.718803 19352 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 274530359b174ff79f046c2d4a6afa76. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:05.723974 19499 tablet_service.cc:1505] Processing CreateTablet for tablet 274530359b174ff79f046c2d4a6afa76 (DEFAULT_TABLE table=default.Table_6b60df181f10452ba3f2e20e847bf502 [id=c456c6418c93499a9d08f652720854ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:05.725229 19499 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 274530359b174ff79f046c2d4a6afa76. 1 dirs total, 0 dirs full, 0 dirs failed
W20260130 02:58:05.729583 19159 leader_election.cc:343] T 0a763e48583c4e748a46a325746ce69a P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:05.664965 19625 ts_tablet_manager.cc:1434] T 481bd4cca6ee4c178873bf12392e3441 P d870164263bb44b6b780a557daea3656: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260130 02:58:05.731343 19625 tablet_bootstrap.cc:492] T e47b22aafe5247c4b62c4193c24f427c P d870164263bb44b6b780a557daea3656: Bootstrap starting.
I20260130 02:58:05.735036 19198 tablet_service.cc:1505] Processing CreateTablet for tablet 5fbe92a2227d44d88b4531a7f8aa0e08 (DEFAULT_TABLE table=default.Table_a3ecda6d64704b7fad065a5845683ad8 [id=e93360fd2af141a28d92401ed449c19d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:05.736191 19198 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5fbe92a2227d44d88b4531a7f8aa0e08. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:05.737185 19204 tablet_service.cc:1505] Processing CreateTablet for tablet f41f4cff7dd4489f948700a45e9a7057 (DEFAULT_TABLE table=default.Table_ed6e2e20d89542199e5cbc7842df964d [id=047c39bd14664d1099e043b973766d32]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:05.738104 19204 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f41f4cff7dd4489f948700a45e9a7057. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:05.738469 20178 consensus_queue.cc:1048] T 5ca9919f80954e74a4d91c94b74de0e4 P 371a36d219164974abdf848841528152 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260130 02:58:05.741434 19201 tablet_service.cc:1505] Processing CreateTablet for tablet b1e8b78eff0d4a0b92059adb2de4a3ce (DEFAULT_TABLE table=default.Table_1cb6165b7dda4b659c1c379902ee4930 [id=f1342f9497b74e7a9ecad1de8082578d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:05.742422 19201 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b1e8b78eff0d4a0b92059adb2de4a3ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:05.684115 19840 raft_consensus.cc:359] T 3cf5154d8cc44f2bb12978202699c193 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:05.750950 19840 raft_consensus.cc:385] T 3cf5154d8cc44f2bb12978202699c193 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:05.751250 19840 raft_consensus.cc:740] T 3cf5154d8cc44f2bb12978202699c193 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78c2c46c7520492db35d51c244fb4f74, State: Initialized, Role: FOLLOWER
I20260130 02:58:05.751940 19840 consensus_queue.cc:260] T 3cf5154d8cc44f2bb12978202699c193 P 78c2c46c7520492db35d51c244fb4f74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:05.754129 19840 ts_tablet_manager.cc:1434] T 3cf5154d8cc44f2bb12978202699c193 P 78c2c46c7520492db35d51c244fb4f74: Time spent starting tablet: real 0.107s user 0.004s sys 0.000s
I20260130 02:58:05.756263 19357 tablet_service.cc:1505] Processing CreateTablet for tablet 03025d13f0e545cfa6288677a2715343 (DEFAULT_TABLE table=default.Table_21178f657dca43a1912ce596834d1618 [id=53b79067a6034189aac5f1554df542c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:05.757225 19357 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 03025d13f0e545cfa6288677a2715343. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:05.759568 19353 tablet_service.cc:1505] Processing CreateTablet for tablet 5fbe92a2227d44d88b4531a7f8aa0e08 (DEFAULT_TABLE table=default.Table_a3ecda6d64704b7fad065a5845683ad8 [id=e93360fd2af141a28d92401ed449c19d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:05.749990 18866 catalog_manager.cc:5654] T e47b22aafe5247c4b62c4193c24f427c P 371a36d219164974abdf848841528152 reported cstate change: term changed from 0 to 1, leader changed from <none> to 371a36d219164974abdf848841528152 (127.18.100.65). New cstate: current_term: 1 leader_uuid: "371a36d219164974abdf848841528152" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } health_report { overall_health: UNKNOWN } } }
I20260130 02:58:05.761366 19356 tablet_service.cc:1505] Processing CreateTablet for tablet f41f4cff7dd4489f948700a45e9a7057 (DEFAULT_TABLE table=default.Table_ed6e2e20d89542199e5cbc7842df964d [id=047c39bd14664d1099e043b973766d32]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:05.761327 18866 catalog_manager.cc:5654] T 7b80b22bd44c4233837c951aa97e6996 P 371a36d219164974abdf848841528152 reported cstate change: term changed from 0 to 1, leader changed from <none> to 371a36d219164974abdf848841528152 (127.18.100.65). New cstate: current_term: 1 leader_uuid: "371a36d219164974abdf848841528152" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } health_report { overall_health: UNKNOWN } } }
I20260130 02:58:05.761469 19353 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5fbe92a2227d44d88b4531a7f8aa0e08. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:05.763454 18866 catalog_manager.cc:5746] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021) does not have the latest schema for tablet 7d76046ee6c8420fa4964decbe8209c1 (table default.Table_55e0ba23f44146339c7c6928440d4948 [id=4c70005d1005479fa6c3a6cb6f5ce658]). Expected version 1 got 0
I20260130 02:58:05.768471 19356 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f41f4cff7dd4489f948700a45e9a7057. 1 dirs total, 0 dirs full, 0 dirs failed
W20260130 02:58:05.771098 20199 log.cc:927] Time spent T 7b80b22bd44c4233837c951aa97e6996 P 371a36d219164974abdf848841528152: Append to log took a long time: real 0.071s user 0.000s sys 0.000s
I20260130 02:58:05.775926 19205 tablet_service.cc:1505] Processing CreateTablet for tablet 7783db3568f4434e91e9e456799cc312 (DEFAULT_TABLE table=default.Table_2d345d0f0957486c86c2cf2e34b00941 [id=4c3e0d3ecd5642a3b7fc7d20e2521418]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:05.775951 19351 tablet_service.cc:1505] Processing CreateTablet for tablet b1e8b78eff0d4a0b92059adb2de4a3ce (DEFAULT_TABLE table=default.Table_1cb6165b7dda4b659c1c379902ee4930 [id=f1342f9497b74e7a9ecad1de8082578d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:05.776899 19205 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7783db3568f4434e91e9e456799cc312. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:05.777077 19351 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b1e8b78eff0d4a0b92059adb2de4a3ce. 1 dirs total, 0 dirs full, 0 dirs failed
W20260130 02:58:05.787460 19462 outbound_call.cc:321] RPC callback for RPC call kudu.master.MasterService.TSHeartbeat -> {remote=127.18.100.125:39711, user_credentials={real_user=slave}} blocked reactor thread for 43669.2us
I20260130 02:58:05.791785 19840 tablet_bootstrap.cc:492] T b8c579ff5fa84651ba9125457d927162 P 78c2c46c7520492db35d51c244fb4f74: Bootstrap starting.
I20260130 02:58:05.800980 19807 tablet_bootstrap.cc:492] T 730bd1d7d15b4cbe945b3ef03b2455f7 P 371a36d219164974abdf848841528152: No bootstrap required, opened a new log
I20260130 02:58:05.801520 19807 ts_tablet_manager.cc:1403] T 730bd1d7d15b4cbe945b3ef03b2455f7 P 371a36d219164974abdf848841528152: Time spent bootstrapping tablet: real 0.400s user 0.011s sys 0.000s
I20260130 02:58:05.803735 19807 raft_consensus.cc:359] T 730bd1d7d15b4cbe945b3ef03b2455f7 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:05.806860 19496 tablet_service.cc:1505] Processing CreateTablet for tablet 5fbe92a2227d44d88b4531a7f8aa0e08 (DEFAULT_TABLE table=default.Table_a3ecda6d64704b7fad065a5845683ad8 [id=e93360fd2af141a28d92401ed449c19d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:05.807654 19355 tablet_service.cc:1505] Processing CreateTablet for tablet 7783db3568f4434e91e9e456799cc312 (DEFAULT_TABLE table=default.Table_2d345d0f0957486c86c2cf2e34b00941 [id=4c3e0d3ecd5642a3b7fc7d20e2521418]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:05.808012 19496 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5fbe92a2227d44d88b4531a7f8aa0e08. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:05.809350 19355 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7783db3568f4434e91e9e456799cc312. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:05.811404 19504 tablet_service.cc:1505] Processing CreateTablet for tablet f41f4cff7dd4489f948700a45e9a7057 (DEFAULT_TABLE table=default.Table_ed6e2e20d89542199e5cbc7842df964d [id=047c39bd14664d1099e043b973766d32]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:05.812461 19504 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f41f4cff7dd4489f948700a45e9a7057. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:05.815310 19495 tablet_service.cc:1505] Processing CreateTablet for tablet b1e8b78eff0d4a0b92059adb2de4a3ce (DEFAULT_TABLE table=default.Table_1cb6165b7dda4b659c1c379902ee4930 [id=f1342f9497b74e7a9ecad1de8082578d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:05.816295 19495 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b1e8b78eff0d4a0b92059adb2de4a3ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:05.818233 20186 raft_consensus.cc:493] T ceda64471a92476f9ba183d2beab30d1 P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 78c2c46c7520492db35d51c244fb4f74)
I20260130 02:58:05.818729 20186 raft_consensus.cc:515] T ceda64471a92476f9ba183d2beab30d1 P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:05.821766 19494 tablet_service.cc:1505] Processing CreateTablet for tablet 7783db3568f4434e91e9e456799cc312 (DEFAULT_TABLE table=default.Table_2d345d0f0957486c86c2cf2e34b00941 [id=4c3e0d3ecd5642a3b7fc7d20e2521418]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:05.822886 19494 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7783db3568f4434e91e9e456799cc312. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:05.826656 19501 tablet_service.cc:1505] Processing CreateTablet for tablet 78e9bc3e55fe4e4a98e8eb01da59d365 (DEFAULT_TABLE table=default.Table_5d29239ce1984d2fad02607a2bdd4dd0 [id=7e271b81bb064a949685b2411b485e3e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:05.827823 19501 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 78e9bc3e55fe4e4a98e8eb01da59d365. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:05.832907 20186 leader_election.cc:290] T ceda64471a92476f9ba183d2beab30d1 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767)
W20260130 02:58:05.836164 19463 leader_election.cc:343] T ceda64471a92476f9ba183d2beab30d1 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Not found: Tablet not found: ceda64471a92476f9ba183d2beab30d1
I20260130 02:58:05.842218 19505 tablet_service.cc:1505] Processing CreateTablet for tablet 03025d13f0e545cfa6288677a2715343 (DEFAULT_TABLE table=default.Table_21178f657dca43a1912ce596834d1618 [id=53b79067a6034189aac5f1554df542c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:05.843199 19505 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 03025d13f0e545cfa6288677a2715343. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:05.851935 19947 ts_tablet_manager.cc:1929] T 37470ce4356f4542b621e21698551209 P d870164263bb44b6b780a557daea3656: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260130 02:58:05.852293 19947 log.cc:1199] T 37470ce4356f4542b621e21698551209 P d870164263bb44b6b780a557daea3656: Deleting WAL directory at /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-2/wal/wals/37470ce4356f4542b621e21698551209
I20260130 02:58:05.852931 19947 ts_tablet_manager.cc:1950] T 37470ce4356f4542b621e21698551209 P d870164263bb44b6b780a557daea3656: Deleting consensus metadata
I20260130 02:58:05.853626 19947 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59068 (request call id 114) took 3424 ms. Trace:
W20260130 02:58:05.853863 19463 consensus_peers.cc:597] T df2c8989d6c1480c97baa23db01141eb P d870164263bb44b6b780a557daea3656 -> Peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Couldn't send request to peer 371a36d219164974abdf848841528152. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: df2c8989d6c1480c97baa23db01141eb. This is attempt 1: this message will repeat every 5th retry.
I20260130 02:58:05.853886 19947 rpcz_store.cc:276] 0130 02:58:02.429216 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:58:02.444123 (+ 14907us) service_pool.cc:225] Handling call
0130 02:58:05.469312 (+3025189us) ts_tablet_manager.cc:1151] Deleting tablet 37470ce4356f4542b621e21698551209
0130 02:58:05.469352 (+ 40us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0130 02:58:05.692779 (+223427us) tablet_metadata.cc:641] Metadata flushed
0130 02:58:05.851730 (+158951us) tablet_metadata.cc:641] Metadata flushed
0130 02:58:05.853578 (+ 1848us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3024639}
I20260130 02:58:05.855677 19947 tablet_replica.cc:333] T ceda64471a92476f9ba183d2beab30d1 P d870164263bb44b6b780a557daea3656: stopping tablet replica
I20260130 02:58:05.856410 19947 raft_consensus.cc:2243] T ceda64471a92476f9ba183d2beab30d1 P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260130 02:58:05.856911 19947 raft_consensus.cc:2272] T ceda64471a92476f9ba183d2beab30d1 P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260130 02:58:05.858776 19947 ts_tablet_manager.cc:1916] T ceda64471a92476f9ba183d2beab30d1 P d870164263bb44b6b780a557daea3656: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260130 02:58:05.828874 19199 tablet_service.cc:1505] Processing CreateTablet for tablet 6e12a8eeb3204325a9a8b7779f72ab29 (DEFAULT_TABLE table=default.Table_8c27a7d10b7847e78029f7e95321e9b6 [id=e31c1bf0c3b9457fa63efe71138c259c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:05.873001 19807 raft_consensus.cc:385] T 730bd1d7d15b4cbe945b3ef03b2455f7 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:05.873277 19807 raft_consensus.cc:740] T 730bd1d7d15b4cbe945b3ef03b2455f7 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 371a36d219164974abdf848841528152, State: Initialized, Role: FOLLOWER
I20260130 02:58:05.873515 19199 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6e12a8eeb3204325a9a8b7779f72ab29. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:05.873821 19807 consensus_queue.cc:260] T 730bd1d7d15b4cbe945b3ef03b2455f7 P 371a36d219164974abdf848841528152 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:05.874203 20189 raft_consensus.cc:493] T 481bd4cca6ee4c178873bf12392e3441 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:05.874643 20189 raft_consensus.cc:515] T 481bd4cca6ee4c178873bf12392e3441 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:05.878955 18860 catalog_manager.cc:4985] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): tablet 37470ce4356f4542b621e21698551209 (table default.Table_007e969f70d345a7bcb3dfbaabbc025d [id=77d4fa8705a3419cb0a84d3ca1bb4ce3]) successfully deleted
I20260130 02:58:05.880162 19353 tablet_service.cc:1505] Processing CreateTablet for tablet 6e12a8eeb3204325a9a8b7779f72ab29 (DEFAULT_TABLE table=default.Table_8c27a7d10b7847e78029f7e95321e9b6 [id=e31c1bf0c3b9457fa63efe71138c259c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:05.881254 19353 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6e12a8eeb3204325a9a8b7779f72ab29. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:05.881522 19840 tablet_bootstrap.cc:654] T b8c579ff5fa84651ba9125457d927162 P 78c2c46c7520492db35d51c244fb4f74: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:05.881700 19495 tablet_service.cc:1505] Processing CreateTablet for tablet 6e12a8eeb3204325a9a8b7779f72ab29 (DEFAULT_TABLE table=default.Table_8c27a7d10b7847e78029f7e95321e9b6 [id=e31c1bf0c3b9457fa63efe71138c259c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:05.882776 19495 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6e12a8eeb3204325a9a8b7779f72ab29. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:05.886111 19625 tablet_bootstrap.cc:654] T e47b22aafe5247c4b62c4193c24f427c P d870164263bb44b6b780a557daea3656: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:05.888703 19201 tablet_service.cc:1505] Processing CreateTablet for tablet a90712651a604b83ba187f893ab17711 (DEFAULT_TABLE table=default.Table_0f3e44ae93bc4c028ed3929b94a49a24 [id=3b9c62fc8bbb44ffb7d66e9604974c30]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:05.892053 19201 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a90712651a604b83ba187f893ab17711. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:05.892601 20180 raft_consensus.cc:493] T a90e92d569254bbf8a5fbf286baf5cce P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:05.894551 20180 raft_consensus.cc:515] T a90e92d569254bbf8a5fbf286baf5cce P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:05.895438 19223 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "481bd4cca6ee4c178873bf12392e3441" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152" is_pre_election: true
I20260130 02:58:05.896435 20180 leader_election.cc:290] T a90e92d569254bbf8a5fbf286baf5cce P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), d870164263bb44b6b780a557daea3656 (127.18.100.67:39571)
I20260130 02:58:05.897641 19223 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a90e92d569254bbf8a5fbf286baf5cce" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152" is_pre_election: true
I20260130 02:58:05.898205 19223 raft_consensus.cc:2468] T a90e92d569254bbf8a5fbf286baf5cce P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78c2c46c7520492db35d51c244fb4f74 in term 0.
I20260130 02:58:05.832774 19205 tablet_service.cc:1505] Processing CreateTablet for tablet 78e9bc3e55fe4e4a98e8eb01da59d365 (DEFAULT_TABLE table=default.Table_5d29239ce1984d2fad02607a2bdd4dd0 [id=7e271b81bb064a949685b2411b485e3e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:05.898505 19354 tablet_service.cc:1505] Processing CreateTablet for tablet a90712651a604b83ba187f893ab17711 (DEFAULT_TABLE table=default.Table_0f3e44ae93bc4c028ed3929b94a49a24 [id=3b9c62fc8bbb44ffb7d66e9604974c30]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:05.899294 19205 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 78e9bc3e55fe4e4a98e8eb01da59d365. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:05.899441 19354 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a90712651a604b83ba187f893ab17711. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:05.899535 20189 leader_election.cc:290] T 481bd4cca6ee4c178873bf12392e3441 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767)
I20260130 02:58:05.903004 19314 leader_election.cc:304] T a90e92d569254bbf8a5fbf286baf5cce P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 371a36d219164974abdf848841528152, 78c2c46c7520492db35d51c244fb4f74; no voters:
I20260130 02:58:05.903900 20180 raft_consensus.cc:2804] T a90e92d569254bbf8a5fbf286baf5cce P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260130 02:58:05.904438 20180 raft_consensus.cc:493] T a90e92d569254bbf8a5fbf286baf5cce P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260130 02:58:05.904819 20180 raft_consensus.cc:3060] T a90e92d569254bbf8a5fbf286baf5cce P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:05.908759 19807 ts_tablet_manager.cc:1434] T 730bd1d7d15b4cbe945b3ef03b2455f7 P 371a36d219164974abdf848841528152: Time spent starting tablet: real 0.107s user 0.004s sys 0.000s
I20260130 02:58:05.919477 19501 tablet_service.cc:1505] Processing CreateTablet for tablet e2aaaf00f53d418b87d5c7af286e4d84 (DEFAULT_TABLE table=default.Table_e84c6277dfc944059f62cbe0d6b937ee [id=747af828147149efb7b7b8d33c49286d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:05.920558 19501 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e2aaaf00f53d418b87d5c7af286e4d84. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:05.920446 19508 tablet_service.cc:1505] Processing CreateTablet for tablet a90712651a604b83ba187f893ab17711 (DEFAULT_TABLE table=default.Table_0f3e44ae93bc4c028ed3929b94a49a24 [id=3b9c62fc8bbb44ffb7d66e9604974c30]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:05.921447 19508 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a90712651a604b83ba187f893ab17711. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:05.929966 19352 tablet_service.cc:1505] Processing CreateTablet for tablet 78e9bc3e55fe4e4a98e8eb01da59d365 (DEFAULT_TABLE table=default.Table_5d29239ce1984d2fad02607a2bdd4dd0 [id=7e271b81bb064a949685b2411b485e3e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:05.931157 19352 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 78e9bc3e55fe4e4a98e8eb01da59d365. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:05.933329 19807 tablet_bootstrap.cc:492] T 6573198830ad45628e71952a7d9c00a0 P 371a36d219164974abdf848841528152: Bootstrap starting.
I20260130 02:58:05.933872 19947 ts_tablet_manager.cc:1929] T ceda64471a92476f9ba183d2beab30d1 P d870164263bb44b6b780a557daea3656: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20260130 02:58:05.935634 19198 tablet_service.cc:1505] Processing CreateTablet for tablet e2aaaf00f53d418b87d5c7af286e4d84 (DEFAULT_TABLE table=default.Table_e84c6277dfc944059f62cbe0d6b937ee [id=747af828147149efb7b7b8d33c49286d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:05.936296 19947 log.cc:1199] T ceda64471a92476f9ba183d2beab30d1 P d870164263bb44b6b780a557daea3656: Deleting WAL directory at /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-2/wal/wals/ceda64471a92476f9ba183d2beab30d1
I20260130 02:58:05.937032 19947 ts_tablet_manager.cc:1950] T ceda64471a92476f9ba183d2beab30d1 P d870164263bb44b6b780a557daea3656: Deleting consensus metadata
I20260130 02:58:05.940276 19198 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e2aaaf00f53d418b87d5c7af286e4d84. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:05.940912 19947 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59068 (request call id 115) took 2888 ms. Trace:
I20260130 02:58:05.946743 19947 rpcz_store.cc:276] 0130 02:58:03.052668 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:58:03.053189 (+ 521us) service_pool.cc:225] Handling call
0130 02:58:05.855502 (+2802313us) ts_tablet_manager.cc:1151] Deleting tablet ceda64471a92476f9ba183d2beab30d1
0130 02:58:05.855583 (+ 81us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0130 02:58:05.869313 (+ 13730us) tablet_metadata.cc:641] Metadata flushed
0130 02:58:05.933673 (+ 64360us) tablet_metadata.cc:641] Metadata flushed
0130 02:58:05.940861 (+ 7188us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2801816}
I20260130 02:58:05.948028 18860 catalog_manager.cc:4985] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): tablet ceda64471a92476f9ba183d2beab30d1 (table default.Table_be9aa687d9744ac6a2aa335772c6cc9a [id=4f59d133b0cd4773bb012145927a92a4]) successfully deleted
I20260130 02:58:05.948447 19947 tablet_replica.cc:333] T 481bd4cca6ee4c178873bf12392e3441 P d870164263bb44b6b780a557daea3656: stopping tablet replica
I20260130 02:58:05.949163 19947 raft_consensus.cc:2243] T 481bd4cca6ee4c178873bf12392e3441 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260130 02:58:05.953656 19947 raft_consensus.cc:2272] T 481bd4cca6ee4c178873bf12392e3441 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260130 02:58:05.958727 19947 ts_tablet_manager.cc:1916] T 481bd4cca6ee4c178873bf12392e3441 P d870164263bb44b6b780a557daea3656: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260130 02:58:05.972417 18873 catalog_manager.cc:5746] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767) does not have the latest schema for tablet 7d76046ee6c8420fa4964decbe8209c1 (table default.Table_55e0ba23f44146339c7c6928440d4948 [id=4c70005d1005479fa6c3a6cb6f5ce658]). Expected version 1 got 0
I20260130 02:58:05.973063 18873 catalog_manager.cc:5746] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767) does not have the latest schema for tablet 91f954f3ef4244c9a575a7421a7e75eb (table default.Table_9f83f87711d94e8394abe1925ac84ad1 [id=db23c1be42c44b6f9eb774f14d2b8fd6]). Expected version 1 got 0
I20260130 02:58:05.973634 18868 catalog_manager.cc:6199] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4: default.Table_526677c8ef4144bd9de7a54a5e78e2be [id=b16908b75587459597a978c71756451c] alter complete (version 1)
I20260130 02:58:05.983032 19494 tablet_service.cc:1552] Processing DeleteTablet for tablet 37470ce4356f4542b621e21698551209 with delete_type TABLET_DATA_DELETED (replaced by tablet 7caf7cbad5054396b5617094e4d536a0) from {username='slave'} at 127.0.0.1:59068
W20260130 02:58:05.987444 18875 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.18.100.66:59691 (request call id 22) took 2802 ms (client timeout 1000 ms). Trace:
I20260130 02:58:06.001048 19200 tablet_service.cc:1552] Processing DeleteTablet for tablet 7a902d0d571a42bb82ebffd48f7eca3f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:57:57 UTC) from {username='slave'} at 127.0.0.1:42870
I20260130 02:58:06.001547 19202 tablet_service.cc:1552] Processing DeleteTablet for tablet ceda64471a92476f9ba183d2beab30d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:58:01 UTC) from {username='slave'} at 127.0.0.1:42870
I20260130 02:58:06.002113 19203 tablet_service.cc:1552] Processing DeleteTablet for tablet 5b2976108584478a98fd90fe6594af66 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:57:58 UTC) from {username='slave'} at 127.0.0.1:42870
I20260130 02:58:06.002934 20194 raft_consensus.cc:493] T 3cf5154d8cc44f2bb12978202699c193 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:06.003368 20194 raft_consensus.cc:515] T 3cf5154d8cc44f2bb12978202699c193 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:06.003880 19203 tablet_service.cc:1552] Processing DeleteTablet for tablet 37470ce4356f4542b621e21698551209 with delete_type TABLET_DATA_DELETED (replaced by tablet 7caf7cbad5054396b5617094e4d536a0) from {username='slave'} at 127.0.0.1:42870
I20260130 02:58:06.003899 19498 tablet_service.cc:1552] Processing DeleteTablet for tablet 5b2976108584478a98fd90fe6594af66 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:57:58 UTC) from {username='slave'} at 127.0.0.1:59068
I20260130 02:58:06.004179 19350 tablet_service.cc:1552] Processing DeleteTablet for tablet d701152960b14396a6ac1bd10480d2b6 with delete_type TABLET_DATA_DELETED (replaced by tablet 481bd4cca6ee4c178873bf12392e3441) from {username='slave'} at 127.0.0.1:60110
I20260130 02:58:06.004508 19497 tablet_service.cc:1552] Processing DeleteTablet for tablet b078934be4bb4fd1915c691eeea9d00f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:58:01 UTC) from {username='slave'} at 127.0.0.1:59068
W20260130 02:58:06.004585 18861 catalog_manager.cc:4952] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021): delete failed for tablet 7a902d0d571a42bb82ebffd48f7eca3f because the tablet was not found. No further retry: Not found: Tablet not found: 7a902d0d571a42bb82ebffd48f7eca3f
I20260130 02:58:06.005111 19500 tablet_service.cc:1552] Processing DeleteTablet for tablet d701152960b14396a6ac1bd10480d2b6 with delete_type TABLET_DATA_DELETED (replaced by tablet 481bd4cca6ee4c178873bf12392e3441) from {username='slave'} at 127.0.0.1:59068
W20260130 02:58:06.005147 18861 catalog_manager.cc:4952] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021): delete failed for tablet ceda64471a92476f9ba183d2beab30d1 because the tablet was not found. No further retry: Not found: Tablet not found: ceda64471a92476f9ba183d2beab30d1
I20260130 02:58:06.005265 20194 leader_election.cc:290] T 3cf5154d8cc44f2bb12978202699c193 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), d870164263bb44b6b780a557daea3656 (127.18.100.67:39571)
W20260130 02:58:06.005656 18861 catalog_manager.cc:4952] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021): delete failed for tablet 5b2976108584478a98fd90fe6594af66 because the tablet was not found. No further retry: Not found: Tablet not found: 5b2976108584478a98fd90fe6594af66
I20260130 02:58:06.005741 19503 tablet_service.cc:1552] Processing DeleteTablet for tablet 073b5941850c4140b1751956dd7cf5ef with delete_type TABLET_DATA_DELETED (replaced by tablet 7d96e93d95f14af79cfb268626559b1e) from {username='slave'} at 127.0.0.1:59068
W20260130 02:58:05.990464 18868 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.18.100.67:47213 (request call id 20) took 2772 ms (client timeout 1000 ms). Trace:
I20260130 02:58:06.005770 19200 tablet_service.cc:1552] Processing DeleteTablet for tablet b078934be4bb4fd1915c691eeea9d00f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:58:01 UTC) from {username='slave'} at 127.0.0.1:42870
I20260130 02:58:06.006771 19502 tablet_service.cc:1552] Processing DeleteTablet for tablet bd43c5d148514836869a8ac648c8ad11 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:58:00 UTC) from {username='slave'} at 127.0.0.1:59068
I20260130 02:58:06.007439 19204 tablet_service.cc:1552] Processing DeleteTablet for tablet 96584ca174b140f7ad1a2ca698baca71 with delete_type TABLET_DATA_DELETED (replaced by tablet 127a84346b9f473d9e7f739b1e65feb8) from {username='slave'} at 127.0.0.1:42870
I20260130 02:58:06.008298 19358 tablet_service.cc:1552] Processing DeleteTablet for tablet b3f3177306914e3c8b253db54ee30fb6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:58:03 UTC) from {username='slave'} at 127.0.0.1:60110
I20260130 02:58:06.008508 19223 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3cf5154d8cc44f2bb12978202699c193" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152" is_pre_election: true
I20260130 02:58:06.009953 19197 tablet_service.cc:1552] Processing DeleteTablet for tablet 073b5941850c4140b1751956dd7cf5ef with delete_type TABLET_DATA_DELETED (replaced by tablet 7d96e93d95f14af79cfb268626559b1e) from {username='slave'} at 127.0.0.1:42870
I20260130 02:58:06.010102 19504 tablet_service.cc:1552] Processing DeleteTablet for tablet ceda64471a92476f9ba183d2beab30d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:58:01 UTC) from {username='slave'} at 127.0.0.1:59068
W20260130 02:58:06.011199 19314 leader_election.cc:343] T 3cf5154d8cc44f2bb12978202699c193 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:06.011849 19356 tablet_service.cc:1552] Processing DeleteTablet for tablet b7b596f944be4daf953b82b65d5cec2a with delete_type TABLET_DATA_DELETED (replaced by tablet 3389db58493448aea3cf00aca411047c) from {username='slave'} at 127.0.0.1:60110
I20260130 02:58:06.012431 19351 tablet_service.cc:1552] Processing DeleteTablet for tablet 5b2976108584478a98fd90fe6594af66 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:57:58 UTC) from {username='slave'} at 127.0.0.1:60110
I20260130 02:58:06.015794 19502 tablet_service.cc:1552] Processing DeleteTablet for tablet a46702bef0e6442d85e4ca49a5f7cfbb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:57:59 UTC) from {username='slave'} at 127.0.0.1:59068
I20260130 02:58:06.016618 19504 tablet_service.cc:1552] Processing DeleteTablet for tablet 7a902d0d571a42bb82ebffd48f7eca3f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:57:57 UTC) from {username='slave'} at 127.0.0.1:59068
I20260130 02:58:06.019273 18871 catalog_manager.cc:6199] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4: default.Table_42cd032296a0408797e7af3a27a103dc [id=d128767e303145a885eec3c5f714a27f] alter complete (version 2)
I20260130 02:58:06.024712 20180 raft_consensus.cc:515] T a90e92d569254bbf8a5fbf286baf5cce P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:05.992290 19357 tablet_service.cc:1505] Processing CreateTablet for tablet e2aaaf00f53d418b87d5c7af286e4d84 (DEFAULT_TABLE table=default.Table_e84c6277dfc944059f62cbe0d6b937ee [id=747af828147149efb7b7b8d33c49286d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:06.027747 19357 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e2aaaf00f53d418b87d5c7af286e4d84. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:06.027801 19807 tablet_bootstrap.cc:654] T 6573198830ad45628e71952a7d9c00a0 P 371a36d219164974abdf848841528152: Neither blocks nor log segments found. Creating new log.
W20260130 02:58:05.993821 18875 rpcz_store.cc:269] 0130 02:58:03.185278 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:58:03.296231 (+110953us) service_pool.cc:225] Handling call
0130 02:58:03.373121 (+ 76890us) spinlock_profiling.cc:243] Waited 45.7 ms on lock 0x7b5400030554. stack: 00007fb0bd3f7373 00007fb0bd3f7270 00007fb0bcf6c574 0000000000508744 00000000005086d9 00007fb0c44490c0 00007fb0c44e7bba 00007fb0c61e6eac 00007fb0c62e110e 00007fb0c3efe4a4 00007fb0c3efe432 00007fb0c3efe361 00007fb0c3efe2dc 00007fb0c3efd592 00007fb0c21dd34c 00007fb0c21dc786
0130 02:58:05.987373 (+2614252us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0130 02:58:03.912670 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0130 02:58:03.913045 (+ 375us) write_op.cc:432] Acquiring schema lock in shared mode
0130 02:58:03.913071 (+ 26us) write_op.cc:435] Acquired schema lock
0130 02:58:03.913084 (+ 13us) tablet.cc:660] Decoding operations
0130 02:58:03.913368 (+ 284us) write_op.cc:620] Acquiring the partition lock for write op
0130 02:58:03.913401 (+ 33us) write_op.cc:641] Partition lock acquired for write op
0130 02:58:03.913422 (+ 21us) tablet.cc:683] Acquiring locks for 2 operations
0130 02:58:03.927718 (+ 14296us) tablet.cc:699] Row locks acquired
0130 02:58:03.927735 (+ 17us) write_op.cc:260] PREPARE: finished
0130 02:58:03.927854 (+ 119us) write_op.cc:270] Start()
0130 02:58:03.927933 (+ 79us) write_op.cc:276] Timestamp: P: 1769741883927777 usec, L: 0
0130 02:58:03.927948 (+ 15us) op_driver.cc:348] REPLICATION: starting
0130 02:58:03.928442 (+ 494us) log.cc:844] Serialized 913 byte log entry
0130 02:58:03.986242 (+ 57800us) op_driver.cc:464] REPLICATION: finished
0130 02:58:03.988101 (+ 1859us) write_op.cc:301] APPLY: starting
0130 02:58:03.988150 (+ 49us) tablet.cc:1365] starting BulkCheckPresence
0130 02:58:03.988248 (+ 98us) tablet.cc:1368] finished BulkCheckPresence
0130 02:58:03.988348 (+ 100us) tablet.cc:1370] starting ApplyRowOperation cycle
0130 02:58:03.988577 (+ 229us) tablet.cc:1381] finished ApplyRowOperation cycle
0130 02:58:03.988705 (+ 128us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0130 02:58:03.988742 (+ 37us) write_op.cc:312] APPLY: finished
0130 02:58:03.988892 (+ 150us) log.cc:844] Serialized 29 byte log entry
0130 02:58:03.989005 (+ 113us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:58:03.989060 (+ 55us) write_op.cc:454] Released schema lock
0130 02:58:03.989087 (+ 27us) write_op.cc:341] FINISH: Updating metrics
0130 02:58:03.989275 (+ 188us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:58:03.989296 (+ 21us) write_op.cc:454] Released schema lock
Related trace 'op':
0130 02:58:04.010864 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0130 02:58:04.011447 (+ 583us) write_op.cc:432] Acquiring schema lock in shared mode
0130 02:58:04.011471 (+ 24us) write_op.cc:435] Acquired schema lock
0130 02:58:04.011505 (+ 34us) tablet.cc:660] Decoding operations
0130 02:58:04.012080 (+ 575us) write_op.cc:620] Acquiring the partition lock for write op
0130 02:58:04.012115 (+ 35us) write_op.cc:641] Partition lock acquired for write op
0130 02:58:04.012159 (+ 44us) tablet.cc:683] Acquiring locks for 1 operations
0130 02:58:04.012375 (+ 216us) tablet.cc:699] Row locks acquired
0130 02:58:04.012413 (+ 38us) write_op.cc:260] PREPARE: finished
0130 02:58:04.012569 (+ 156us) write_op.cc:270] Start()
0130 02:58:04.012711 (+ 142us) write_op.cc:276] Timestamp: P: 1769741884012464 usec, L: 0
0130 02:58:04.012726 (+ 15us) op_driver.cc:348] REPLICATION: starting
0130 02:58:04.026432 (+ 13706us) log.cc:844] Serialized 356 byte log entry
0130 02:58:04.181495 (+155063us) op_driver.cc:464] REPLICATION: finished
0130 02:58:04.185200 (+ 3705us) write_op.cc:301] APPLY: starting
0130 02:58:04.185250 (+ 50us) tablet.cc:1365] starting BulkCheckPresence
0130 02:58:04.185337 (+ 87us) tablet.cc:1368] finished BulkCheckPresence
0130 02:58:04.185417 (+ 80us) tablet.cc:1370] starting ApplyRowOperation cycle
0130 02:58:04.185542 (+ 125us) tablet.cc:1381] finished ApplyRowOperation cycle
0130 02:58:04.185685 (+ 143us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0130 02:58:04.185729 (+ 44us) write_op.cc:312] APPLY: finished
0130 02:58:04.185875 (+ 146us) log.cc:844] Serialized 23 byte log entry
0130 02:58:04.185992 (+ 117us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:58:04.186042 (+ 50us) write_op.cc:454] Released schema lock
0130 02:58:04.186070 (+ 28us) write_op.cc:341] FINISH: Updating metrics
Related trace 'op':
0130 02:58:04.289942 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0130 02:58:04.290321 (+ 379us) write_op.cc:432] Acquiring schema lock in shared mode
0130 02:58:04.290342 (+ 21us) write_op.cc:435] Acquired schema lock
0130 02:58:04.290353 (+ 11us) tablet.cc:660] Decoding operations
0130 02:58:04.293996 (+ 3643us) write_op.cc:620] Acquiring the partition lock for write op
0130 02:58:04.294032 (+ 36us) write_op.cc:641] Partition lock acquired for write op
0130 02:58:04.294054 (+ 22us) tablet.cc:683] Acquiring locks for 1 operations
0130 02:58:04.294200 (+ 146us) tablet.cc:699] Row locks acquired
0130 02:58:04.294213 (+ 13us) write_op.cc:260] PREPARE: finished
0130 02:58:04.294320 (+ 107us) write_op.cc:270] Start()
0130 02:58:04.294393 (+ 73us) write_op.cc:276] Timestamp: P: 1769741884294258 usec, L: 0
0130 02:58:04.294407 (+ 14us) op_driver.cc:348] REPLICATION: starting
0130 02:58:04.294920 (+ 513us) log.cc:844] Serialized 356 byte log entry
0130 02:58:04.364922 (+ 70002us) op_driver.cc:464] REPLICATION: finished
0130 02:58:04.368343 (+ 3421us) write_op.cc:301] APPLY: starting
0130 02:58:04.368387 (+ 44us) tablet.cc:1365] starting BulkCheckPresence
0130 02:58:04.368470 (+ 83us) tablet.cc:1368] finished BulkCheckPresence
0130 02:58:04.368570 (+ 100us) tablet.cc:1370] starting ApplyRowOperation cycle
0130 02:58:04.368690 (+ 120us) tablet.cc:1381] finished ApplyRowOperation cycle
0130 02:58:04.368806 (+ 116us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0130 02:58:04.368845 (+ 39us) write_op.cc:312] APPLY: finished
0130 02:58:04.368977 (+ 132us) log.cc:844] Serialized 23 byte log entry
0130 02:58:04.369083 (+ 106us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:58:04.369128 (+ 45us) write_op.cc:454] Released schema lock
0130 02:58:04.369158 (+ 30us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"mutex_wait_us":45201,"reported_tablets":28,"spinlock_wait_cycles":105128320,"child_traces":[["op",{"apply.queue_time_us":1751,"apply.run_cpu_time_us":1137,"apply.run_wall_time_us":1137,"num_ops":2,"prepare.queue_time_us":90,"prepare.run_cpu_time_us":2147,"prepare.run_wall_time_us":30205,"raft.queue_time_us":1397,"raft.run_cpu_time_us":18,"raft.run_wall_time_us":19,"replication_time_us":58225}],["op",{"apply.queue_time_us":3588,"apply.run_cpu_time_us":1040,"apply.run_wall_time_us":1041,"num_ops":1,"prepare.queue_time_us":5117,"prepare.run_cpu_time_us":2412,"prepare.run_wall_time_us":31745,"raft.queue_time_us":961,"raft.run_cpu_time_us":22,"raft.run_wall_time_us":23,"replication_time_us":168712,"thread_start_us":30942,"threads_started":1}],["op",{"apply.queue_time_us":3317,"apply.run_cpu_time_us":1015,"apply.run_wall_time_us":25043,"num_ops":1,"prepare.queue_time_us":103,"prepare.run_cpu_time_us":1916,"prepare.run_wall_time_us":6043,"replication_time_us":70458,"thread_start_us":28491,"threads_started":1}]]}
W20260130 02:58:06.029214 18861 catalog_manager.cc:4952] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021): delete failed for tablet 37470ce4356f4542b621e21698551209 because the tablet was not found. No further retry: Not found: Tablet not found: 37470ce4356f4542b621e21698551209
W20260130 02:58:06.029837 18861 catalog_manager.cc:4952] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021): delete failed for tablet b078934be4bb4fd1915c691eeea9d00f because the tablet was not found. No further retry: Not found: Tablet not found: b078934be4bb4fd1915c691eeea9d00f
W20260130 02:58:06.030357 18861 catalog_manager.cc:4952] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021): delete failed for tablet 96584ca174b140f7ad1a2ca698baca71 because the tablet was not found. No further retry: Not found: Tablet not found: 96584ca174b140f7ad1a2ca698baca71
W20260130 02:58:06.030879 18861 catalog_manager.cc:4952] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021): delete failed for tablet 073b5941850c4140b1751956dd7cf5ef because the tablet was not found. No further retry: Not found: Tablet not found: 073b5941850c4140b1751956dd7cf5ef
W20260130 02:58:06.031299 18867 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.18.100.67:47213 (request call id 19) took 3602 ms (client timeout 1000 ms). Trace:
W20260130 02:58:06.039781 18867 rpcz_store.cc:269] 0130 02:58:02.428603 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:58:02.433648 (+ 5045us) service_pool.cc:225] Handling call
0130 02:58:06.031227 (+3597579us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":485,"reported_tablets":27}
W20260130 02:58:06.040908 18872 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.18.100.65:51021 (request call id 25) took 3483 ms (client timeout 1000 ms). Trace:
W20260130 02:58:06.041260 18872 rpcz_store.cc:269] 0130 02:58:02.557179 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:58:02.584285 (+ 27106us) service_pool.cc:225] Handling call
0130 02:58:06.040843 (+3456558us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0130 02:58:03.874110 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0130 02:58:03.874455 (+ 345us) write_op.cc:432] Acquiring schema lock in shared mode
0130 02:58:03.874475 (+ 20us) write_op.cc:435] Acquired schema lock
0130 02:58:03.874486 (+ 11us) tablet.cc:660] Decoding operations
0130 02:58:03.874795 (+ 309us) write_op.cc:620] Acquiring the partition lock for write op
0130 02:58:03.874827 (+ 32us) write_op.cc:641] Partition lock acquired for write op
0130 02:58:03.874846 (+ 19us) tablet.cc:683] Acquiring locks for 1 operations
0130 02:58:03.874993 (+ 147us) tablet.cc:699] Row locks acquired
0130 02:58:03.875006 (+ 13us) write_op.cc:260] PREPARE: finished
0130 02:58:03.875106 (+ 100us) write_op.cc:270] Start()
0130 02:58:03.875177 (+ 71us) write_op.cc:276] Timestamp: P: 1769741883875040 usec, L: 0
0130 02:58:03.875190 (+ 13us) op_driver.cc:348] REPLICATION: starting
0130 02:58:03.875613 (+ 423us) log.cc:844] Serialized 356 byte log entry
0130 02:58:03.945773 (+ 70160us) op_driver.cc:464] REPLICATION: finished
0130 02:58:03.947650 (+ 1877us) write_op.cc:301] APPLY: starting
0130 02:58:03.947692 (+ 42us) tablet.cc:1365] starting BulkCheckPresence
0130 02:58:03.947774 (+ 82us) tablet.cc:1368] finished BulkCheckPresence
0130 02:58:03.947865 (+ 91us) tablet.cc:1370] starting ApplyRowOperation cycle
0130 02:58:03.948010 (+ 145us) tablet.cc:1381] finished ApplyRowOperation cycle
0130 02:58:03.948123 (+ 113us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0130 02:58:03.948164 (+ 41us) write_op.cc:312] APPLY: finished
0130 02:58:03.948309 (+ 145us) log.cc:844] Serialized 23 byte log entry
0130 02:58:03.948422 (+ 113us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:58:03.948472 (+ 50us) write_op.cc:454] Released schema lock
0130 02:58:03.948499 (+ 27us) write_op.cc:341] FINISH: Updating metrics
0130 02:58:03.948685 (+ 186us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:58:03.948708 (+ 23us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":23,"child_traces":[["op",{"apply.queue_time_us":1766,"apply.run_cpu_time_us":1008,"apply.run_wall_time_us":1008,"num_ops":1,"prepare.queue_time_us":5323,"prepare.run_cpu_time_us":1666,"prepare.run_wall_time_us":2529,"replication_time_us":70512}]]}
I20260130 02:58:06.043989 18872 master_service.cc:954] ReplaceTablet: received request to replace tablet 927c90b84d7449efa36bc5e2035bf0da from {username='slave'} at 127.0.0.1:50284
W20260130 02:58:06.044333 18860 catalog_manager.cc:4952] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767): delete failed for tablet d701152960b14396a6ac1bd10480d2b6 because the tablet was not found. No further retry: Not found: Tablet not found: d701152960b14396a6ac1bd10480d2b6
I20260130 02:58:06.049521 19378 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "481bd4cca6ee4c178873bf12392e3441" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74" is_pre_election: true
I20260130 02:58:06.050493 18867 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50284:
table { table_name: "default.Table_f7aab7c003284493810119a94032eca0" } new_table_name: "default.Table_3413dae2efac41488448e6e5975cf710" modify_external_catalogs: true
W20260130 02:58:06.051594 19462 consensus_peers.cc:597] T df2c8989d6c1480c97baa23db01141eb P d870164263bb44b6b780a557daea3656 -> Peer 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767): Couldn't send request to peer 78c2c46c7520492db35d51c244fb4f74. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: df2c8989d6c1480c97baa23db01141eb. This is attempt 1: this message will repeat every 5th retry.
W20260130 02:58:06.006302 18868 rpcz_store.cc:269] 0130 02:58:03.217973 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:58:03.331785 (+113812us) service_pool.cc:225] Handling call
0130 02:58:03.361389 (+ 29604us) spinlock_profiling.cc:243] Waited 24.9 ms on lock 0x7b5400030554. stack: 00007fb0bd3f7373 00007fb0bd3f7270 00007fb0bcf6c574 0000000000508744 00000000005086d9 00007fb0c44490c0 00007fb0c44e7bba 00007fb0c61e6eac 00007fb0c62e110e 00007fb0c3efe4a4 00007fb0c3efe432 00007fb0c3efe361 00007fb0c3efe2dc 00007fb0c3efd592 00007fb0c21dd34c 00007fb0c21dc786
0130 02:58:05.990397 (+2629008us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0130 02:58:04.394232 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0130 02:58:04.396075 (+ 1843us) write_op.cc:432] Acquiring schema lock in shared mode
0130 02:58:04.396100 (+ 25us) write_op.cc:435] Acquired schema lock
0130 02:58:04.396112 (+ 12us) tablet.cc:660] Decoding operations
0130 02:58:04.396373 (+ 261us) write_op.cc:620] Acquiring the partition lock for write op
0130 02:58:04.396407 (+ 34us) write_op.cc:641] Partition lock acquired for write op
0130 02:58:04.396428 (+ 21us) tablet.cc:683] Acquiring locks for 1 operations
0130 02:58:04.396566 (+ 138us) tablet.cc:699] Row locks acquired
0130 02:58:04.396580 (+ 14us) write_op.cc:260] PREPARE: finished
0130 02:58:04.396687 (+ 107us) write_op.cc:270] Start()
0130 02:58:04.396774 (+ 87us) write_op.cc:276] Timestamp: P: 1769741884396614 usec, L: 0
0130 02:58:04.396789 (+ 15us) op_driver.cc:348] REPLICATION: starting
0130 02:58:04.397278 (+ 489us) log.cc:844] Serialized 356 byte log entry
0130 02:58:04.434473 (+ 37195us) op_driver.cc:464] REPLICATION: finished
0130 02:58:04.435487 (+ 1014us) write_op.cc:301] APPLY: starting
0130 02:58:04.435536 (+ 49us) tablet.cc:1365] starting BulkCheckPresence
0130 02:58:04.435622 (+ 86us) tablet.cc:1368] finished BulkCheckPresence
0130 02:58:04.435709 (+ 87us) tablet.cc:1370] starting ApplyRowOperation cycle
0130 02:58:04.435872 (+ 163us) tablet.cc:1381] finished ApplyRowOperation cycle
0130 02:58:04.435989 (+ 117us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0130 02:58:04.436032 (+ 43us) write_op.cc:312] APPLY: finished
0130 02:58:04.436193 (+ 161us) log.cc:844] Serialized 23 byte log entry
0130 02:58:04.436325 (+ 132us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:58:04.436386 (+ 61us) write_op.cc:454] Released schema lock
0130 02:58:04.436416 (+ 30us) write_op.cc:341] FINISH: Updating metrics
0130 02:58:04.436759 (+ 343us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:58:04.436783 (+ 24us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":28,"spinlock_wait_cycles":57198336,"child_traces":[["op",{"apply.queue_time_us":877,"apply.run_cpu_time_us":1119,"apply.run_wall_time_us":1230,"num_ops":1,"prepare.queue_time_us":17845,"prepare.run_cpu_time_us":1844,"prepare.run_wall_time_us":4463,"replication_time_us":37596}]]}
I20260130 02:58:06.054714 18868 master_service.cc:954] ReplaceTablet: received request to replace tablet 2112b0402de5464cb1caf2c2abb67d78 from {username='slave'} at 127.0.0.1:50284
I20260130 02:58:06.056015 18866 catalog_manager.cc:6199] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4: default.Table_f359d249f8274f319f5ff4e4ea89dd22 [id=27adf356fd004597a54c7a9887dd3ca8] alter complete (version 4)
I20260130 02:58:06.054749 19947 ts_tablet_manager.cc:1929] T 481bd4cca6ee4c178873bf12392e3441 P d870164263bb44b6b780a557daea3656: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260130 02:58:06.061733 19947 log.cc:1199] T 481bd4cca6ee4c178873bf12392e3441 P d870164263bb44b6b780a557daea3656: Deleting WAL directory at /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-2/wal/wals/481bd4cca6ee4c178873bf12392e3441
I20260130 02:58:06.063069 19840 tablet_bootstrap.cc:492] T b8c579ff5fa84651ba9125457d927162 P 78c2c46c7520492db35d51c244fb4f74: No bootstrap required, opened a new log
I20260130 02:58:06.063473 19840 ts_tablet_manager.cc:1403] T b8c579ff5fa84651ba9125457d927162 P 78c2c46c7520492db35d51c244fb4f74: Time spent bootstrapping tablet: real 0.272s user 0.007s sys 0.002s
I20260130 02:58:06.067011 19840 raft_consensus.cc:359] T b8c579ff5fa84651ba9125457d927162 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:06.067683 19840 raft_consensus.cc:385] T b8c579ff5fa84651ba9125457d927162 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:06.067939 19840 raft_consensus.cc:740] T b8c579ff5fa84651ba9125457d927162 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78c2c46c7520492db35d51c244fb4f74, State: Initialized, Role: FOLLOWER
I20260130 02:58:06.068572 19840 consensus_queue.cc:260] T b8c579ff5fa84651ba9125457d927162 P 78c2c46c7520492db35d51c244fb4f74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:06.069583 18870 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:50284 (request call id 181) took 1276 ms. Trace:
I20260130 02:58:06.073918 19205 tablet_service.cc:1552] Processing DeleteTablet for tablet d701152960b14396a6ac1bd10480d2b6 with delete_type TABLET_DATA_DELETED (replaced by tablet 481bd4cca6ee4c178873bf12392e3441) from {username='slave'} at 127.0.0.1:42870
I20260130 02:58:06.078735 18870 rpcz_store.cc:276] 0130 02:58:04.792965 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:58:04.793205 (+ 240us) service_pool.cc:225] Handling call
0130 02:58:04.794068 (+ 863us) catalog_manager.cc:2393] Looking up, locking, and authorizing table
0130 02:58:06.028184 (+1234116us) catalog_manager.cc:3708] Apply alter schema
0130 02:58:06.028736 (+ 552us) catalog_manager.cc:3740] Acquired catalog manager lock
0130 02:58:06.029024 (+ 288us) catalog_manager.cc:3898] Updating metadata on disk
0130 02:58:06.068411 (+ 39387us) catalog_manager.cc:3929] Committing alterations to in-memory state
0130 02:58:06.069523 (+ 1112us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0130 02:58:06.042394 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0130 02:58:06.042792 (+ 398us) write_op.cc:432] Acquiring schema lock in shared mode
0130 02:58:06.042815 (+ 23us) write_op.cc:435] Acquired schema lock
0130 02:58:06.042826 (+ 11us) tablet.cc:660] Decoding operations
0130 02:58:06.043096 (+ 270us) write_op.cc:620] Acquiring the partition lock for write op
0130 02:58:06.043132 (+ 36us) write_op.cc:641] Partition lock acquired for write op
0130 02:58:06.043153 (+ 21us) tablet.cc:683] Acquiring locks for 1 operations
0130 02:58:06.043282 (+ 129us) tablet.cc:699] Row locks acquired
0130 02:58:06.043294 (+ 12us) write_op.cc:260] PREPARE: finished
0130 02:58:06.043394 (+ 100us) write_op.cc:270] Start()
0130 02:58:06.043462 (+ 68us) write_op.cc:276] Timestamp: P: 1769741886043318 usec, L: 0
0130 02:58:06.043476 (+ 14us) op_driver.cc:348] REPLICATION: starting
0130 02:58:06.043910 (+ 434us) log.cc:844] Serialized 405 byte log entry
0130 02:58:06.066642 (+ 22732us) op_driver.cc:464] REPLICATION: finished
0130 02:58:06.066846 (+ 204us) write_op.cc:301] APPLY: starting
0130 02:58:06.066896 (+ 50us) tablet.cc:1365] starting BulkCheckPresence
0130 02:58:06.066985 (+ 89us) tablet.cc:1368] finished BulkCheckPresence
0130 02:58:06.067080 (+ 95us) tablet.cc:1370] starting ApplyRowOperation cycle
0130 02:58:06.067249 (+ 169us) tablet.cc:1381] finished ApplyRowOperation cycle
0130 02:58:06.067521 (+ 272us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0130 02:58:06.067568 (+ 47us) write_op.cc:312] APPLY: finished
0130 02:58:06.067729 (+ 161us) log.cc:844] Serialized 23 byte log entry
0130 02:58:06.067866 (+ 137us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:58:06.067924 (+ 58us) write_op.cc:454] Released schema lock
0130 02:58:06.067957 (+ 33us) write_op.cc:341] FINISH: Updating metrics
0130 02:58:06.068168 (+ 211us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:58:06.068191 (+ 23us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":76,"apply.run_cpu_time_us":1293,"apply.run_wall_time_us":1293,"num_ops":1,"prepare.queue_time_us":678,"prepare.run_cpu_time_us":1697,"prepare.run_wall_time_us":1696,"replication_time_us":23066}]]}
W20260130 02:58:06.075603 18861 catalog_manager.cc:4952] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021): delete failed for tablet d701152960b14396a6ac1bd10480d2b6 because the tablet was not found. No further retry: Not found: Tablet not found: d701152960b14396a6ac1bd10480d2b6
W20260130 02:58:06.081979 18860 catalog_manager.cc:4952] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767): delete failed for tablet b3f3177306914e3c8b253db54ee30fb6 because the tablet was not found. No further retry: Not found: Tablet not found: b3f3177306914e3c8b253db54ee30fb6
I20260130 02:58:06.082515 18872 catalog_manager.cc:5941] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4: Sending DeleteTablet for 3 replicas of tablet 927c90b84d7449efa36bc5e2035bf0da
W20260130 02:58:06.082592 18860 catalog_manager.cc:4952] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767): delete failed for tablet b7b596f944be4daf953b82b65d5cec2a because the tablet was not found. No further retry: Not found: Tablet not found: b7b596f944be4daf953b82b65d5cec2a
W20260130 02:58:06.083218 18860 catalog_manager.cc:4952] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767): delete failed for tablet 5b2976108584478a98fd90fe6594af66 because the tablet was not found. No further retry: Not found: Tablet not found: 5b2976108584478a98fd90fe6594af66
I20260130 02:58:06.085455 19495 tablet_service.cc:1552] Processing DeleteTablet for tablet 927c90b84d7449efa36bc5e2035bf0da with delete_type TABLET_DATA_DELETED (replaced by tablet 25fd48e693424537b91ba7c3fcc96878) from {username='slave'} at 127.0.0.1:59068
I20260130 02:58:06.086066 19205 tablet_service.cc:1552] Processing DeleteTablet for tablet 927c90b84d7449efa36bc5e2035bf0da with delete_type TABLET_DATA_DELETED (replaced by tablet 25fd48e693424537b91ba7c3fcc96878) from {username='slave'} at 127.0.0.1:42870
I20260130 02:58:06.086731 19754 tablet_replica.cc:333] T 927c90b84d7449efa36bc5e2035bf0da P 371a36d219164974abdf848841528152: stopping tablet replica
I20260130 02:58:06.087448 19754 raft_consensus.cc:2243] T 927c90b84d7449efa36bc5e2035bf0da P 371a36d219164974abdf848841528152 [term 1 LEADER]: Raft consensus shutting down.
I20260130 02:58:06.088220 19754 raft_consensus.cc:2272] T 927c90b84d7449efa36bc5e2035bf0da P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260130 02:58:06.090180 19754 ts_tablet_manager.cc:1916] T 927c90b84d7449efa36bc5e2035bf0da P 371a36d219164974abdf848841528152: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260130 02:58:06.093598 19350 tablet_service.cc:1552] Processing DeleteTablet for tablet 927c90b84d7449efa36bc5e2035bf0da with delete_type TABLET_DATA_DELETED (replaced by tablet 25fd48e693424537b91ba7c3fcc96878) from {username='slave'} at 127.0.0.1:60110
W20260130 02:58:06.093815 19462 leader_election.cc:343] T ceda64471a92476f9ba183d2beab30d1 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767): Not found: Tablet not found: ceda64471a92476f9ba183d2beab30d1
I20260130 02:58:06.094233 19462 leader_election.cc:304] T ceda64471a92476f9ba183d2beab30d1 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d870164263bb44b6b780a557daea3656; no voters: 371a36d219164974abdf848841528152, 78c2c46c7520492db35d51c244fb4f74
W20260130 02:58:06.099367 19462 raft_consensus.cc:2686] T ceda64471a92476f9ba183d2beab30d1 P d870164263bb44b6b780a557daea3656: unable to run election callback: Service unavailable: Thread pool token was shut down
I20260130 02:58:06.099488 19947 ts_tablet_manager.cc:1950] T 481bd4cca6ee4c178873bf12392e3441 P d870164263bb44b6b780a557daea3656: Deleting consensus metadata
I20260130 02:58:06.101346 19462 leader_election.cc:304] T 481bd4cca6ee4c178873bf12392e3441 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d870164263bb44b6b780a557daea3656; no voters: 371a36d219164974abdf848841528152, 78c2c46c7520492db35d51c244fb4f74
W20260130 02:58:06.102222 19462 raft_consensus.cc:2686] T 481bd4cca6ee4c178873bf12392e3441 P d870164263bb44b6b780a557daea3656: unable to run election callback: Service unavailable: Thread pool token was shut down
I20260130 02:58:06.104833 19528 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3cf5154d8cc44f2bb12978202699c193" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
I20260130 02:58:06.106559 19525 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a90e92d569254bbf8a5fbf286baf5cce" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
I20260130 02:58:06.111608 20208 tablet_replica.cc:333] T 927c90b84d7449efa36bc5e2035bf0da P 78c2c46c7520492db35d51c244fb4f74: stopping tablet replica
I20260130 02:58:06.112543 20208 raft_consensus.cc:2243] T 927c90b84d7449efa36bc5e2035bf0da P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260130 02:58:06.114687 18872 catalog_manager.cc:6724] ReplaceTablet: tablet 927c90b84d7449efa36bc5e2035bf0da deleted and replaced by tablet 25fd48e693424537b91ba7c3fcc96878
I20260130 02:58:06.115698 18872 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50284:
table { table_name: "default.Table_55e0ba23f44146339c7c6928440d4948" } modify_external_catalogs: true
I20260130 02:58:06.116206 18872 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50284:
table { table_name: "default.Table_55e0ba23f44146339c7c6928440d4948" } modify_external_catalogs: true
I20260130 02:58:06.122241 19947 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59068 (request call id 117) took 3042 ms. Trace:
I20260130 02:58:06.122622 19947 rpcz_store.cc:276] 0130 02:58:03.079738 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:58:03.079872 (+ 134us) service_pool.cc:225] Handling call
0130 02:58:05.948232 (+2868360us) ts_tablet_manager.cc:1151] Deleting tablet 481bd4cca6ee4c178873bf12392e3441
0130 02:58:05.948273 (+ 41us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0130 02:58:05.976294 (+ 28021us) tablet_metadata.cc:641] Metadata flushed
0130 02:58:06.054501 (+ 78207us) tablet_metadata.cc:641] Metadata flushed
0130 02:58:06.120856 (+ 66355us) spinlock_profiling.cc:243] Waited 20.4 ms on lock 0x7b5800000098. stack: 00007f22d9409373 00007f22d9409270 00007f22d8f7e574 0000000000508744 00000000005086d9 00007f22d927b8b1 00007f22d93fddb1 00007f22d93fe3e2 00007f22e2258189 00007f22e14ab108 00007f22e1572462 00007f22e15a2f80 00007f22e158954d 00007f22e15894f9 00007f22e1589489 00007f22e1589451
0130 02:58:06.122177 (+ 1321us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":82337,"spinlock_wait_cycles":46858112,"tablet-delete.queue_time_us":2845475}
I20260130 02:58:06.125032 20180 leader_election.cc:290] T a90e92d569254bbf8a5fbf286baf5cce P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 election: Requested vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), d870164263bb44b6b780a557daea3656 (127.18.100.67:39571)
I20260130 02:58:06.125429 19223 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a90e92d569254bbf8a5fbf286baf5cce" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152"
I20260130 02:58:06.126112 19223 raft_consensus.cc:3060] T a90e92d569254bbf8a5fbf286baf5cce P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:06.131981 20176 consensus_queue.cc:1048] T 2ca0df71836145748cada4e4559a5656 P 371a36d219164974abdf848841528152 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260130 02:58:06.135672 19840 ts_tablet_manager.cc:1434] T b8c579ff5fa84651ba9125457d927162 P 78c2c46c7520492db35d51c244fb4f74: Time spent starting tablet: real 0.072s user 0.002s sys 0.004s
I20260130 02:58:06.136693 19840 tablet_bootstrap.cc:492] T 54be22352e404ff9b798a075693f7f21 P 78c2c46c7520492db35d51c244fb4f74: Bootstrap starting.
I20260130 02:58:06.137713 19223 raft_consensus.cc:2468] T a90e92d569254bbf8a5fbf286baf5cce P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78c2c46c7520492db35d51c244fb4f74 in term 1.
I20260130 02:58:06.152307 19947 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59068 (request call id 122) took 2556 ms. Trace:
I20260130 02:58:06.155288 19947 rpcz_store.cc:276] 0130 02:58:03.595743 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:58:03.595874 (+ 131us) service_pool.cc:225] Handling call
0130 02:58:06.151978 (+2556104us) ts_tablet_manager.cc:1151] Deleting tablet 6573198830ad45628e71952a7d9c00a0
0130 02:58:06.152032 (+ 54us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0130 02:58:06.152254 (+ 222us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2555539}
I20260130 02:58:06.152925 19526 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a90e92d569254bbf8a5fbf286baf5cce" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656"
I20260130 02:58:06.160041 19947 tablet_replica.cc:333] T b7b596f944be4daf953b82b65d5cec2a P d870164263bb44b6b780a557daea3656: stopping tablet replica
I20260130 02:58:06.138859 20215 tablet.cc:1722] T 7d76046ee6c8420fa4964decbe8209c1 P 371a36d219164974abdf848841528152: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260130 02:58:06.160339 20214 tablet.cc:1722] T 7d76046ee6c8420fa4964decbe8209c1 P d870164263bb44b6b780a557daea3656: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260130 02:58:06.162612 19947 raft_consensus.cc:2243] T b7b596f944be4daf953b82b65d5cec2a P d870164263bb44b6b780a557daea3656 [term 1 LEADER]: Raft consensus shutting down.
I20260130 02:58:06.163599 19947 raft_consensus.cc:2272] T b7b596f944be4daf953b82b65d5cec2a P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260130 02:58:06.166981 20208 raft_consensus.cc:2272] T 927c90b84d7449efa36bc5e2035bf0da P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260130 02:58:06.167698 18860 catalog_manager.cc:4985] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): tablet 481bd4cca6ee4c178873bf12392e3441 (table default.Table_e3b63d7d5d874900aac850654d98d62e [id=bc8cc55916fa4a41b78019f59e6fbe39]) successfully deleted
I20260130 02:58:06.168460 19625 tablet_bootstrap.cc:492] T e47b22aafe5247c4b62c4193c24f427c P d870164263bb44b6b780a557daea3656: No bootstrap required, opened a new log
I20260130 02:58:06.152343 20176 raft_consensus.cc:493] T 730bd1d7d15b4cbe945b3ef03b2455f7 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:06.170243 19625 ts_tablet_manager.cc:1403] T e47b22aafe5247c4b62c4193c24f427c P d870164263bb44b6b780a557daea3656: Time spent bootstrapping tablet: real 0.439s user 0.011s sys 0.001s
I20260130 02:58:06.174990 20176 raft_consensus.cc:515] T 730bd1d7d15b4cbe945b3ef03b2455f7 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:06.176779 20176 leader_election.cc:290] T 730bd1d7d15b4cbe945b3ef03b2455f7 P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d870164263bb44b6b780a557daea3656 (127.18.100.67:39571), 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767)
I20260130 02:58:06.170168 19314 leader_election.cc:304] T a90e92d569254bbf8a5fbf286baf5cce P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 371a36d219164974abdf848841528152, 78c2c46c7520492db35d51c244fb4f74; no voters:
I20260130 02:58:06.178601 19377 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "730bd1d7d15b4cbe945b3ef03b2455f7" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74" is_pre_election: true
I20260130 02:58:06.179162 19377 raft_consensus.cc:2468] T 730bd1d7d15b4cbe945b3ef03b2455f7 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 371a36d219164974abdf848841528152 in term 0.
I20260130 02:58:06.179184 19526 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "730bd1d7d15b4cbe945b3ef03b2455f7" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
I20260130 02:58:06.180871 20009 raft_consensus.cc:2804] T a90e92d569254bbf8a5fbf286baf5cce P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Leader election won for term 1
I20260130 02:58:06.180810 19159 leader_election.cc:304] T 730bd1d7d15b4cbe945b3ef03b2455f7 P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 371a36d219164974abdf848841528152, 78c2c46c7520492db35d51c244fb4f74; no voters:
I20260130 02:58:06.181316 20009 raft_consensus.cc:697] T a90e92d569254bbf8a5fbf286baf5cce P 78c2c46c7520492db35d51c244fb4f74 [term 1 LEADER]: Becoming Leader. State: Replica: 78c2c46c7520492db35d51c244fb4f74, State: Running, Role: LEADER
W20260130 02:58:06.181528 19159 leader_election.cc:343] T 730bd1d7d15b4cbe945b3ef03b2455f7 P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:06.182533 20178 raft_consensus.cc:2804] T 730bd1d7d15b4cbe945b3ef03b2455f7 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260130 02:58:06.182157 20009 consensus_queue.cc:237] T a90e92d569254bbf8a5fbf286baf5cce P 78c2c46c7520492db35d51c244fb4f74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:06.182889 20178 raft_consensus.cc:493] T 730bd1d7d15b4cbe945b3ef03b2455f7 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260130 02:58:06.183166 20178 raft_consensus.cc:3060] T 730bd1d7d15b4cbe945b3ef03b2455f7 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:06.184197 18868 catalog_manager.cc:5941] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4: Sending DeleteTablet for 3 replicas of tablet 2112b0402de5464cb1caf2c2abb67d78
I20260130 02:58:06.185202 20208 ts_tablet_manager.cc:1916] T 927c90b84d7449efa36bc5e2035bf0da P 78c2c46c7520492db35d51c244fb4f74: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260130 02:58:06.186089 18868 catalog_manager.cc:6724] ReplaceTablet: tablet 2112b0402de5464cb1caf2c2abb67d78 deleted and replaced by tablet 838e49e5e8874db393bdc15a1c92f87e
I20260130 02:58:06.186904 18868 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50284:
table { table_name: "default.Table_bb334df0beee420e883c3225d93892c0" } modify_external_catalogs: true
I20260130 02:58:06.180642 19947 ts_tablet_manager.cc:1916] T b7b596f944be4daf953b82b65d5cec2a P d870164263bb44b6b780a557daea3656: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260130 02:58:06.187420 18868 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50284:
table { table_name: "default.Table_bb334df0beee420e883c3225d93892c0" } modify_external_catalogs: true
W20260130 02:58:06.182382 18860 catalog_manager.cc:4965] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): delete failed for tablet 6573198830ad45628e71952a7d9c00a0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6573198830ad45628e71952a7d9c00a0 already in progress: creating tablet
W20260130 02:58:06.202716 19313 leader_election.cc:343] T 3cf5154d8cc44f2bb12978202699c193 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:06.203119 19313 leader_election.cc:304] T 3cf5154d8cc44f2bb12978202699c193 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78c2c46c7520492db35d51c244fb4f74; no voters: 371a36d219164974abdf848841528152, d870164263bb44b6b780a557daea3656
I20260130 02:58:06.206211 19508 tablet_service.cc:1552] Processing DeleteTablet for tablet 2112b0402de5464cb1caf2c2abb67d78 with delete_type TABLET_DATA_DELETED (replaced by tablet 838e49e5e8874db393bdc15a1c92f87e) from {username='slave'} at 127.0.0.1:59068
I20260130 02:58:06.202478 20178 raft_consensus.cc:515] T 730bd1d7d15b4cbe945b3ef03b2455f7 P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:06.216674 19807 tablet_bootstrap.cc:492] T 6573198830ad45628e71952a7d9c00a0 P 371a36d219164974abdf848841528152: No bootstrap required, opened a new log
I20260130 02:58:06.217098 19807 ts_tablet_manager.cc:1403] T 6573198830ad45628e71952a7d9c00a0 P 371a36d219164974abdf848841528152: Time spent bootstrapping tablet: real 0.284s user 0.016s sys 0.000s
I20260130 02:58:06.217355 20178 leader_election.cc:290] T 730bd1d7d15b4cbe945b3ef03b2455f7 P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 election: Requested vote from peers d870164263bb44b6b780a557daea3656 (127.18.100.67:39571), 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767)
I20260130 02:58:06.219939 19198 tablet_service.cc:1552] Processing DeleteTablet for tablet 2112b0402de5464cb1caf2c2abb67d78 with delete_type TABLET_DATA_DELETED (replaced by tablet 838e49e5e8874db393bdc15a1c92f87e) from {username='slave'} at 127.0.0.1:42870
I20260130 02:58:06.222648 20194 raft_consensus.cc:2749] T 3cf5154d8cc44f2bb12978202699c193 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260130 02:58:06.234308 19525 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "730bd1d7d15b4cbe945b3ef03b2455f7" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656"
W20260130 02:58:06.235388 19159 leader_election.cc:343] T 730bd1d7d15b4cbe945b3ef03b2455f7 P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:06.238093 20179 consensus_queue.cc:1048] T 2ca0df71836145748cada4e4559a5656 P 371a36d219164974abdf848841528152 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260130 02:58:06.239434 19807 raft_consensus.cc:359] T 6573198830ad45628e71952a7d9c00a0 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:06.240047 19807 raft_consensus.cc:385] T 6573198830ad45628e71952a7d9c00a0 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:06.240298 19807 raft_consensus.cc:740] T 6573198830ad45628e71952a7d9c00a0 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 371a36d219164974abdf848841528152, State: Initialized, Role: FOLLOWER
I20260130 02:58:06.232462 19355 tablet_service.cc:1552] Processing DeleteTablet for tablet 2112b0402de5464cb1caf2c2abb67d78 with delete_type TABLET_DATA_DELETED (replaced by tablet 838e49e5e8874db393bdc15a1c92f87e) from {username='slave'} at 127.0.0.1:60110
I20260130 02:58:06.244437 19625 raft_consensus.cc:359] T e47b22aafe5247c4b62c4193c24f427c P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:06.245083 19625 raft_consensus.cc:385] T e47b22aafe5247c4b62c4193c24f427c P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:06.236215 20222 tablet.cc:1722] T f2d01bfdc9c4494386bd02bed655fe63 P 371a36d219164974abdf848841528152: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20260130 02:58:06.245328 19625 raft_consensus.cc:740] T e47b22aafe5247c4b62c4193c24f427c P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d870164263bb44b6b780a557daea3656, State: Initialized, Role: FOLLOWER
I20260130 02:58:06.234032 19377 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "730bd1d7d15b4cbe945b3ef03b2455f7" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74"
I20260130 02:58:06.246565 19377 raft_consensus.cc:3060] T 730bd1d7d15b4cbe945b3ef03b2455f7 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:06.254935 19807 consensus_queue.cc:260] T 6573198830ad45628e71952a7d9c00a0 P 371a36d219164974abdf848841528152 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:06.258219 19754 ts_tablet_manager.cc:1929] T 927c90b84d7449efa36bc5e2035bf0da P 371a36d219164974abdf848841528152: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260130 02:58:06.258608 19754 log.cc:1199] T 927c90b84d7449efa36bc5e2035bf0da P 371a36d219164974abdf848841528152: Deleting WAL directory at /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-0/wal/wals/927c90b84d7449efa36bc5e2035bf0da
I20260130 02:58:06.259313 19754 ts_tablet_manager.cc:1950] T 927c90b84d7449efa36bc5e2035bf0da P 371a36d219164974abdf848841528152: Deleting consensus metadata
I20260130 02:58:06.261741 18861 catalog_manager.cc:4985] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021): tablet 927c90b84d7449efa36bc5e2035bf0da (table default.Table_ebc8b8379c274c18b0a06a919dbe0ee0 [id=28156e0843674fce88f5914a3e4195ae]) successfully deleted
W20260130 02:58:06.237998 19313 leader_election.cc:343] T a90e92d569254bbf8a5fbf286baf5cce P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): Illegal state: must be running to vote when last-logged opid is not known
W20260130 02:58:06.263437 19313 leader_election.cc:343] T a90e92d569254bbf8a5fbf286baf5cce P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:06.262935 19625 consensus_queue.cc:260] T e47b22aafe5247c4b62c4193c24f427c P d870164263bb44b6b780a557daea3656 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:06.266559 19807 ts_tablet_manager.cc:1434] T 6573198830ad45628e71952a7d9c00a0 P 371a36d219164974abdf848841528152: Time spent starting tablet: real 0.049s user 0.005s sys 0.000s
I20260130 02:58:06.269596 20221 tablet.cc:1722] T f2d01bfdc9c4494386bd02bed655fe63 P d870164263bb44b6b780a557daea3656: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20260130 02:58:06.284389 19807 tablet_bootstrap.cc:492] T 111287b5557d4abda1c5cd560fb0d019 P 371a36d219164974abdf848841528152: Bootstrap starting.
I20260130 02:58:06.289191 19840 tablet_bootstrap.cc:654] T 54be22352e404ff9b798a075693f7f21 P 78c2c46c7520492db35d51c244fb4f74: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:06.290923 19625 ts_tablet_manager.cc:1434] T e47b22aafe5247c4b62c4193c24f427c P d870164263bb44b6b780a557daea3656: Time spent starting tablet: real 0.120s user 0.006s sys 0.001s
I20260130 02:58:06.291556 19754 tablet_replica.cc:333] T 2112b0402de5464cb1caf2c2abb67d78 P 371a36d219164974abdf848841528152: stopping tablet replica
I20260130 02:58:06.292304 19754 raft_consensus.cc:2243] T 2112b0402de5464cb1caf2c2abb67d78 P 371a36d219164974abdf848841528152 [term 1 LEADER]: Raft consensus shutting down.
I20260130 02:58:06.293192 19754 raft_consensus.cc:2272] T 2112b0402de5464cb1caf2c2abb67d78 P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260130 02:58:06.300137 19625 tablet_bootstrap.cc:492] T 91f954f3ef4244c9a575a7421a7e75eb P d870164263bb44b6b780a557daea3656: Bootstrap starting.
W20260130 02:58:06.303179 19159 consensus_peers.cc:597] T 0a763e48583c4e748a46a325746ce69a P 371a36d219164974abdf848841528152 -> Peer d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): Couldn't send request to peer d870164263bb44b6b780a557daea3656. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260130 02:58:06.307258 19754 ts_tablet_manager.cc:1916] T 2112b0402de5464cb1caf2c2abb67d78 P 371a36d219164974abdf848841528152: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260130 02:58:06.309803 19807 tablet_bootstrap.cc:654] T 111287b5557d4abda1c5cd560fb0d019 P 371a36d219164974abdf848841528152: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:06.316969 19947 ts_tablet_manager.cc:1929] T b7b596f944be4daf953b82b65d5cec2a P d870164263bb44b6b780a557daea3656: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260130 02:58:06.317332 19947 log.cc:1199] T b7b596f944be4daf953b82b65d5cec2a P d870164263bb44b6b780a557daea3656: Deleting WAL directory at /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-2/wal/wals/b7b596f944be4daf953b82b65d5cec2a
I20260130 02:58:06.317981 19947 ts_tablet_manager.cc:1950] T b7b596f944be4daf953b82b65d5cec2a P d870164263bb44b6b780a557daea3656: Deleting consensus metadata
I20260130 02:58:06.319490 19625 tablet_bootstrap.cc:654] T 91f954f3ef4244c9a575a7421a7e75eb P d870164263bb44b6b780a557daea3656: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:06.339919 19947 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59068 (request call id 124) took 2686 ms. Trace:
I20260130 02:58:06.340279 19947 rpcz_store.cc:276] 0130 02:58:03.653890 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:58:03.654031 (+ 141us) service_pool.cc:225] Handling call
0130 02:58:06.159892 (+2505861us) ts_tablet_manager.cc:1151] Deleting tablet b7b596f944be4daf953b82b65d5cec2a
0130 02:58:06.159933 (+ 41us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0130 02:58:06.271145 (+111212us) tablet_metadata.cc:641] Metadata flushed
0130 02:58:06.316733 (+ 45588us) tablet_metadata.cc:641] Metadata flushed
0130 02:58:06.339856 (+ 23123us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2505294}
I20260130 02:58:06.341475 19947 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59068 (request call id 123) took 2689 ms. Trace:
I20260130 02:58:06.341796 19947 rpcz_store.cc:276] 0130 02:58:03.651686 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:58:03.655925 (+ 4239us) service_pool.cc:225] Handling call
0130 02:58:06.341140 (+2685215us) ts_tablet_manager.cc:1151] Deleting tablet 730bd1d7d15b4cbe945b3ef03b2455f7
0130 02:58:06.341181 (+ 41us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0130 02:58:06.341418 (+ 237us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2684376}
I20260130 02:58:06.342741 19947 tablet_replica.cc:333] T b3f3177306914e3c8b253db54ee30fb6 P d870164263bb44b6b780a557daea3656: stopping tablet replica
I20260130 02:58:06.343480 19947 raft_consensus.cc:2243] T b3f3177306914e3c8b253db54ee30fb6 P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260130 02:58:06.343946 19947 raft_consensus.cc:2272] T b3f3177306914e3c8b253db54ee30fb6 P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260130 02:58:06.345743 19947 ts_tablet_manager.cc:1916] T b3f3177306914e3c8b253db54ee30fb6 P d870164263bb44b6b780a557daea3656: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260130 02:58:06.348166 19807 tablet_bootstrap.cc:492] T 111287b5557d4abda1c5cd560fb0d019 P 371a36d219164974abdf848841528152: No bootstrap required, opened a new log
I20260130 02:58:06.349459 19807 ts_tablet_manager.cc:1403] T 111287b5557d4abda1c5cd560fb0d019 P 371a36d219164974abdf848841528152: Time spent bootstrapping tablet: real 0.065s user 0.005s sys 0.005s
I20260130 02:58:06.351114 18860 catalog_manager.cc:4985] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): tablet b7b596f944be4daf953b82b65d5cec2a (table default.Table_22d10332a83f4700ac21892bab67f46b [id=da3a9cb1f6d5419abc1afc93659b5324]) successfully deleted
I20260130 02:58:06.356402 19625 tablet_bootstrap.cc:492] T 91f954f3ef4244c9a575a7421a7e75eb P d870164263bb44b6b780a557daea3656: No bootstrap required, opened a new log
I20260130 02:58:06.356914 19625 ts_tablet_manager.cc:1403] T 91f954f3ef4244c9a575a7421a7e75eb P d870164263bb44b6b780a557daea3656: Time spent bootstrapping tablet: real 0.057s user 0.001s sys 0.008s
I20260130 02:58:06.351701 19807 raft_consensus.cc:359] T 111287b5557d4abda1c5cd560fb0d019 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:06.367213 19807 raft_consensus.cc:385] T 111287b5557d4abda1c5cd560fb0d019 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:06.367498 19807 raft_consensus.cc:740] T 111287b5557d4abda1c5cd560fb0d019 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 371a36d219164974abdf848841528152, State: Initialized, Role: FOLLOWER
W20260130 02:58:06.368499 18860 catalog_manager.cc:4965] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): delete failed for tablet 730bd1d7d15b4cbe945b3ef03b2455f7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 730bd1d7d15b4cbe945b3ef03b2455f7 already in progress: creating tablet
I20260130 02:58:06.369266 19807 consensus_queue.cc:260] T 111287b5557d4abda1c5cd560fb0d019 P 371a36d219164974abdf848841528152 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:06.345878 20227 tablet.cc:1722] T f2d01bfdc9c4494386bd02bed655fe63 P 78c2c46c7520492db35d51c244fb4f74: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20260130 02:58:06.359243 19625 raft_consensus.cc:359] T 91f954f3ef4244c9a575a7421a7e75eb P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:06.326422 19377 raft_consensus.cc:2468] T 730bd1d7d15b4cbe945b3ef03b2455f7 P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 371a36d219164974abdf848841528152 in term 1.
I20260130 02:58:06.376816 19159 leader_election.cc:304] T 730bd1d7d15b4cbe945b3ef03b2455f7 P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 371a36d219164974abdf848841528152, 78c2c46c7520492db35d51c244fb4f74; no voters: d870164263bb44b6b780a557daea3656
I20260130 02:58:06.377503 19625 raft_consensus.cc:385] T 91f954f3ef4244c9a575a7421a7e75eb P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:06.377938 19625 raft_consensus.cc:740] T 91f954f3ef4244c9a575a7421a7e75eb P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d870164263bb44b6b780a557daea3656, State: Initialized, Role: FOLLOWER
I20260130 02:58:06.378659 19625 consensus_queue.cc:260] T 91f954f3ef4244c9a575a7421a7e75eb P d870164263bb44b6b780a557daea3656 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:06.381196 20178 raft_consensus.cc:2804] T 730bd1d7d15b4cbe945b3ef03b2455f7 P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Leader election won for term 1
I20260130 02:58:06.381302 19807 ts_tablet_manager.cc:1434] T 111287b5557d4abda1c5cd560fb0d019 P 371a36d219164974abdf848841528152: Time spent starting tablet: real 0.032s user 0.005s sys 0.000s
I20260130 02:58:06.381649 20178 raft_consensus.cc:697] T 730bd1d7d15b4cbe945b3ef03b2455f7 P 371a36d219164974abdf848841528152 [term 1 LEADER]: Becoming Leader. State: Replica: 371a36d219164974abdf848841528152, State: Running, Role: LEADER
I20260130 02:58:06.382210 19807 tablet_bootstrap.cc:492] T 3cf5154d8cc44f2bb12978202699c193 P 371a36d219164974abdf848841528152: Bootstrap starting.
I20260130 02:58:06.382388 20178 consensus_queue.cc:237] T 730bd1d7d15b4cbe945b3ef03b2455f7 P 371a36d219164974abdf848841528152 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:06.396075 19625 ts_tablet_manager.cc:1434] T 91f954f3ef4244c9a575a7421a7e75eb P d870164263bb44b6b780a557daea3656: Time spent starting tablet: real 0.039s user 0.006s sys 0.000s
I20260130 02:58:06.411029 19625 tablet_bootstrap.cc:492] T 534dc31a9b7e492c82f3672c75742e6c P d870164263bb44b6b780a557daea3656: Bootstrap starting.
I20260130 02:58:06.416823 19754 ts_tablet_manager.cc:1929] T 2112b0402de5464cb1caf2c2abb67d78 P 371a36d219164974abdf848841528152: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260130 02:58:06.430742 19947 ts_tablet_manager.cc:1929] T b3f3177306914e3c8b253db54ee30fb6 P d870164263bb44b6b780a557daea3656: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260130 02:58:06.430727 19754 log.cc:1199] T 2112b0402de5464cb1caf2c2abb67d78 P 371a36d219164974abdf848841528152: Deleting WAL directory at /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-0/wal/wals/2112b0402de5464cb1caf2c2abb67d78
I20260130 02:58:06.431125 19947 log.cc:1199] T b3f3177306914e3c8b253db54ee30fb6 P d870164263bb44b6b780a557daea3656: Deleting WAL directory at /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-2/wal/wals/b3f3177306914e3c8b253db54ee30fb6
I20260130 02:58:06.431793 19947 ts_tablet_manager.cc:1950] T b3f3177306914e3c8b253db54ee30fb6 P d870164263bb44b6b780a557daea3656: Deleting consensus metadata
I20260130 02:58:06.432317 19754 ts_tablet_manager.cc:1950] T 2112b0402de5464cb1caf2c2abb67d78 P 371a36d219164974abdf848841528152: Deleting consensus metadata
I20260130 02:58:06.433724 19947 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59068 (request call id 129) took 2469 ms. Trace:
I20260130 02:58:06.434058 19947 rpcz_store.cc:276] 0130 02:58:03.992556 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:58:03.994332 (+ 1776us) service_pool.cc:225] Handling call
0130 02:58:06.342604 (+2348272us) ts_tablet_manager.cc:1151] Deleting tablet b3f3177306914e3c8b253db54ee30fb6
0130 02:58:06.342645 (+ 41us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0130 02:58:06.392713 (+ 50068us) tablet_metadata.cc:641] Metadata flushed
0130 02:58:06.430513 (+ 37800us) tablet_metadata.cc:641] Metadata flushed
0130 02:58:06.433656 (+ 3143us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2347595}
I20260130 02:58:06.435168 18860 catalog_manager.cc:4985] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): tablet b3f3177306914e3c8b253db54ee30fb6 (table default.Table_87dbbd65d92043b0b91c801517e56dfe [id=adced31928ba4204b79e3b43f8b737ee]) successfully deleted
I20260130 02:58:06.435915 18861 catalog_manager.cc:4985] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021): tablet 2112b0402de5464cb1caf2c2abb67d78 (table default.Table_da3d581b15a94840898777f74611994e [id=b7e241ced53e4160ac6fa1f65ef3674e]) successfully deleted
I20260130 02:58:06.423629 19807 tablet_bootstrap.cc:654] T 3cf5154d8cc44f2bb12978202699c193 P 371a36d219164974abdf848841528152: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:06.449608 19947 tablet_replica.cc:333] T a743a82540014c85aeab0e9d0994b0dc P d870164263bb44b6b780a557daea3656: stopping tablet replica
I20260130 02:58:06.449443 19198 tablet_service.cc:1505] Processing CreateTablet for tablet cf1b66a20fce4c4d96bc0bc32a7bafa8 (DEFAULT_TABLE table=default.Table_d24e400f7451443a82d23140a90adc57 [id=e478df970dac40f7a576ce584c3abf2c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:06.450399 19198 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cf1b66a20fce4c4d96bc0bc32a7bafa8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:06.450348 19947 raft_consensus.cc:2243] T a743a82540014c85aeab0e9d0994b0dc P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260130 02:58:06.450825 19947 raft_consensus.cc:2272] T a743a82540014c85aeab0e9d0994b0dc P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260130 02:58:06.452630 19947 ts_tablet_manager.cc:1916] T a743a82540014c85aeab0e9d0994b0dc P d870164263bb44b6b780a557daea3656: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260130 02:58:06.450886 19625 tablet_bootstrap.cc:654] T 534dc31a9b7e492c82f3672c75742e6c P d870164263bb44b6b780a557daea3656: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:06.463092 19201 tablet_service.cc:1505] Processing CreateTablet for tablet 6218f2f7d73a42428b57a45858f53afd (DEFAULT_TABLE table=default.Table_95f0bff8e17d4ca995c0af0d0c24e84d [id=b7795905d0c440579c59d6ac00473f5a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:06.464154 19201 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6218f2f7d73a42428b57a45858f53afd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:06.470503 19205 tablet_service.cc:1505] Processing CreateTablet for tablet dcba6b534f9d406dad4fb502be40921c (DEFAULT_TABLE table=default.Table_baaf376733384fdbbac49ec8f2357432 [id=dea97f19a50141878f97820194afa59e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:06.471505 19205 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dcba6b534f9d406dad4fb502be40921c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:06.389534 20194 raft_consensus.cc:493] T 3b148691090e4ee581d46f9845a6e49a P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:06.490422 20194 raft_consensus.cc:515] T 3b148691090e4ee581d46f9845a6e49a P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:06.493276 19224 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3b148691090e4ee581d46f9845a6e49a" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152" is_pre_election: true
W20260130 02:58:06.494861 19314 leader_election.cc:343] T 3b148691090e4ee581d46f9845a6e49a P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:06.496256 20194 leader_election.cc:290] T 3b148691090e4ee581d46f9845a6e49a P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), d870164263bb44b6b780a557daea3656 (127.18.100.67:39571)
I20260130 02:58:06.497081 19625 tablet_bootstrap.cc:492] T 534dc31a9b7e492c82f3672c75742e6c P d870164263bb44b6b780a557daea3656: No bootstrap required, opened a new log
I20260130 02:58:06.497467 19625 ts_tablet_manager.cc:1403] T 534dc31a9b7e492c82f3672c75742e6c P d870164263bb44b6b780a557daea3656: Time spent bootstrapping tablet: real 0.087s user 0.010s sys 0.000s
I20260130 02:58:06.499483 19501 tablet_service.cc:1505] Processing CreateTablet for tablet cf1b66a20fce4c4d96bc0bc32a7bafa8 (DEFAULT_TABLE table=default.Table_d24e400f7451443a82d23140a90adc57 [id=e478df970dac40f7a576ce584c3abf2c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:06.499715 19625 raft_consensus.cc:359] T 534dc31a9b7e492c82f3672c75742e6c P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:06.500355 19625 raft_consensus.cc:385] T 534dc31a9b7e492c82f3672c75742e6c P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:06.500468 19947 ts_tablet_manager.cc:1929] T a743a82540014c85aeab0e9d0994b0dc P d870164263bb44b6b780a557daea3656: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260130 02:58:06.500617 19625 raft_consensus.cc:740] T 534dc31a9b7e492c82f3672c75742e6c P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d870164263bb44b6b780a557daea3656, State: Initialized, Role: FOLLOWER
I20260130 02:58:06.500784 19501 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cf1b66a20fce4c4d96bc0bc32a7bafa8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:06.476817 18870 catalog_manager.cc:5746] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571) does not have the latest schema for tablet f2d01bfdc9c4494386bd02bed655fe63 (table default.Table_89d4117da99842719125411e30d10169 [id=d128767e303145a885eec3c5f714a27f]). Expected version 3 got 2
I20260130 02:58:06.503949 18870 catalog_manager.cc:5654] T 5ca9919f80954e74a4d91c94b74de0e4 P d870164263bb44b6b780a557daea3656 reported cstate change: term changed from 0 to 1, leader changed from <none> to 371a36d219164974abdf848841528152 (127.18.100.65). New cstate: current_term: 1 leader_uuid: "371a36d219164974abdf848841528152" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } }
I20260130 02:58:06.506479 19205 tablet_service.cc:1505] Processing CreateTablet for tablet f9e5f01e69254d658c369f3e21cef523 (DEFAULT_TABLE table=default.Table_12b505700b3d44e28353172e1786e5c8 [id=562776405d6f4c7aab0622f3866bf45b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:06.507007 20231 tablet.cc:1722] T 7d76046ee6c8420fa4964decbe8209c1 P 78c2c46c7520492db35d51c244fb4f74: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260130 02:58:06.500809 19947 log.cc:1199] T a743a82540014c85aeab0e9d0994b0dc P d870164263bb44b6b780a557daea3656: Deleting WAL directory at /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-2/wal/wals/a743a82540014c85aeab0e9d0994b0dc
I20260130 02:58:06.511360 19947 ts_tablet_manager.cc:1950] T a743a82540014c85aeab0e9d0994b0dc P d870164263bb44b6b780a557daea3656: Deleting consensus metadata
I20260130 02:58:06.513211 19947 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59068 (request call id 134) took 2260 ms. Trace:
I20260130 02:58:06.513494 19947 rpcz_store.cc:276] 0130 02:58:04.252521 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:58:04.255087 (+ 2566us) service_pool.cc:225] Handling call
0130 02:58:06.449457 (+2194370us) ts_tablet_manager.cc:1151] Deleting tablet a743a82540014c85aeab0e9d0994b0dc
0130 02:58:06.449499 (+ 42us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0130 02:58:06.477753 (+ 28254us) tablet_metadata.cc:641] Metadata flushed
0130 02:58:06.500262 (+ 22509us) tablet_metadata.cc:641] Metadata flushed
0130 02:58:06.513158 (+ 12896us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2169843}
I20260130 02:58:06.514930 19947 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59068 (request call id 137) took 1966 ms. Trace:
I20260130 02:58:06.515225 19947 rpcz_store.cc:276] 0130 02:58:04.548822 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:58:04.549681 (+ 859us) service_pool.cc:225] Handling call
0130 02:58:06.514639 (+1964958us) ts_tablet_manager.cc:1151] Deleting tablet 7d96e93d95f14af79cfb268626559b1e
0130 02:58:06.514677 (+ 38us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0130 02:58:06.514887 (+ 210us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1901448}
I20260130 02:58:06.517899 19205 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f9e5f01e69254d658c369f3e21cef523. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:06.524147 19947 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59068 (request call id 135) took 1976 ms. Trace:
I20260130 02:58:06.524250 19807 tablet_bootstrap.cc:492] T 3cf5154d8cc44f2bb12978202699c193 P 371a36d219164974abdf848841528152: No bootstrap required, opened a new log
I20260130 02:58:06.524750 19807 ts_tablet_manager.cc:1403] T 3cf5154d8cc44f2bb12978202699c193 P 371a36d219164974abdf848841528152: Time spent bootstrapping tablet: real 0.143s user 0.012s sys 0.004s
I20260130 02:58:06.524458 19947 rpcz_store.cc:276] 0130 02:58:04.548201 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:58:04.548333 (+ 132us) service_pool.cc:225] Handling call
0130 02:58:06.520354 (+1972021us) ts_tablet_manager.cc:1151] Deleting tablet f54addb1a4304ce69c711ae0cc3b452c
0130 02:58:06.523838 (+ 3484us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0130 02:58:06.524090 (+ 252us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1906896}
I20260130 02:58:06.525683 19947 tablet_replica.cc:333] T c3fce8cc99fb451ebd532fa37390e862 P d870164263bb44b6b780a557daea3656: stopping tablet replica
I20260130 02:58:06.526360 19947 raft_consensus.cc:2243] T c3fce8cc99fb451ebd532fa37390e862 P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260130 02:58:06.526846 19947 raft_consensus.cc:2272] T c3fce8cc99fb451ebd532fa37390e862 P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260130 02:58:06.528539 19947 ts_tablet_manager.cc:1916] T c3fce8cc99fb451ebd532fa37390e862 P d870164263bb44b6b780a557daea3656: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260130 02:58:06.529141 19807 raft_consensus.cc:359] T 3cf5154d8cc44f2bb12978202699c193 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:06.529951 19625 consensus_queue.cc:260] T 534dc31a9b7e492c82f3672c75742e6c P d870164263bb44b6b780a557daea3656 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:06.530886 19807 raft_consensus.cc:385] T 3cf5154d8cc44f2bb12978202699c193 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:06.531327 19807 raft_consensus.cc:740] T 3cf5154d8cc44f2bb12978202699c193 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 371a36d219164974abdf848841528152, State: Initialized, Role: FOLLOWER
I20260130 02:58:06.533121 19625 ts_tablet_manager.cc:1434] T 534dc31a9b7e492c82f3672c75742e6c P d870164263bb44b6b780a557daea3656: Time spent starting tablet: real 0.035s user 0.005s sys 0.000s
I20260130 02:58:06.533682 19807 consensus_queue.cc:260] T 3cf5154d8cc44f2bb12978202699c193 P 371a36d219164974abdf848841528152 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:06.535254 19625 tablet_bootstrap.cc:492] T a90e92d569254bbf8a5fbf286baf5cce P d870164263bb44b6b780a557daea3656: Bootstrap starting.
I20260130 02:58:06.537757 19807 ts_tablet_manager.cc:1434] T 3cf5154d8cc44f2bb12978202699c193 P 371a36d219164974abdf848841528152: Time spent starting tablet: real 0.013s user 0.004s sys 0.000s
I20260130 02:58:06.546692 19807 tablet_bootstrap.cc:492] T 91f954f3ef4244c9a575a7421a7e75eb P 371a36d219164974abdf848841528152: Bootstrap starting.
I20260130 02:58:06.552340 19807 tablet_bootstrap.cc:654] T 91f954f3ef4244c9a575a7421a7e75eb P 371a36d219164974abdf848841528152: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:06.552403 19625 tablet_bootstrap.cc:654] T a90e92d569254bbf8a5fbf286baf5cce P d870164263bb44b6b780a557daea3656: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:06.565944 19508 tablet_service.cc:1505] Processing CreateTablet for tablet 6218f2f7d73a42428b57a45858f53afd (DEFAULT_TABLE table=default.Table_95f0bff8e17d4ca995c0af0d0c24e84d [id=b7795905d0c440579c59d6ac00473f5a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:06.567420 19508 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6218f2f7d73a42428b57a45858f53afd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:06.570566 20224 raft_consensus.cc:493] T e47b22aafe5247c4b62c4193c24f427c P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:06.571019 20224 raft_consensus.cc:515] T e47b22aafe5247c4b62c4193c24f427c P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:06.573084 19947 ts_tablet_manager.cc:1929] T c3fce8cc99fb451ebd532fa37390e862 P d870164263bb44b6b780a557daea3656: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260130 02:58:06.573396 19947 log.cc:1199] T c3fce8cc99fb451ebd532fa37390e862 P d870164263bb44b6b780a557daea3656: Deleting WAL directory at /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-2/wal/wals/c3fce8cc99fb451ebd532fa37390e862
I20260130 02:58:06.572849 20224 leader_election.cc:290] T e47b22aafe5247c4b62c4193c24f427c P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767)
I20260130 02:58:06.574054 19807 tablet_bootstrap.cc:492] T 91f954f3ef4244c9a575a7421a7e75eb P 371a36d219164974abdf848841528152: No bootstrap required, opened a new log
I20260130 02:58:06.574054 19947 ts_tablet_manager.cc:1950] T c3fce8cc99fb451ebd532fa37390e862 P d870164263bb44b6b780a557daea3656: Deleting consensus metadata
I20260130 02:58:06.574431 19807 ts_tablet_manager.cc:1403] T 91f954f3ef4244c9a575a7421a7e75eb P 371a36d219164974abdf848841528152: Time spent bootstrapping tablet: real 0.028s user 0.009s sys 0.002s
I20260130 02:58:06.575922 19947 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59068 (request call id 138) took 1898 ms. Trace:
I20260130 02:58:06.576231 19947 rpcz_store.cc:276] 0130 02:58:04.677290 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:58:04.677424 (+ 134us) service_pool.cc:225] Handling call
0130 02:58:06.525553 (+1848129us) ts_tablet_manager.cc:1151] Deleting tablet c3fce8cc99fb451ebd532fa37390e862
0130 02:58:06.525589 (+ 36us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0130 02:58:06.542423 (+ 16834us) tablet_metadata.cc:641] Metadata flushed
0130 02:58:06.572870 (+ 30447us) tablet_metadata.cc:641] Metadata flushed
0130 02:58:06.575865 (+ 2995us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1847577}
W20260130 02:58:06.577280 18859 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet 9fe0448bcc4a4346b3c67fc3f626ed3c (table default.Table_5c1c447e85524ac5a61ffd8a48509d36 [id=70aaf21ad9bd4d959f46939c6b4a4880], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 9fe0448bcc4a4346b3c67fc3f626ed3c consensus state has no leader
I20260130 02:58:06.576754 19807 raft_consensus.cc:359] T 91f954f3ef4244c9a575a7421a7e75eb P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:06.578480 19807 raft_consensus.cc:385] T 91f954f3ef4244c9a575a7421a7e75eb P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:06.578819 19807 raft_consensus.cc:740] T 91f954f3ef4244c9a575a7421a7e75eb P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 371a36d219164974abdf848841528152, State: Initialized, Role: FOLLOWER
I20260130 02:58:06.579433 19807 consensus_queue.cc:260] T 91f954f3ef4244c9a575a7421a7e75eb P 371a36d219164974abdf848841528152 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:06.581949 19357 tablet_service.cc:1505] Processing CreateTablet for tablet cf1b66a20fce4c4d96bc0bc32a7bafa8 (DEFAULT_TABLE table=default.Table_d24e400f7451443a82d23140a90adc57 [id=e478df970dac40f7a576ce584c3abf2c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:06.583107 19357 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cf1b66a20fce4c4d96bc0bc32a7bafa8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:06.584081 19807 ts_tablet_manager.cc:1434] T 91f954f3ef4244c9a575a7421a7e75eb P 371a36d219164974abdf848841528152: Time spent starting tablet: real 0.009s user 0.004s sys 0.000s
I20260130 02:58:06.584890 19807 tablet_bootstrap.cc:492] T b8c579ff5fa84651ba9125457d927162 P 371a36d219164974abdf848841528152: Bootstrap starting.
I20260130 02:58:06.585352 19525 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3b148691090e4ee581d46f9845a6e49a" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
I20260130 02:58:06.588748 19947 tablet_replica.cc:333] T df2c8989d6c1480c97baa23db01141eb P d870164263bb44b6b780a557daea3656: stopping tablet replica
I20260130 02:58:06.589466 19947 raft_consensus.cc:2243] T df2c8989d6c1480c97baa23db01141eb P d870164263bb44b6b780a557daea3656 [term 1 LEADER]: Raft consensus shutting down.
I20260130 02:58:06.590646 19947 raft_consensus.cc:2272] T df2c8989d6c1480c97baa23db01141eb P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260130 02:58:06.590279 19496 tablet_service.cc:1505] Processing CreateTablet for tablet dcba6b534f9d406dad4fb502be40921c (DEFAULT_TABLE table=default.Table_baaf376733384fdbbac49ec8f2357432 [id=dea97f19a50141878f97820194afa59e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:06.591528 19496 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dcba6b534f9d406dad4fb502be40921c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:06.592605 19947 ts_tablet_manager.cc:1916] T df2c8989d6c1480c97baa23db01141eb P d870164263bb44b6b780a557daea3656: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260130 02:58:06.595366 19625 tablet_bootstrap.cc:492] T a90e92d569254bbf8a5fbf286baf5cce P d870164263bb44b6b780a557daea3656: No bootstrap required, opened a new log
I20260130 02:58:06.595754 19625 ts_tablet_manager.cc:1403] T a90e92d569254bbf8a5fbf286baf5cce P d870164263bb44b6b780a557daea3656: Time spent bootstrapping tablet: real 0.061s user 0.009s sys 0.002s
I20260130 02:58:06.607910 19505 tablet_service.cc:1505] Processing CreateTablet for tablet f9e5f01e69254d658c369f3e21cef523 (DEFAULT_TABLE table=default.Table_12b505700b3d44e28353172e1786e5c8 [id=562776405d6f4c7aab0622f3866bf45b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:06.608353 19807 tablet_bootstrap.cc:654] T b8c579ff5fa84651ba9125457d927162 P 371a36d219164974abdf848841528152: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:06.609076 19505 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f9e5f01e69254d658c369f3e21cef523. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:06.613018 19625 raft_consensus.cc:359] T a90e92d569254bbf8a5fbf286baf5cce P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:06.613739 19625 raft_consensus.cc:385] T a90e92d569254bbf8a5fbf286baf5cce P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:06.613989 19625 raft_consensus.cc:740] T a90e92d569254bbf8a5fbf286baf5cce P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d870164263bb44b6b780a557daea3656, State: Initialized, Role: FOLLOWER
I20260130 02:58:06.614636 19625 consensus_queue.cc:260] T a90e92d569254bbf8a5fbf286baf5cce P d870164263bb44b6b780a557daea3656 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:06.616619 19625 ts_tablet_manager.cc:1434] T a90e92d569254bbf8a5fbf286baf5cce P d870164263bb44b6b780a557daea3656: Time spent starting tablet: real 0.021s user 0.006s sys 0.000s
I20260130 02:58:06.618340 19625 tablet_bootstrap.cc:492] T 111287b5557d4abda1c5cd560fb0d019 P d870164263bb44b6b780a557daea3656: Bootstrap starting.
I20260130 02:58:06.624396 19807 tablet_bootstrap.cc:492] T b8c579ff5fa84651ba9125457d927162 P 371a36d219164974abdf848841528152: No bootstrap required, opened a new log
I20260130 02:58:06.624826 19807 ts_tablet_manager.cc:1403] T b8c579ff5fa84651ba9125457d927162 P 371a36d219164974abdf848841528152: Time spent bootstrapping tablet: real 0.040s user 0.009s sys 0.000s
I20260130 02:58:06.626994 19807 raft_consensus.cc:359] T b8c579ff5fa84651ba9125457d927162 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:06.630725 19807 raft_consensus.cc:385] T b8c579ff5fa84651ba9125457d927162 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:06.630993 19807 raft_consensus.cc:740] T b8c579ff5fa84651ba9125457d927162 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 371a36d219164974abdf848841528152, State: Initialized, Role: FOLLOWER
I20260130 02:58:06.631614 19807 consensus_queue.cc:260] T b8c579ff5fa84651ba9125457d927162 P 371a36d219164974abdf848841528152 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:06.633818 19807 ts_tablet_manager.cc:1434] T b8c579ff5fa84651ba9125457d927162 P 371a36d219164974abdf848841528152: Time spent starting tablet: real 0.009s user 0.006s sys 0.000s
I20260130 02:58:06.638823 19625 tablet_bootstrap.cc:654] T 111287b5557d4abda1c5cd560fb0d019 P d870164263bb44b6b780a557daea3656: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:06.658881 19625 tablet_bootstrap.cc:492] T 111287b5557d4abda1c5cd560fb0d019 P d870164263bb44b6b780a557daea3656: No bootstrap required, opened a new log
I20260130 02:58:06.658946 19947 ts_tablet_manager.cc:1929] T df2c8989d6c1480c97baa23db01141eb P d870164263bb44b6b780a557daea3656: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20260130 02:58:06.659346 19947 log.cc:1199] T df2c8989d6c1480c97baa23db01141eb P d870164263bb44b6b780a557daea3656: Deleting WAL directory at /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-2/wal/wals/df2c8989d6c1480c97baa23db01141eb
I20260130 02:58:06.659396 19625 ts_tablet_manager.cc:1403] T 111287b5557d4abda1c5cd560fb0d019 P d870164263bb44b6b780a557daea3656: Time spent bootstrapping tablet: real 0.041s user 0.008s sys 0.005s
I20260130 02:58:06.660022 19947 ts_tablet_manager.cc:1950] T df2c8989d6c1480c97baa23db01141eb P d870164263bb44b6b780a557daea3656: Deleting consensus metadata
I20260130 02:58:06.661993 19947 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59068 (request call id 139) took 1870 ms. Trace:
I20260130 02:58:06.661911 19625 raft_consensus.cc:359] T 111287b5557d4abda1c5cd560fb0d019 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:06.662500 19625 raft_consensus.cc:385] T 111287b5557d4abda1c5cd560fb0d019 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:06.662786 19625 raft_consensus.cc:740] T 111287b5557d4abda1c5cd560fb0d019 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d870164263bb44b6b780a557daea3656, State: Initialized, Role: FOLLOWER
I20260130 02:58:06.663331 18860 catalog_manager.cc:4985] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): tablet a743a82540014c85aeab0e9d0994b0dc (table default.Table_71ecfab2cca249b49070a8340586ce5e [id=19941bcff5454783adb47df8dd77f352]) successfully deleted
I20260130 02:58:06.663534 19625 consensus_queue.cc:260] T 111287b5557d4abda1c5cd560fb0d019 P d870164263bb44b6b780a557daea3656 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:06.662328 19947 rpcz_store.cc:276] 0130 02:58:04.791406 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:58:04.791563 (+ 157us) service_pool.cc:225] Handling call
0130 02:58:06.588587 (+1797024us) ts_tablet_manager.cc:1151] Deleting tablet df2c8989d6c1480c97baa23db01141eb
0130 02:58:06.588629 (+ 42us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0130 02:58:06.617856 (+ 29227us) tablet_metadata.cc:641] Metadata flushed
0130 02:58:06.658705 (+ 40849us) tablet_metadata.cc:641] Metadata flushed
0130 02:58:06.661934 (+ 3229us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1796379}
I20260130 02:58:06.672220 19947 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59068 (request call id 147) took 1641 ms. Trace:
I20260130 02:58:06.672533 19947 rpcz_store.cc:276] 0130 02:58:05.031379 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:58:05.031486 (+ 107us) service_pool.cc:225] Handling call
0130 02:58:06.671863 (+1640377us) ts_tablet_manager.cc:1151] Deleting tablet b2ac91da01954c2390b6b7dcc06c394f
0130 02:58:06.671904 (+ 41us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0130 02:58:06.672166 (+ 262us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1639813}
W20260130 02:58:06.675580 18860 catalog_manager.cc:4965] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): delete failed for tablet 7d96e93d95f14af79cfb268626559b1e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7d96e93d95f14af79cfb268626559b1e already in progress: creating tablet
W20260130 02:58:06.676365 18860 catalog_manager.cc:4965] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): delete failed for tablet f54addb1a4304ce69c711ae0cc3b452c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f54addb1a4304ce69c711ae0cc3b452c already in progress: creating tablet
I20260130 02:58:06.677050 18860 catalog_manager.cc:4985] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): tablet c3fce8cc99fb451ebd532fa37390e862 (table default.Table_5d29239ce1984d2fad02607a2bdd4dd0 [id=7e271b81bb064a949685b2411b485e3e]) successfully deleted
W20260130 02:58:06.686638 19313 leader_election.cc:343] T 3b148691090e4ee581d46f9845a6e49a P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:06.687119 19313 leader_election.cc:304] T 3b148691090e4ee581d46f9845a6e49a P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78c2c46c7520492db35d51c244fb4f74; no voters: 371a36d219164974abdf848841528152, d870164263bb44b6b780a557daea3656
I20260130 02:58:06.696178 20009 raft_consensus.cc:2749] T 3b148691090e4ee581d46f9845a6e49a P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260130 02:58:06.684897 20194 raft_consensus.cc:493] T b8c579ff5fa84651ba9125457d927162 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:06.698916 19376 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e47b22aafe5247c4b62c4193c24f427c" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74" is_pre_election: true
I20260130 02:58:06.707185 19352 tablet_service.cc:1505] Processing CreateTablet for tablet 6218f2f7d73a42428b57a45858f53afd (DEFAULT_TABLE table=default.Table_95f0bff8e17d4ca995c0af0d0c24e84d [id=b7795905d0c440579c59d6ac00473f5a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:06.708382 19352 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6218f2f7d73a42428b57a45858f53afd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:06.714929 19947 tablet_replica.cc:333] T 534dc31a9b7e492c82f3672c75742e6c P d870164263bb44b6b780a557daea3656: stopping tablet replica
I20260130 02:58:06.716027 19355 tablet_service.cc:1505] Processing CreateTablet for tablet dcba6b534f9d406dad4fb502be40921c (DEFAULT_TABLE table=default.Table_baaf376733384fdbbac49ec8f2357432 [id=dea97f19a50141878f97820194afa59e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:06.717303 19355 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dcba6b534f9d406dad4fb502be40921c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:06.719386 19353 tablet_service.cc:1505] Processing CreateTablet for tablet f9e5f01e69254d658c369f3e21cef523 (DEFAULT_TABLE table=default.Table_12b505700b3d44e28353172e1786e5c8 [id=562776405d6f4c7aab0622f3866bf45b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:06.719779 19625 ts_tablet_manager.cc:1434] T 111287b5557d4abda1c5cd560fb0d019 P d870164263bb44b6b780a557daea3656: Time spent starting tablet: real 0.060s user 0.007s sys 0.011s
I20260130 02:58:06.720038 19947 raft_consensus.cc:2243] T 534dc31a9b7e492c82f3672c75742e6c P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260130 02:58:06.720470 19353 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f9e5f01e69254d658c369f3e21cef523. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:06.720512 19947 raft_consensus.cc:2272] T 534dc31a9b7e492c82f3672c75742e6c P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260130 02:58:06.722846 20224 raft_consensus.cc:493] T 91f954f3ef4244c9a575a7421a7e75eb P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:06.704823 20194 raft_consensus.cc:515] T b8c579ff5fa84651ba9125457d927162 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:06.723223 20224 raft_consensus.cc:515] T 91f954f3ef4244c9a575a7421a7e75eb P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:06.723790 19947 ts_tablet_manager.cc:1916] T 534dc31a9b7e492c82f3672c75742e6c P d870164263bb44b6b780a557daea3656: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260130 02:58:06.724830 20194 leader_election.cc:290] T b8c579ff5fa84651ba9125457d927162 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), d870164263bb44b6b780a557daea3656 (127.18.100.67:39571)
I20260130 02:58:06.724977 20224 leader_election.cc:290] T 91f954f3ef4244c9a575a7421a7e75eb P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767)
I20260130 02:58:06.737803 19947 ts_tablet_manager.cc:1929] T 534dc31a9b7e492c82f3672c75742e6c P d870164263bb44b6b780a557daea3656: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260130 02:58:06.738188 19947 log.cc:1199] T 534dc31a9b7e492c82f3672c75742e6c P d870164263bb44b6b780a557daea3656: Deleting WAL directory at /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-2/wal/wals/534dc31a9b7e492c82f3672c75742e6c
I20260130 02:58:06.738775 19947 ts_tablet_manager.cc:1950] T 534dc31a9b7e492c82f3672c75742e6c P d870164263bb44b6b780a557daea3656: Deleting consensus metadata
I20260130 02:58:06.740499 19947 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59068 (request call id 152) took 1608 ms. Trace:
I20260130 02:58:06.741660 19625 tablet_bootstrap.cc:492] T 730bd1d7d15b4cbe945b3ef03b2455f7 P d870164263bb44b6b780a557daea3656: Bootstrap starting.
I20260130 02:58:06.748750 19625 tablet_bootstrap.cc:654] T 730bd1d7d15b4cbe945b3ef03b2455f7 P d870164263bb44b6b780a557daea3656: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:06.749220 19840 tablet_bootstrap.cc:492] T 54be22352e404ff9b798a075693f7f21 P 78c2c46c7520492db35d51c244fb4f74: No bootstrap required, opened a new log
I20260130 02:58:06.749742 19840 ts_tablet_manager.cc:1403] T 54be22352e404ff9b798a075693f7f21 P 78c2c46c7520492db35d51c244fb4f74: Time spent bootstrapping tablet: real 0.613s user 0.007s sys 0.010s
I20260130 02:58:06.752568 19840 raft_consensus.cc:359] T 54be22352e404ff9b798a075693f7f21 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:06.740789 19947 rpcz_store.cc:276] 0130 02:58:05.131867 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:58:05.132021 (+ 154us) service_pool.cc:225] Handling call
0130 02:58:06.714705 (+1582684us) ts_tablet_manager.cc:1151] Deleting tablet 534dc31a9b7e492c82f3672c75742e6c
0130 02:58:06.714813 (+ 108us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0130 02:58:06.731386 (+ 16573us) tablet_metadata.cc:641] Metadata flushed
0130 02:58:06.737600 (+ 6214us) tablet_metadata.cc:641] Metadata flushed
0130 02:58:06.740447 (+ 2847us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":4266,"tablet-delete.queue_time_us":1582137}
W20260130 02:58:06.766721 18860 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.18.100.67:39571, user_credentials={real_user=slave}} blocked reactor thread for 89677us
I20260130 02:58:06.767570 18860 catalog_manager.cc:4985] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): tablet df2c8989d6c1480c97baa23db01141eb (table default.Table_e84c6277dfc944059f62cbe0d6b937ee [id=747af828147149efb7b7b8d33c49286d]) successfully deleted
W20260130 02:58:06.775344 18860 catalog_manager.cc:4965] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): delete failed for tablet b2ac91da01954c2390b6b7dcc06c394f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b2ac91da01954c2390b6b7dcc06c394f already in progress: creating tablet
I20260130 02:58:06.775983 18860 catalog_manager.cc:4985] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): tablet 534dc31a9b7e492c82f3672c75742e6c (table default.Table_081113b62fb743189a6e04bf85dcc5fa [id=66e19e6011bc4d86bf579a4a1b3db867]) successfully deleted
I20260130 02:58:06.779600 19947 tablet_replica.cc:333] T bd43c5d148514836869a8ac648c8ad11 P d870164263bb44b6b780a557daea3656: stopping tablet replica
I20260130 02:58:06.780321 19947 raft_consensus.cc:2243] T bd43c5d148514836869a8ac648c8ad11 P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260130 02:58:06.780793 19947 raft_consensus.cc:2272] T bd43c5d148514836869a8ac648c8ad11 P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260130 02:58:06.783620 19947 ts_tablet_manager.cc:1916] T bd43c5d148514836869a8ac648c8ad11 P d870164263bb44b6b780a557daea3656: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260130 02:58:06.784193 19625 tablet_bootstrap.cc:492] T 730bd1d7d15b4cbe945b3ef03b2455f7 P d870164263bb44b6b780a557daea3656: No bootstrap required, opened a new log
I20260130 02:58:06.784749 19625 ts_tablet_manager.cc:1403] T 730bd1d7d15b4cbe945b3ef03b2455f7 P d870164263bb44b6b780a557daea3656: Time spent bootstrapping tablet: real 0.043s user 0.020s sys 0.000s
I20260130 02:58:06.787307 19625 raft_consensus.cc:359] T 730bd1d7d15b4cbe945b3ef03b2455f7 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:06.787941 19625 raft_consensus.cc:385] T 730bd1d7d15b4cbe945b3ef03b2455f7 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:06.788198 19625 raft_consensus.cc:740] T 730bd1d7d15b4cbe945b3ef03b2455f7 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d870164263bb44b6b780a557daea3656, State: Initialized, Role: FOLLOWER
I20260130 02:58:06.788872 19625 consensus_queue.cc:260] T 730bd1d7d15b4cbe945b3ef03b2455f7 P d870164263bb44b6b780a557daea3656 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:06.792495 19625 ts_tablet_manager.cc:1434] T 730bd1d7d15b4cbe945b3ef03b2455f7 P d870164263bb44b6b780a557daea3656: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260130 02:58:06.793329 19625 tablet_bootstrap.cc:492] T 341041cb0be2456e88403367c0814704 P d870164263bb44b6b780a557daea3656: Bootstrap starting.
W20260130 02:58:06.796670 18860 catalog_manager.cc:4952] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): delete failed for tablet 37470ce4356f4542b621e21698551209 because the tablet was not found. No further retry: Not found: Tablet not found: 37470ce4356f4542b621e21698551209
W20260130 02:58:06.797334 18860 catalog_manager.cc:4952] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): delete failed for tablet 5b2976108584478a98fd90fe6594af66 because the tablet was not found. No further retry: Not found: Tablet not found: 5b2976108584478a98fd90fe6594af66
W20260130 02:58:06.798192 18860 catalog_manager.cc:4952] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): delete failed for tablet b078934be4bb4fd1915c691eeea9d00f because the tablet was not found. No further retry: Not found: Tablet not found: b078934be4bb4fd1915c691eeea9d00f
W20260130 02:58:06.798805 18860 catalog_manager.cc:4952] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): delete failed for tablet d701152960b14396a6ac1bd10480d2b6 because the tablet was not found. No further retry: Not found: Tablet not found: d701152960b14396a6ac1bd10480d2b6
W20260130 02:58:06.799638 18860 catalog_manager.cc:4952] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): delete failed for tablet 073b5941850c4140b1751956dd7cf5ef because the tablet was not found. No further retry: Not found: Tablet not found: 073b5941850c4140b1751956dd7cf5ef
I20260130 02:58:06.806783 19222 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "91f954f3ef4244c9a575a7421a7e75eb" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152" is_pre_election: true
I20260130 02:58:06.806785 19223 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e47b22aafe5247c4b62c4193c24f427c" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152" is_pre_election: true
I20260130 02:58:06.807955 19222 raft_consensus.cc:2468] T 91f954f3ef4244c9a575a7421a7e75eb P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d870164263bb44b6b780a557daea3656 in term 0.
I20260130 02:58:06.813066 19463 leader_election.cc:304] T 91f954f3ef4244c9a575a7421a7e75eb P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 371a36d219164974abdf848841528152, d870164263bb44b6b780a557daea3656; no voters:
I20260130 02:58:06.813923 20224 raft_consensus.cc:2804] T 91f954f3ef4244c9a575a7421a7e75eb P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260130 02:58:06.814276 20224 raft_consensus.cc:493] T 91f954f3ef4244c9a575a7421a7e75eb P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260130 02:58:06.814610 20224 raft_consensus.cc:3060] T 91f954f3ef4244c9a575a7421a7e75eb P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:06.814363 20115 consensus_queue.cc:1048] T 0a763e48583c4e748a46a325746ce69a P 371a36d219164974abdf848841528152 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260130 02:58:06.818545 19807 tablet_bootstrap.cc:492] T 341041cb0be2456e88403367c0814704 P 371a36d219164974abdf848841528152: Bootstrap starting.
I20260130 02:58:06.821031 20224 raft_consensus.cc:515] T 91f954f3ef4244c9a575a7421a7e75eb P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:06.823686 19222 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "91f954f3ef4244c9a575a7421a7e75eb" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152"
I20260130 02:58:06.824250 19222 raft_consensus.cc:3060] T 91f954f3ef4244c9a575a7421a7e75eb P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:06.827092 19807 tablet_bootstrap.cc:654] T 341041cb0be2456e88403367c0814704 P 371a36d219164974abdf848841528152: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:06.827608 19947 ts_tablet_manager.cc:1929] T bd43c5d148514836869a8ac648c8ad11 P d870164263bb44b6b780a557daea3656: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260130 02:58:06.827950 19947 log.cc:1199] T bd43c5d148514836869a8ac648c8ad11 P d870164263bb44b6b780a557daea3656: Deleting WAL directory at /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-2/wal/wals/bd43c5d148514836869a8ac648c8ad11
I20260130 02:58:06.828058 19840 raft_consensus.cc:385] T 54be22352e404ff9b798a075693f7f21 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:06.828333 19840 raft_consensus.cc:740] T 54be22352e404ff9b798a075693f7f21 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78c2c46c7520492db35d51c244fb4f74, State: Initialized, Role: FOLLOWER
I20260130 02:58:06.828606 19947 ts_tablet_manager.cc:1950] T bd43c5d148514836869a8ac648c8ad11 P d870164263bb44b6b780a557daea3656: Deleting consensus metadata
I20260130 02:58:06.829037 19840 consensus_queue.cc:260] T 54be22352e404ff9b798a075693f7f21 P 78c2c46c7520492db35d51c244fb4f74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:06.831923 19840 ts_tablet_manager.cc:1434] T 54be22352e404ff9b798a075693f7f21 P 78c2c46c7520492db35d51c244fb4f74: Time spent starting tablet: real 0.082s user 0.001s sys 0.004s
I20260130 02:58:06.832015 19222 raft_consensus.cc:2468] T 91f954f3ef4244c9a575a7421a7e75eb P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d870164263bb44b6b780a557daea3656 in term 1.
I20260130 02:58:06.832942 19840 tablet_bootstrap.cc:492] T 3b4908bd1a634ca0907234dd3c39e1c3 P 78c2c46c7520492db35d51c244fb4f74: Bootstrap starting.
I20260130 02:58:06.833354 19463 leader_election.cc:304] T 91f954f3ef4244c9a575a7421a7e75eb P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 371a36d219164974abdf848841528152, d870164263bb44b6b780a557daea3656; no voters:
I20260130 02:58:06.834093 20186 raft_consensus.cc:2804] T 91f954f3ef4244c9a575a7421a7e75eb P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Leader election won for term 1
I20260130 02:58:06.767184 20208 ts_tablet_manager.cc:1929] T 927c90b84d7449efa36bc5e2035bf0da P 78c2c46c7520492db35d51c244fb4f74: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
W20260130 02:58:06.785300 19313 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.100.67:39571, user_credentials={real_user=slave}} blocked reactor thread for 98707.9us
I20260130 02:58:06.834497 20186 raft_consensus.cc:697] T 91f954f3ef4244c9a575a7421a7e75eb P d870164263bb44b6b780a557daea3656 [term 1 LEADER]: Becoming Leader. State: Replica: d870164263bb44b6b780a557daea3656, State: Running, Role: LEADER
I20260130 02:58:06.835256 20186 consensus_queue.cc:237] T 91f954f3ef4244c9a575a7421a7e75eb P d870164263bb44b6b780a557daea3656 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:06.829690 19225 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b8c579ff5fa84651ba9125457d927162" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152" is_pre_election: true
I20260130 02:58:06.838333 19225 raft_consensus.cc:2468] T b8c579ff5fa84651ba9125457d927162 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78c2c46c7520492db35d51c244fb4f74 in term 0.
I20260130 02:58:06.838832 19525 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b8c579ff5fa84651ba9125457d927162" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
I20260130 02:58:06.841256 19314 leader_election.cc:304] T b8c579ff5fa84651ba9125457d927162 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 371a36d219164974abdf848841528152, 78c2c46c7520492db35d51c244fb4f74; no voters:
I20260130 02:58:06.844923 20194 raft_consensus.cc:2804] T b8c579ff5fa84651ba9125457d927162 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260130 02:58:06.845304 20194 raft_consensus.cc:493] T b8c579ff5fa84651ba9125457d927162 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260130 02:58:06.845660 20194 raft_consensus.cc:3060] T b8c579ff5fa84651ba9125457d927162 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:06.849627 19947 tablet_replica.cc:333] T 927c90b84d7449efa36bc5e2035bf0da P d870164263bb44b6b780a557daea3656: stopping tablet replica
I20260130 02:58:06.850320 19947 raft_consensus.cc:2243] T 927c90b84d7449efa36bc5e2035bf0da P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260130 02:58:06.851641 19462 leader_election.cc:304] T e47b22aafe5247c4b62c4193c24f427c P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d870164263bb44b6b780a557daea3656; no voters: 371a36d219164974abdf848841528152, 78c2c46c7520492db35d51c244fb4f74
I20260130 02:58:06.853176 20245 raft_consensus.cc:2749] T e47b22aafe5247c4b62c4193c24f427c P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260130 02:58:06.853860 19947 raft_consensus.cc:2272] T 927c90b84d7449efa36bc5e2035bf0da P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260130 02:58:06.853905 20179 raft_consensus.cc:493] T 111287b5557d4abda1c5cd560fb0d019 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:06.854372 20179 raft_consensus.cc:515] T 111287b5557d4abda1c5cd560fb0d019 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:06.855697 19947 ts_tablet_manager.cc:1916] T 927c90b84d7449efa36bc5e2035bf0da P d870164263bb44b6b780a557daea3656: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260130 02:58:06.856230 20179 leader_election.cc:290] T 111287b5557d4abda1c5cd560fb0d019 P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d870164263bb44b6b780a557daea3656 (127.18.100.67:39571), 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767)
I20260130 02:58:06.857295 19377 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "91f954f3ef4244c9a575a7421a7e75eb" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74" is_pre_election: true
I20260130 02:58:06.857832 19377 raft_consensus.cc:2468] T 91f954f3ef4244c9a575a7421a7e75eb P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d870164263bb44b6b780a557daea3656 in term 0.
I20260130 02:58:06.858624 20179 raft_consensus.cc:493] T 6573198830ad45628e71952a7d9c00a0 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260130 02:58:06.859123 19313 leader_election.cc:343] T b8c579ff5fa84651ba9125457d927162 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:06.859059 20179 raft_consensus.cc:515] T 6573198830ad45628e71952a7d9c00a0 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:06.860666 20179 leader_election.cc:290] T 6573198830ad45628e71952a7d9c00a0 P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d870164263bb44b6b780a557daea3656 (127.18.100.67:39571), 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767)
I20260130 02:58:06.864702 19525 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "111287b5557d4abda1c5cd560fb0d019" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
I20260130 02:58:06.865309 19525 raft_consensus.cc:2468] T 111287b5557d4abda1c5cd560fb0d019 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 371a36d219164974abdf848841528152 in term 0.
I20260130 02:58:06.868567 19377 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "111287b5557d4abda1c5cd560fb0d019" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74" is_pre_election: true
I20260130 02:58:06.868669 19525 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6573198830ad45628e71952a7d9c00a0" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
I20260130 02:58:06.869088 19377 raft_consensus.cc:2468] T 111287b5557d4abda1c5cd560fb0d019 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 371a36d219164974abdf848841528152 in term 0.
I20260130 02:58:06.874840 20224 leader_election.cc:290] T 91f954f3ef4244c9a575a7421a7e75eb P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 election: Requested vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767)
I20260130 02:58:06.878847 20208 log.cc:1199] T 927c90b84d7449efa36bc5e2035bf0da P 78c2c46c7520492db35d51c244fb4f74: Deleting WAL directory at /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-1/wal/wals/927c90b84d7449efa36bc5e2035bf0da
I20260130 02:58:06.880494 19625 tablet_bootstrap.cc:654] T 341041cb0be2456e88403367c0814704 P d870164263bb44b6b780a557daea3656: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:06.885854 19377 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6573198830ad45628e71952a7d9c00a0" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74" is_pre_election: true
I20260130 02:58:06.886430 19377 raft_consensus.cc:2468] T 6573198830ad45628e71952a7d9c00a0 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 371a36d219164974abdf848841528152 in term 0.
I20260130 02:58:06.887890 20208 ts_tablet_manager.cc:1950] T 927c90b84d7449efa36bc5e2035bf0da P 78c2c46c7520492db35d51c244fb4f74: Deleting consensus metadata
I20260130 02:58:06.892350 20240 raft_consensus.cc:493] T 54be22352e404ff9b798a075693f7f21 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:06.892745 20240 raft_consensus.cc:515] T 54be22352e404ff9b798a075693f7f21 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
W20260130 02:58:06.895216 18860 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.18.100.67:39571, user_credentials={real_user=slave}} blocked reactor thread for 95581.4us
I20260130 02:58:06.895565 20240 leader_election.cc:290] T 54be22352e404ff9b798a075693f7f21 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), d870164263bb44b6b780a557daea3656 (127.18.100.67:39571)
I20260130 02:58:06.895462 19225 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "54be22352e404ff9b798a075693f7f21" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152" is_pre_election: true
I20260130 02:58:06.896638 19525 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "54be22352e404ff9b798a075693f7f21" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
W20260130 02:58:06.896749 19314 leader_election.cc:343] T 54be22352e404ff9b798a075693f7f21 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:06.897164 19947 ts_tablet_manager.cc:1929] T 927c90b84d7449efa36bc5e2035bf0da P d870164263bb44b6b780a557daea3656: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260130 02:58:06.897514 19947 log.cc:1199] T 927c90b84d7449efa36bc5e2035bf0da P d870164263bb44b6b780a557daea3656: Deleting WAL directory at /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-2/wal/wals/927c90b84d7449efa36bc5e2035bf0da
I20260130 02:58:06.898149 19947 ts_tablet_manager.cc:1950] T 927c90b84d7449efa36bc5e2035bf0da P d870164263bb44b6b780a557daea3656: Deleting consensus metadata
I20260130 02:58:06.903379 19159 leader_election.cc:304] T 111287b5557d4abda1c5cd560fb0d019 P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 371a36d219164974abdf848841528152, 78c2c46c7520492db35d51c244fb4f74; no voters:
W20260130 02:58:06.898219 19313 leader_election.cc:343] T 54be22352e404ff9b798a075693f7f21 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:06.904175 19159 leader_election.cc:304] T 6573198830ad45628e71952a7d9c00a0 P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 371a36d219164974abdf848841528152, 78c2c46c7520492db35d51c244fb4f74; no voters:
I20260130 02:58:06.905628 20178 raft_consensus.cc:2804] T 111287b5557d4abda1c5cd560fb0d019 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260130 02:58:06.905997 20178 raft_consensus.cc:493] T 111287b5557d4abda1c5cd560fb0d019 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260130 02:58:06.906133 20179 raft_consensus.cc:2804] T 6573198830ad45628e71952a7d9c00a0 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260130 02:58:06.906333 20178 raft_consensus.cc:3060] T 111287b5557d4abda1c5cd560fb0d019 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:06.906564 19313 leader_election.cc:304] T 54be22352e404ff9b798a075693f7f21 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78c2c46c7520492db35d51c244fb4f74; no voters: 371a36d219164974abdf848841528152, d870164263bb44b6b780a557daea3656
W20260130 02:58:06.908321 19462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.100.66:40767, user_credentials={real_user=slave}} blocked reactor thread for 56748.6us
I20260130 02:58:06.911267 19947 tablet_replica.cc:333] T 2112b0402de5464cb1caf2c2abb67d78 P d870164263bb44b6b780a557daea3656: stopping tablet replica
I20260130 02:58:06.911929 19947 raft_consensus.cc:2243] T 2112b0402de5464cb1caf2c2abb67d78 P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260130 02:58:06.912436 19947 raft_consensus.cc:2272] T 2112b0402de5464cb1caf2c2abb67d78 P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260130 02:58:06.912694 19376 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "91f954f3ef4244c9a575a7421a7e75eb" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74"
I20260130 02:58:06.913213 19376 raft_consensus.cc:3060] T 91f954f3ef4244c9a575a7421a7e75eb P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:06.914181 19947 ts_tablet_manager.cc:1916] T 2112b0402de5464cb1caf2c2abb67d78 P d870164263bb44b6b780a557daea3656: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260130 02:58:06.917630 20178 raft_consensus.cc:515] T 111287b5557d4abda1c5cd560fb0d019 P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:06.906457 20179 raft_consensus.cc:493] T 6573198830ad45628e71952a7d9c00a0 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260130 02:58:06.919247 20179 raft_consensus.cc:3060] T 6573198830ad45628e71952a7d9c00a0 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:06.919764 19376 raft_consensus.cc:2468] T 91f954f3ef4244c9a575a7421a7e75eb P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d870164263bb44b6b780a557daea3656 in term 1.
I20260130 02:58:06.935371 19807 tablet_bootstrap.cc:492] T 341041cb0be2456e88403367c0814704 P 371a36d219164974abdf848841528152: No bootstrap required, opened a new log
I20260130 02:58:06.935316 18860 catalog_manager.cc:4985] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767): tablet 927c90b84d7449efa36bc5e2035bf0da (table default.Table_ebc8b8379c274c18b0a06a919dbe0ee0 [id=28156e0843674fce88f5914a3e4195ae]) successfully deleted
I20260130 02:58:06.938189 18860 catalog_manager.cc:4985] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): tablet bd43c5d148514836869a8ac648c8ad11 (table default.Table_a9476edf8a5445c883f8d6ce786554a2 [id=7e29f588b41b491e8a9d1ad61e50b033]) successfully deleted
I20260130 02:58:06.940855 20208 tablet_replica.cc:333] T 2112b0402de5464cb1caf2c2abb67d78 P 78c2c46c7520492db35d51c244fb4f74: stopping tablet replica
I20260130 02:58:06.941912 20208 raft_consensus.cc:2243] T 2112b0402de5464cb1caf2c2abb67d78 P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260130 02:58:06.942415 18868 catalog_manager.cc:5941] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4: Sending DeleteTablet for 3 replicas of tablet 3cf5154d8cc44f2bb12978202699c193
I20260130 02:58:06.942633 20208 raft_consensus.cc:2272] T 2112b0402de5464cb1caf2c2abb67d78 P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260130 02:58:06.943817 18868 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:50284 (request call id 190) took 1615 ms. Trace:
I20260130 02:58:06.910776 20194 raft_consensus.cc:515] T b8c579ff5fa84651ba9125457d927162 P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:06.951712 20180 raft_consensus.cc:2749] T 54be22352e404ff9b798a075693f7f21 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260130 02:58:06.952893 20194 leader_election.cc:290] T b8c579ff5fa84651ba9125457d927162 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 election: Requested vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), d870164263bb44b6b780a557daea3656 (127.18.100.67:39571)
I20260130 02:58:06.954375 19525 raft_consensus.cc:3060] T a90e92d569254bbf8a5fbf286baf5cce P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:06.944109 18868 rpcz_store.cc:276] 0130 02:58:05.335648 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:58:06.186603 (+850955us) service_pool.cc:225] Handling call
0130 02:58:06.187822 (+ 1219us) catalog_manager.cc:2393] Looking up, locking, and authorizing table
0130 02:58:06.188356 (+ 534us) catalog_manager.cc:2815] Modifying in-memory table state
0130 02:58:06.188470 (+ 114us) catalog_manager.cc:2824] Locking tablets
0130 02:58:06.443240 (+254770us) catalog_manager.cc:2838] Removing table and tablets from system table
0130 02:58:06.942050 (+498810us) catalog_manager.cc:2856] Removing table from by-name map
0130 02:58:06.942238 (+ 188us) catalog_manager.cc:2872] Committing in-memory state
0130 02:58:06.942321 (+ 83us) catalog_manager.cc:2876] Aborting table tasks
0130 02:58:06.943739 (+ 1418us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0130 02:58:06.446024 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0130 02:58:06.446414 (+ 390us) write_op.cc:432] Acquiring schema lock in shared mode
0130 02:58:06.446436 (+ 22us) write_op.cc:435] Acquired schema lock
0130 02:58:06.446449 (+ 13us) tablet.cc:660] Decoding operations
0130 02:58:06.446764 (+ 315us) write_op.cc:620] Acquiring the partition lock for write op
0130 02:58:06.446805 (+ 41us) write_op.cc:641] Partition lock acquired for write op
0130 02:58:06.446828 (+ 23us) tablet.cc:683] Acquiring locks for 2 operations
0130 02:58:06.447066 (+ 238us) tablet.cc:699] Row locks acquired
0130 02:58:06.447092 (+ 26us) write_op.cc:260] PREPARE: finished
0130 02:58:06.447220 (+ 128us) write_op.cc:270] Start()
0130 02:58:06.447319 (+ 99us) write_op.cc:276] Timestamp: P: 1769741886447140 usec, L: 0
0130 02:58:06.447335 (+ 16us) op_driver.cc:348] REPLICATION: starting
0130 02:58:06.447851 (+ 516us) log.cc:844] Serialized 731 byte log entry
0130 02:58:06.937529 (+489678us) op_driver.cc:464] REPLICATION: finished
0130 02:58:06.938187 (+ 658us) write_op.cc:301] APPLY: starting
0130 02:58:06.938241 (+ 54us) tablet.cc:1365] starting BulkCheckPresence
0130 02:58:06.938341 (+ 100us) tablet.cc:1368] finished BulkCheckPresence
0130 02:58:06.938436 (+ 95us) tablet.cc:1370] starting ApplyRowOperation cycle
0130 02:58:06.938844 (+ 408us) tablet.cc:1381] finished ApplyRowOperation cycle
0130 02:58:06.939003 (+ 159us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0130 02:58:06.939048 (+ 45us) write_op.cc:312] APPLY: finished
0130 02:58:06.939246 (+ 198us) log.cc:844] Serialized 29 byte log entry
0130 02:58:06.939380 (+ 134us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:58:06.939441 (+ 61us) write_op.cc:454] Released schema lock
0130 02:58:06.939471 (+ 30us) write_op.cc:341] FINISH: Updating metrics
0130 02:58:06.939700 (+ 229us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:58:06.939723 (+ 23us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":514,"apply.run_cpu_time_us":1384,"apply.run_wall_time_us":1484,"num_ops":2,"prepare.queue_time_us":905,"prepare.run_cpu_time_us":2027,"prepare.run_wall_time_us":2029,"replication_time_us":490090}]]}
I20260130 02:58:06.966807 20240 consensus_queue.cc:1048] T a90e92d569254bbf8a5fbf286baf5cce P 78c2c46c7520492db35d51c244fb4f74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260130 02:58:06.972039 19947 ts_tablet_manager.cc:1929] T 2112b0402de5464cb1caf2c2abb67d78 P d870164263bb44b6b780a557daea3656: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260130 02:58:06.972514 19947 log.cc:1199] T 2112b0402de5464cb1caf2c2abb67d78 P d870164263bb44b6b780a557daea3656: Deleting WAL directory at /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-2/wal/wals/2112b0402de5464cb1caf2c2abb67d78
I20260130 02:58:06.973222 19947 ts_tablet_manager.cc:1950] T 2112b0402de5464cb1caf2c2abb67d78 P d870164263bb44b6b780a557daea3656: Deleting consensus metadata
I20260130 02:58:06.979064 19525 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b8c579ff5fa84651ba9125457d927162" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656"
W20260130 02:58:06.940047 18860 catalog_manager.cc:4952] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): delete failed for tablet ceda64471a92476f9ba183d2beab30d1 because the tablet was not found. No further retry: Not found: Tablet not found: ceda64471a92476f9ba183d2beab30d1
W20260130 02:58:06.982601 19313 leader_election.cc:343] T b8c579ff5fa84651ba9125457d927162 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): Illegal state: must be running to vote when last-logged opid is not known
W20260130 02:58:06.983575 18860 catalog_manager.cc:4952] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): delete failed for tablet a46702bef0e6442d85e4ca49a5f7cfbb because the tablet was not found. No further retry: Not found: Tablet not found: a46702bef0e6442d85e4ca49a5f7cfbb
I20260130 02:58:06.985199 20208 ts_tablet_manager.cc:1916] T 2112b0402de5464cb1caf2c2abb67d78 P 78c2c46c7520492db35d51c244fb4f74: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260130 02:58:06.986105 18860 catalog_manager.cc:4952] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): delete failed for tablet 7a902d0d571a42bb82ebffd48f7eca3f because the tablet was not found. No further retry: Not found: Tablet not found: 7a902d0d571a42bb82ebffd48f7eca3f
I20260130 02:58:06.987888 18860 catalog_manager.cc:4985] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): tablet 927c90b84d7449efa36bc5e2035bf0da (table default.Table_ebc8b8379c274c18b0a06a919dbe0ee0 [id=28156e0843674fce88f5914a3e4195ae]) successfully deleted
I20260130 02:58:06.993436 18860 catalog_manager.cc:4985] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): tablet 2112b0402de5464cb1caf2c2abb67d78 (table default.Table_da3d581b15a94840898777f74611994e [id=b7e241ced53e4160ac6fa1f65ef3674e]) successfully deleted
I20260130 02:58:06.940243 20178 leader_election.cc:290] T 111287b5557d4abda1c5cd560fb0d019 P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 election: Requested vote from peers d870164263bb44b6b780a557daea3656 (127.18.100.67:39571), 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767)
I20260130 02:58:06.949582 20179 raft_consensus.cc:515] T 6573198830ad45628e71952a7d9c00a0 P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:06.999676 20179 leader_election.cc:290] T 6573198830ad45628e71952a7d9c00a0 P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 election: Requested vote from peers d870164263bb44b6b780a557daea3656 (127.18.100.67:39571), 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767)
I20260130 02:58:06.952827 19225 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b8c579ff5fa84651ba9125457d927162" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152"
I20260130 02:58:07.000476 19225 raft_consensus.cc:3060] T b8c579ff5fa84651ba9125457d927162 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:06.952991 19205 tablet_service.cc:1552] Processing DeleteTablet for tablet 3cf5154d8cc44f2bb12978202699c193 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:58:06 UTC) from {username='slave'} at 127.0.0.1:42870
I20260130 02:58:07.010277 19225 raft_consensus.cc:2468] T b8c579ff5fa84651ba9125457d927162 P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78c2c46c7520492db35d51c244fb4f74 in term 1.
I20260130 02:58:07.016259 19314 leader_election.cc:304] T b8c579ff5fa84651ba9125457d927162 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 371a36d219164974abdf848841528152, 78c2c46c7520492db35d51c244fb4f74; no voters: d870164263bb44b6b780a557daea3656
I20260130 02:58:07.017772 20194 raft_consensus.cc:2804] T b8c579ff5fa84651ba9125457d927162 P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Leader election won for term 1
I20260130 02:58:07.018187 20194 raft_consensus.cc:697] T b8c579ff5fa84651ba9125457d927162 P 78c2c46c7520492db35d51c244fb4f74 [term 1 LEADER]: Becoming Leader. State: Replica: 78c2c46c7520492db35d51c244fb4f74, State: Running, Role: LEADER
I20260130 02:58:07.018925 20194 consensus_queue.cc:237] T b8c579ff5fa84651ba9125457d927162 P 78c2c46c7520492db35d51c244fb4f74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
W20260130 02:58:07.019487 18860 consensus_peers.cc:597] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4 -> Peer 3749baa287ed4f249329a74071781edd (127.18.100.124:34629): Couldn't send request to peer 3749baa287ed4f249329a74071781edd. Status: Network error: Client connection negotiation failed: client connection to 127.18.100.124:34629: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20260130 02:58:06.970460 18875 catalog_manager.cc:5746] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021) does not have the latest schema for tablet f2d01bfdc9c4494386bd02bed655fe63 (table default.Table_89d4117da99842719125411e30d10169 [id=d128767e303145a885eec3c5f714a27f]). Expected version 3 got 2
I20260130 02:58:07.025701 18875 catalog_manager.cc:5654] T 2ca0df71836145748cada4e4559a5656 P 371a36d219164974abdf848841528152 reported cstate change: term changed from 0 to 1, leader changed from <none> to 371a36d219164974abdf848841528152 (127.18.100.65). New cstate: current_term: 1 leader_uuid: "371a36d219164974abdf848841528152" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } health_report { overall_health: UNKNOWN } } }
I20260130 02:58:07.029013 18875 catalog_manager.cc:5654] T 0a763e48583c4e748a46a325746ce69a P 371a36d219164974abdf848841528152 reported cstate change: term changed from 0 to 1, leader changed from <none> to 371a36d219164974abdf848841528152 (127.18.100.65). New cstate: current_term: 1 leader_uuid: "371a36d219164974abdf848841528152" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } health_report { overall_health: UNKNOWN } } }
I20260130 02:58:07.033505 19840 tablet_bootstrap.cc:654] T 3b4908bd1a634ca0907234dd3c39e1c3 P 78c2c46c7520492db35d51c244fb4f74: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:07.041497 19625 tablet_bootstrap.cc:492] T 341041cb0be2456e88403367c0814704 P d870164263bb44b6b780a557daea3656: No bootstrap required, opened a new log
I20260130 02:58:07.041934 19625 ts_tablet_manager.cc:1403] T 341041cb0be2456e88403367c0814704 P d870164263bb44b6b780a557daea3656: Time spent bootstrapping tablet: real 0.249s user 0.008s sys 0.001s
I20260130 02:58:07.043708 19807 ts_tablet_manager.cc:1403] T 341041cb0be2456e88403367c0814704 P 371a36d219164974abdf848841528152: Time spent bootstrapping tablet: real 0.409s user 0.015s sys 0.000s
I20260130 02:58:07.046265 19807 raft_consensus.cc:359] T 341041cb0be2456e88403367c0814704 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:07.046902 19807 raft_consensus.cc:385] T 341041cb0be2456e88403367c0814704 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:07.047155 19807 raft_consensus.cc:740] T 341041cb0be2456e88403367c0814704 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 371a36d219164974abdf848841528152, State: Initialized, Role: FOLLOWER
I20260130 02:58:07.052657 19501 tablet_service.cc:1552] Processing DeleteTablet for tablet 3cf5154d8cc44f2bb12978202699c193 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:58:06 UTC) from {username='slave'} at 127.0.0.1:59068
W20260130 02:58:07.052948 19159 leader_election.cc:343] T 6573198830ad45628e71952a7d9c00a0 P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:07.056221 19352 tablet_service.cc:1552] Processing DeleteTablet for tablet 3cf5154d8cc44f2bb12978202699c193 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:58:06 UTC) from {username='slave'} at 127.0.0.1:60110
I20260130 02:58:07.083207 20240 consensus_queue.cc:1048] T a90e92d569254bbf8a5fbf286baf5cce P 78c2c46c7520492db35d51c244fb4f74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20260130 02:58:07.088193 19840 tablet_bootstrap.cc:492] T 3b4908bd1a634ca0907234dd3c39e1c3 P 78c2c46c7520492db35d51c244fb4f74: No bootstrap required, opened a new log
I20260130 02:58:07.088590 19840 ts_tablet_manager.cc:1403] T 3b4908bd1a634ca0907234dd3c39e1c3 P 78c2c46c7520492db35d51c244fb4f74: Time spent bootstrapping tablet: real 0.256s user 0.010s sys 0.000s
I20260130 02:58:07.093310 20256 tablet_replica.cc:333] T 3cf5154d8cc44f2bb12978202699c193 P 371a36d219164974abdf848841528152: stopping tablet replica
I20260130 02:58:07.094266 20256 raft_consensus.cc:2243] T 3cf5154d8cc44f2bb12978202699c193 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260130 02:58:07.094851 20256 raft_consensus.cc:2272] T 3cf5154d8cc44f2bb12978202699c193 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Raft consensus is shut down!
W20260130 02:58:07.102974 19159 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.100.67:39571, user_credentials={real_user=slave}} blocked reactor thread for 49267.3us
I20260130 02:58:07.105646 19376 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "111287b5557d4abda1c5cd560fb0d019" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74"
I20260130 02:58:07.106401 19376 raft_consensus.cc:3060] T 111287b5557d4abda1c5cd560fb0d019 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:07.113111 19376 raft_consensus.cc:2468] T 111287b5557d4abda1c5cd560fb0d019 P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 371a36d219164974abdf848841528152 in term 1.
I20260130 02:58:07.121258 19525 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "111287b5557d4abda1c5cd560fb0d019" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656"
I20260130 02:58:07.121790 19525 raft_consensus.cc:3060] T 111287b5557d4abda1c5cd560fb0d019 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:07.123997 19840 raft_consensus.cc:359] T 3b4908bd1a634ca0907234dd3c39e1c3 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:07.124701 19840 raft_consensus.cc:385] T 3b4908bd1a634ca0907234dd3c39e1c3 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:07.124959 19840 raft_consensus.cc:740] T 3b4908bd1a634ca0907234dd3c39e1c3 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78c2c46c7520492db35d51c244fb4f74, State: Initialized, Role: FOLLOWER
I20260130 02:58:07.133005 19840 consensus_queue.cc:260] T 3b4908bd1a634ca0907234dd3c39e1c3 P 78c2c46c7520492db35d51c244fb4f74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:07.133607 19525 raft_consensus.cc:2468] T 111287b5557d4abda1c5cd560fb0d019 P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 371a36d219164974abdf848841528152 in term 1.
I20260130 02:58:07.144078 19840 ts_tablet_manager.cc:1434] T 3b4908bd1a634ca0907234dd3c39e1c3 P 78c2c46c7520492db35d51c244fb4f74: Time spent starting tablet: real 0.055s user 0.008s sys 0.000s
I20260130 02:58:07.149008 19625 raft_consensus.cc:359] T 341041cb0be2456e88403367c0814704 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:07.149637 19625 raft_consensus.cc:385] T 341041cb0be2456e88403367c0814704 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:07.149874 19625 raft_consensus.cc:740] T 341041cb0be2456e88403367c0814704 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d870164263bb44b6b780a557daea3656, State: Initialized, Role: FOLLOWER
W20260130 02:58:07.155133 18860 catalog_manager.cc:4965] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): delete failed for tablet 3cf5154d8cc44f2bb12978202699c193 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3cf5154d8cc44f2bb12978202699c193 already in progress: creating tablet
I20260130 02:58:07.156940 19525 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6573198830ad45628e71952a7d9c00a0" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656"
I20260130 02:58:07.159462 19840 tablet_bootstrap.cc:492] T 8bd3e1c06b89426da57e2b259e7983af P 78c2c46c7520492db35d51c244fb4f74: Bootstrap starting.
I20260130 02:58:07.165450 19807 consensus_queue.cc:260] T 341041cb0be2456e88403367c0814704 P 371a36d219164974abdf848841528152 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:07.167804 19807 ts_tablet_manager.cc:1434] T 341041cb0be2456e88403367c0814704 P 371a36d219164974abdf848841528152: Time spent starting tablet: real 0.124s user 0.005s sys 0.000s
I20260130 02:58:07.168860 19807 tablet_bootstrap.cc:492] T 3b4908bd1a634ca0907234dd3c39e1c3 P 371a36d219164974abdf848841528152: Bootstrap starting.
I20260130 02:58:07.172191 19377 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6573198830ad45628e71952a7d9c00a0" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74"
I20260130 02:58:07.172920 19377 raft_consensus.cc:3060] T 6573198830ad45628e71952a7d9c00a0 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:07.174510 20256 ts_tablet_manager.cc:1916] T 3cf5154d8cc44f2bb12978202699c193 P 371a36d219164974abdf848841528152: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260130 02:58:07.193986 20208 ts_tablet_manager.cc:1929] T 2112b0402de5464cb1caf2c2abb67d78 P 78c2c46c7520492db35d51c244fb4f74: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260130 02:58:07.203060 20208 log.cc:1199] T 2112b0402de5464cb1caf2c2abb67d78 P 78c2c46c7520492db35d51c244fb4f74: Deleting WAL directory at /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-1/wal/wals/2112b0402de5464cb1caf2c2abb67d78
I20260130 02:58:07.202646 19159 leader_election.cc:304] T 111287b5557d4abda1c5cd560fb0d019 P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 371a36d219164974abdf848841528152, 78c2c46c7520492db35d51c244fb4f74; no voters:
I20260130 02:58:07.205408 20179 raft_consensus.cc:2804] T 111287b5557d4abda1c5cd560fb0d019 P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Leader election won for term 1
I20260130 02:58:07.205878 20179 raft_consensus.cc:697] T 111287b5557d4abda1c5cd560fb0d019 P 371a36d219164974abdf848841528152 [term 1 LEADER]: Becoming Leader. State: Replica: 371a36d219164974abdf848841528152, State: Running, Role: LEADER
I20260130 02:58:07.206681 20179 consensus_queue.cc:237] T 111287b5557d4abda1c5cd560fb0d019 P 371a36d219164974abdf848841528152 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:07.201283 19625 consensus_queue.cc:260] T 341041cb0be2456e88403367c0814704 P d870164263bb44b6b780a557daea3656 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:07.213557 19625 ts_tablet_manager.cc:1434] T 341041cb0be2456e88403367c0814704 P d870164263bb44b6b780a557daea3656: Time spent starting tablet: real 0.171s user 0.003s sys 0.005s
I20260130 02:58:07.214665 20208 ts_tablet_manager.cc:1950] T 2112b0402de5464cb1caf2c2abb67d78 P 78c2c46c7520492db35d51c244fb4f74: Deleting consensus metadata
I20260130 02:58:07.216740 19377 raft_consensus.cc:2468] T 6573198830ad45628e71952a7d9c00a0 P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 371a36d219164974abdf848841528152 in term 1.
I20260130 02:58:07.229007 20208 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60110 (request call id 181) took 1005 ms. Trace:
I20260130 02:58:07.230543 19625 tablet_bootstrap.cc:492] T 0a763e48583c4e748a46a325746ce69a P d870164263bb44b6b780a557daea3656: Bootstrap starting.
I20260130 02:58:07.229584 20208 rpcz_store.cc:276] 0130 02:58:06.223723 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:58:06.231802 (+ 8079us) service_pool.cc:225] Handling call
0130 02:58:06.940639 (+708837us) ts_tablet_manager.cc:1151] Deleting tablet 2112b0402de5464cb1caf2c2abb67d78
0130 02:58:06.940696 (+ 57us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0130 02:58:07.114887 (+174191us) tablet_metadata.cc:641] Metadata flushed
0130 02:58:07.193683 (+ 78796us) tablet_metadata.cc:641] Metadata flushed
0130 02:58:07.228928 (+ 35245us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":697461}
I20260130 02:58:07.237205 18860 catalog_manager.cc:4985] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767): tablet 2112b0402de5464cb1caf2c2abb67d78 (table default.Table_da3d581b15a94840898777f74611994e [id=b7e241ced53e4160ac6fa1f65ef3674e]) successfully deleted
I20260130 02:58:07.238309 20208 tablet_replica.cc:333] T 3cf5154d8cc44f2bb12978202699c193 P 78c2c46c7520492db35d51c244fb4f74: stopping tablet replica
I20260130 02:58:07.246423 19625 tablet_bootstrap.cc:654] T 0a763e48583c4e748a46a325746ce69a P d870164263bb44b6b780a557daea3656: Neither blocks nor log segments found. Creating new log.
W20260130 02:58:07.235028 19159 leader_election.cc:343] T 6573198830ad45628e71952a7d9c00a0 P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:07.261991 19625 tablet_bootstrap.cc:492] T 0a763e48583c4e748a46a325746ce69a P d870164263bb44b6b780a557daea3656: No bootstrap required, opened a new log
I20260130 02:58:07.263535 19625 ts_tablet_manager.cc:1403] T 0a763e48583c4e748a46a325746ce69a P d870164263bb44b6b780a557daea3656: Time spent bootstrapping tablet: real 0.045s user 0.007s sys 0.003s
I20260130 02:58:07.273273 19159 leader_election.cc:304] T 6573198830ad45628e71952a7d9c00a0 P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 371a36d219164974abdf848841528152, 78c2c46c7520492db35d51c244fb4f74; no voters: d870164263bb44b6b780a557daea3656
I20260130 02:58:07.274464 19807 tablet_bootstrap.cc:654] T 3b4908bd1a634ca0907234dd3c39e1c3 P 371a36d219164974abdf848841528152: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:07.275341 19625 raft_consensus.cc:359] T 0a763e48583c4e748a46a325746ce69a P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:07.275974 19625 raft_consensus.cc:385] T 0a763e48583c4e748a46a325746ce69a P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:07.276228 19625 raft_consensus.cc:740] T 0a763e48583c4e748a46a325746ce69a P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d870164263bb44b6b780a557daea3656, State: Initialized, Role: FOLLOWER
I20260130 02:58:07.276804 19625 consensus_queue.cc:260] T 0a763e48583c4e748a46a325746ce69a P d870164263bb44b6b780a557daea3656 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:07.277665 20243 raft_consensus.cc:2804] T 6573198830ad45628e71952a7d9c00a0 P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Leader election won for term 1
I20260130 02:58:07.278111 20243 raft_consensus.cc:697] T 6573198830ad45628e71952a7d9c00a0 P 371a36d219164974abdf848841528152 [term 1 LEADER]: Becoming Leader. State: Replica: 371a36d219164974abdf848841528152, State: Running, Role: LEADER
I20260130 02:58:07.278842 20243 consensus_queue.cc:237] T 6573198830ad45628e71952a7d9c00a0 P 371a36d219164974abdf848841528152 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:07.244150 19840 tablet_bootstrap.cc:654] T 8bd3e1c06b89426da57e2b259e7983af P 78c2c46c7520492db35d51c244fb4f74: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:07.282627 20208 raft_consensus.cc:2243] T 3cf5154d8cc44f2bb12978202699c193 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260130 02:58:07.285427 19625 ts_tablet_manager.cc:1434] T 0a763e48583c4e748a46a325746ce69a P d870164263bb44b6b780a557daea3656: Time spent starting tablet: real 0.022s user 0.003s sys 0.001s
I20260130 02:58:07.286298 19625 tablet_bootstrap.cc:492] T 3cf5154d8cc44f2bb12978202699c193 P d870164263bb44b6b780a557daea3656: Bootstrap starting.
I20260130 02:58:07.289399 20208 raft_consensus.cc:2272] T 3cf5154d8cc44f2bb12978202699c193 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260130 02:58:07.297317 20186 raft_consensus.cc:493] T 341041cb0be2456e88403367c0814704 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:07.297789 20186 raft_consensus.cc:515] T 341041cb0be2456e88403367c0814704 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:07.299654 20186 leader_election.cc:290] T 341041cb0be2456e88403367c0814704 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767)
I20260130 02:58:07.302125 19222 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "341041cb0be2456e88403367c0814704" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152" is_pre_election: true
I20260130 02:58:07.302727 19222 raft_consensus.cc:2468] T 341041cb0be2456e88403367c0814704 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d870164263bb44b6b780a557daea3656 in term 0.
I20260130 02:58:07.305639 19463 leader_election.cc:304] T 341041cb0be2456e88403367c0814704 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 371a36d219164974abdf848841528152, d870164263bb44b6b780a557daea3656; no voters:
I20260130 02:58:07.300829 19376 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "341041cb0be2456e88403367c0814704" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74" is_pre_election: true
I20260130 02:58:07.306803 19376 raft_consensus.cc:2468] T 341041cb0be2456e88403367c0814704 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d870164263bb44b6b780a557daea3656 in term 0.
I20260130 02:58:07.307884 20186 raft_consensus.cc:2804] T 341041cb0be2456e88403367c0814704 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260130 02:58:07.308207 20186 raft_consensus.cc:493] T 341041cb0be2456e88403367c0814704 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260130 02:58:07.308410 20208 ts_tablet_manager.cc:1916] T 3cf5154d8cc44f2bb12978202699c193 P 78c2c46c7520492db35d51c244fb4f74: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260130 02:58:07.308532 20186 raft_consensus.cc:3060] T 341041cb0be2456e88403367c0814704 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:07.311213 19205 tablet_service.cc:1505] Processing CreateTablet for tablet 25fd48e693424537b91ba7c3fcc96878 (DEFAULT_TABLE table=default.Table_ebc8b8379c274c18b0a06a919dbe0ee0 [id=28156e0843674fce88f5914a3e4195ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:07.312287 19205 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 25fd48e693424537b91ba7c3fcc96878. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:07.315359 18868 catalog_manager.cc:5654] T a90e92d569254bbf8a5fbf286baf5cce P 78c2c46c7520492db35d51c244fb4f74 reported cstate change: term changed from 0 to 1, leader changed from <none> to 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66). New cstate: current_term: 1 leader_uuid: "78c2c46c7520492db35d51c244fb4f74" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } health_report { overall_health: HEALTHY } } }
I20260130 02:58:07.317947 20186 raft_consensus.cc:515] T 341041cb0be2456e88403367c0814704 P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:07.319730 19501 tablet_service.cc:1505] Processing CreateTablet for tablet 25fd48e693424537b91ba7c3fcc96878 (DEFAULT_TABLE table=default.Table_ebc8b8379c274c18b0a06a919dbe0ee0 [id=28156e0843674fce88f5914a3e4195ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:07.321045 19501 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 25fd48e693424537b91ba7c3fcc96878. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:07.312027 19352 tablet_service.cc:1505] Processing CreateTablet for tablet 25fd48e693424537b91ba7c3fcc96878 (DEFAULT_TABLE table=default.Table_ebc8b8379c274c18b0a06a919dbe0ee0 [id=28156e0843674fce88f5914a3e4195ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:07.330525 19222 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "341041cb0be2456e88403367c0814704" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152"
I20260130 02:58:07.331139 19222 raft_consensus.cc:3060] T 341041cb0be2456e88403367c0814704 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:07.336851 19840 tablet_bootstrap.cc:492] T 8bd3e1c06b89426da57e2b259e7983af P 78c2c46c7520492db35d51c244fb4f74: No bootstrap required, opened a new log
I20260130 02:58:07.337345 19840 ts_tablet_manager.cc:1403] T 8bd3e1c06b89426da57e2b259e7983af P 78c2c46c7520492db35d51c244fb4f74: Time spent bootstrapping tablet: real 0.178s user 0.010s sys 0.001s
I20260130 02:58:07.337878 19222 raft_consensus.cc:2468] T 341041cb0be2456e88403367c0814704 P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d870164263bb44b6b780a557daea3656 in term 1.
I20260130 02:58:07.338925 19463 leader_election.cc:304] T 341041cb0be2456e88403367c0814704 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 371a36d219164974abdf848841528152, d870164263bb44b6b780a557daea3656; no voters:
I20260130 02:58:07.339335 19352 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 25fd48e693424537b91ba7c3fcc96878. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:07.339675 20224 raft_consensus.cc:2804] T 341041cb0be2456e88403367c0814704 P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Leader election won for term 1
I20260130 02:58:07.340092 20224 raft_consensus.cc:697] T 341041cb0be2456e88403367c0814704 P d870164263bb44b6b780a557daea3656 [term 1 LEADER]: Becoming Leader. State: Replica: d870164263bb44b6b780a557daea3656, State: Running, Role: LEADER
I20260130 02:58:07.340476 19840 raft_consensus.cc:359] T 8bd3e1c06b89426da57e2b259e7983af P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:07.341202 19840 raft_consensus.cc:385] T 8bd3e1c06b89426da57e2b259e7983af P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:07.340801 20224 consensus_queue.cc:237] T 341041cb0be2456e88403367c0814704 P d870164263bb44b6b780a557daea3656 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:07.341454 19840 raft_consensus.cc:740] T 8bd3e1c06b89426da57e2b259e7983af P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78c2c46c7520492db35d51c244fb4f74, State: Initialized, Role: FOLLOWER
I20260130 02:58:07.342229 19840 consensus_queue.cc:260] T 8bd3e1c06b89426da57e2b259e7983af P 78c2c46c7520492db35d51c244fb4f74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:07.350526 19355 tablet_service.cc:1505] Processing CreateTablet for tablet 838e49e5e8874db393bdc15a1c92f87e (DEFAULT_TABLE table=default.Table_da3d581b15a94840898777f74611994e [id=b7e241ced53e4160ac6fa1f65ef3674e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:07.351586 19355 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 838e49e5e8874db393bdc15a1c92f87e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:07.352142 19201 tablet_service.cc:1505] Processing CreateTablet for tablet 838e49e5e8874db393bdc15a1c92f87e (DEFAULT_TABLE table=default.Table_da3d581b15a94840898777f74611994e [id=b7e241ced53e4160ac6fa1f65ef3674e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:07.353077 19201 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 838e49e5e8874db393bdc15a1c92f87e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:07.355530 20240 raft_consensus.cc:493] T 8bd3e1c06b89426da57e2b259e7983af P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:07.355957 20240 raft_consensus.cc:515] T 8bd3e1c06b89426da57e2b259e7983af P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:07.357723 20240 leader_election.cc:290] T 8bd3e1c06b89426da57e2b259e7983af P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), d870164263bb44b6b780a557daea3656 (127.18.100.67:39571)
I20260130 02:58:07.360718 19225 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8bd3e1c06b89426da57e2b259e7983af" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152" is_pre_election: true
W20260130 02:58:07.363034 19314 leader_election.cc:343] T 8bd3e1c06b89426da57e2b259e7983af P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:07.375890 18868 catalog_manager.cc:5746] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767) does not have the latest schema for tablet 3b148691090e4ee581d46f9845a6e49a (table default.Table_8edea1fae34d4d18b54b0f801a145225 [id=2009496525c94d7f8ee6ce59694b652d]). Expected version 2 got 0
I20260130 02:58:07.378494 18868 catalog_manager.cc:5746] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767) does not have the latest schema for tablet 91f954f3ef4244c9a575a7421a7e75eb (table default.Table_9f83f87711d94e8394abe1925ac84ad1 [id=db23c1be42c44b6f9eb774f14d2b8fd6]). Expected version 1 got 0
I20260130 02:58:07.380540 19840 ts_tablet_manager.cc:1434] T 8bd3e1c06b89426da57e2b259e7983af P 78c2c46c7520492db35d51c244fb4f74: Time spent starting tablet: real 0.043s user 0.005s sys 0.000s
I20260130 02:58:07.384341 19807 tablet_bootstrap.cc:492] T 3b4908bd1a634ca0907234dd3c39e1c3 P 371a36d219164974abdf848841528152: No bootstrap required, opened a new log
I20260130 02:58:07.384759 19807 ts_tablet_manager.cc:1403] T 3b4908bd1a634ca0907234dd3c39e1c3 P 371a36d219164974abdf848841528152: Time spent bootstrapping tablet: real 0.216s user 0.008s sys 0.003s
I20260130 02:58:07.388707 19525 raft_consensus.cc:3060] T 0a763e48583c4e748a46a325746ce69a P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:07.395210 19840 tablet_bootstrap.cc:492] T f54addb1a4304ce69c711ae0cc3b452c P 78c2c46c7520492db35d51c244fb4f74: Bootstrap starting.
I20260130 02:58:07.395824 18872 catalog_manager.cc:5941] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4: Sending DeleteTablet for 3 replicas of tablet 7d76046ee6c8420fa4964decbe8209c1
I20260130 02:58:07.396962 19528 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8bd3e1c06b89426da57e2b259e7983af" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
I20260130 02:58:07.397372 18872 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:50284 (request call id 189) took 2137 ms. Trace:
I20260130 02:58:07.387122 19807 raft_consensus.cc:359] T 3b4908bd1a634ca0907234dd3c39e1c3 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:07.397874 19807 raft_consensus.cc:385] T 3b4908bd1a634ca0907234dd3c39e1c3 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:07.398144 19807 raft_consensus.cc:740] T 3b4908bd1a634ca0907234dd3c39e1c3 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 371a36d219164974abdf848841528152, State: Initialized, Role: FOLLOWER
I20260130 02:58:07.400589 19496 tablet_service.cc:1505] Processing CreateTablet for tablet 838e49e5e8874db393bdc15a1c92f87e (DEFAULT_TABLE table=default.Table_da3d581b15a94840898777f74611994e [id=b7e241ced53e4160ac6fa1f65ef3674e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:07.401674 19496 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 838e49e5e8874db393bdc15a1c92f87e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:07.402778 19505 tablet_service.cc:1552] Processing DeleteTablet for tablet 7d76046ee6c8420fa4964decbe8209c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:58:06 UTC) from {username='slave'} at 127.0.0.1:59068
I20260130 02:58:07.398722 19807 consensus_queue.cc:260] T 3b4908bd1a634ca0907234dd3c39e1c3 P 371a36d219164974abdf848841528152 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:07.397702 18872 rpcz_store.cc:276] 0130 02:58:05.259962 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:58:06.115396 (+855434us) service_pool.cc:225] Handling call
0130 02:58:06.116585 (+ 1189us) catalog_manager.cc:2393] Looking up, locking, and authorizing table
0130 02:58:06.147023 (+ 30438us) catalog_manager.cc:2815] Modifying in-memory table state
0130 02:58:06.147174 (+ 151us) catalog_manager.cc:2824] Locking tablets
0130 02:58:06.147832 (+ 658us) catalog_manager.cc:2838] Removing table and tablets from system table
0130 02:58:06.272258 (+124426us) catalog_manager.cc:2856] Removing table from by-name map
0130 02:58:06.272448 (+ 190us) catalog_manager.cc:2872] Committing in-memory state
0130 02:58:07.395693 (+1123245us) catalog_manager.cc:2876] Aborting table tasks
0130 02:58:07.397299 (+ 1606us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0130 02:58:06.191681 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0130 02:58:06.192084 (+ 403us) write_op.cc:432] Acquiring schema lock in shared mode
0130 02:58:06.192107 (+ 23us) write_op.cc:435] Acquired schema lock
0130 02:58:06.192118 (+ 11us) tablet.cc:660] Decoding operations
0130 02:58:06.192411 (+ 293us) write_op.cc:620] Acquiring the partition lock for write op
0130 02:58:06.192446 (+ 35us) write_op.cc:641] Partition lock acquired for write op
0130 02:58:06.192468 (+ 22us) tablet.cc:683] Acquiring locks for 2 operations
0130 02:58:06.192723 (+ 255us) tablet.cc:699] Row locks acquired
0130 02:58:06.192737 (+ 14us) write_op.cc:260] PREPARE: finished
0130 02:58:06.192841 (+ 104us) write_op.cc:270] Start()
0130 02:58:06.192920 (+ 79us) write_op.cc:276] Timestamp: P: 1769741886192795 usec, L: 0
0130 02:58:06.192934 (+ 14us) op_driver.cc:348] REPLICATION: starting
0130 02:58:06.193431 (+ 497us) log.cc:844] Serialized 796 byte log entry
0130 02:58:06.269716 (+ 76285us) op_driver.cc:464] REPLICATION: finished
0130 02:58:06.270174 (+ 458us) write_op.cc:301] APPLY: starting
0130 02:58:06.270226 (+ 52us) tablet.cc:1365] starting BulkCheckPresence
0130 02:58:06.270327 (+ 101us) tablet.cc:1368] finished BulkCheckPresence
0130 02:58:06.270419 (+ 92us) tablet.cc:1370] starting ApplyRowOperation cycle
0130 02:58:06.270725 (+ 306us) tablet.cc:1381] finished ApplyRowOperation cycle
0130 02:58:06.270863 (+ 138us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0130 02:58:06.270900 (+ 37us) write_op.cc:312] APPLY: finished
0130 02:58:06.271074 (+ 174us) log.cc:844] Serialized 29 byte log entry
0130 02:58:06.271205 (+ 131us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:58:06.271271 (+ 66us) write_op.cc:454] Released schema lock
0130 02:58:06.271301 (+ 30us) write_op.cc:341] FINISH: Updating metrics
0130 02:58:06.271511 (+ 210us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:58:06.271537 (+ 26us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":326,"apply.run_cpu_time_us":1304,"apply.run_wall_time_us":1305,"num_ops":2,"prepare.queue_time_us":42090,"prepare.run_cpu_time_us":2033,"prepare.run_wall_time_us":2034,"raft.queue_time_us":11371,"raft.run_cpu_time_us":13,"raft.run_wall_time_us":15,"replication_time_us":76694}]]}
I20260130 02:58:07.403469 19947 tablet_replica.cc:333] T 7d76046ee6c8420fa4964decbe8209c1 P d870164263bb44b6b780a557daea3656: stopping tablet replica
I20260130 02:58:07.403802 19198 tablet_service.cc:1552] Processing DeleteTablet for tablet 7d76046ee6c8420fa4964decbe8209c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:58:06 UTC) from {username='slave'} at 127.0.0.1:42870
I20260130 02:58:07.404181 19947 raft_consensus.cc:2243] T 7d76046ee6c8420fa4964decbe8209c1 P d870164263bb44b6b780a557daea3656 [term 1 LEADER]: Raft consensus shutting down.
I20260130 02:58:07.404872 19275 heartbeater.cc:344] Connected to a master server at 127.18.100.126:38191
I20260130 02:58:07.405011 19947 raft_consensus.cc:2272] T 7d76046ee6c8420fa4964decbe8209c1 P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260130 02:58:07.408437 19947 ts_tablet_manager.cc:1916] T 7d76046ee6c8420fa4964decbe8209c1 P d870164263bb44b6b780a557daea3656: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260130 02:58:07.415058 20256 ts_tablet_manager.cc:1929] T 3cf5154d8cc44f2bb12978202699c193 P 371a36d219164974abdf848841528152: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260130 02:58:07.435524 20256 log.cc:1199] T 3cf5154d8cc44f2bb12978202699c193 P 371a36d219164974abdf848841528152: Deleting WAL directory at /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-0/wal/wals/3cf5154d8cc44f2bb12978202699c193
I20260130 02:58:07.436519 20256 ts_tablet_manager.cc:1950] T 3cf5154d8cc44f2bb12978202699c193 P 371a36d219164974abdf848841528152: Deleting consensus metadata
I20260130 02:58:07.433099 19807 ts_tablet_manager.cc:1434] T 3b4908bd1a634ca0907234dd3c39e1c3 P 371a36d219164974abdf848841528152: Time spent starting tablet: real 0.048s user 0.005s sys 0.000s
I20260130 02:58:07.440778 19807 tablet_bootstrap.cc:492] T 3b148691090e4ee581d46f9845a6e49a P 371a36d219164974abdf848841528152: Bootstrap starting.
I20260130 02:58:07.444703 19495 tablet_service.cc:1552] Processing DeleteTablet for tablet 37470ce4356f4542b621e21698551209 with delete_type TABLET_DATA_DELETED (replaced by tablet 7caf7cbad5054396b5617094e4d536a0) from {username='slave'} at 127.0.0.1:59068
I20260130 02:58:07.445389 19504 tablet_service.cc:1552] Processing DeleteTablet for tablet c3fce8cc99fb451ebd532fa37390e862 with delete_type TABLET_DATA_DELETED (replaced by tablet 78e9bc3e55fe4e4a98e8eb01da59d365) from {username='slave'} at 127.0.0.1:59068
I20260130 02:58:07.445950 19502 tablet_service.cc:1552] Processing DeleteTablet for tablet ceda64471a92476f9ba183d2beab30d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:58:01 UTC) from {username='slave'} at 127.0.0.1:59068
I20260130 02:58:07.446605 19503 tablet_service.cc:1552] Processing DeleteTablet for tablet 5b2976108584478a98fd90fe6594af66 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:57:58 UTC) from {username='slave'} at 127.0.0.1:59068
I20260130 02:58:07.447459 19500 tablet_service.cc:1552] Processing DeleteTablet for tablet df2c8989d6c1480c97baa23db01141eb with delete_type TABLET_DATA_DELETED (replaced by tablet e2aaaf00f53d418b87d5c7af286e4d84) from {username='slave'} at 127.0.0.1:59068
I20260130 02:58:07.450788 19503 tablet_service.cc:1552] Processing DeleteTablet for tablet d701152960b14396a6ac1bd10480d2b6 with delete_type TABLET_DATA_DELETED (replaced by tablet 481bd4cca6ee4c178873bf12392e3441) from {username='slave'} at 127.0.0.1:59068
I20260130 02:58:07.451848 20256 tablet_replica.cc:333] T 7d76046ee6c8420fa4964decbe8209c1 P 371a36d219164974abdf848841528152: stopping tablet replica
I20260130 02:58:07.450515 19500 tablet_service.cc:1552] Processing DeleteTablet for tablet b7b596f944be4daf953b82b65d5cec2a with delete_type TABLET_DATA_DELETED (replaced by tablet 3389db58493448aea3cf00aca411047c) from {username='slave'} at 127.0.0.1:59068
I20260130 02:58:07.452847 19503 tablet_service.cc:1552] Processing DeleteTablet for tablet a743a82540014c85aeab0e9d0994b0dc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:58:03 UTC) from {username='slave'} at 127.0.0.1:59068
I20260130 02:58:07.453459 19505 tablet_service.cc:1552] Processing DeleteTablet for tablet b3f3177306914e3c8b253db54ee30fb6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:58:03 UTC) from {username='slave'} at 127.0.0.1:59068
I20260130 02:58:07.454085 19502 tablet_service.cc:1552] Processing DeleteTablet for tablet bd43c5d148514836869a8ac648c8ad11 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:58:00 UTC) from {username='slave'} at 127.0.0.1:59068
I20260130 02:58:07.460881 19204 tablet_service.cc:1552] Processing DeleteTablet for tablet d701152960b14396a6ac1bd10480d2b6 with delete_type TABLET_DATA_DELETED (replaced by tablet 481bd4cca6ee4c178873bf12392e3441) from {username='slave'} at 127.0.0.1:42870
I20260130 02:58:07.461174 19200 tablet_service.cc:1552] Processing DeleteTablet for tablet a743a82540014c85aeab0e9d0994b0dc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:58:03 UTC) from {username='slave'} at 127.0.0.1:42870
I20260130 02:58:07.462966 19521 raft_consensus.cc:3060] T 730bd1d7d15b4cbe945b3ef03b2455f7 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:07.476712 18861 catalog_manager.cc:4985] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021): tablet 3cf5154d8cc44f2bb12978202699c193 (table default.Table_bb334df0beee420e883c3225d93892c0 [id=b7b1d250632741a2ad0ed194aae9b197]) successfully deleted
I20260130 02:58:07.461289 20256 raft_consensus.cc:2243] T 7d76046ee6c8420fa4964decbe8209c1 P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260130 02:58:07.474476 20243 consensus_queue.cc:1048] T 730bd1d7d15b4cbe945b3ef03b2455f7 P 371a36d219164974abdf848841528152 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.007s
I20260130 02:58:07.463698 19200 tablet_service.cc:1552] Processing DeleteTablet for tablet bd43c5d148514836869a8ac648c8ad11 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:58:00 UTC) from {username='slave'} at 127.0.0.1:42870
I20260130 02:58:07.480614 20256 raft_consensus.cc:2272] T 7d76046ee6c8420fa4964decbe8209c1 P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260130 02:58:07.463096 19203 tablet_service.cc:1552] Processing DeleteTablet for tablet b3f3177306914e3c8b253db54ee30fb6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:58:03 UTC) from {username='slave'} at 127.0.0.1:42870
I20260130 02:58:07.462738 19197 tablet_service.cc:1552] Processing DeleteTablet for tablet 7a902d0d571a42bb82ebffd48f7eca3f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:57:57 UTC) from {username='slave'} at 127.0.0.1:42870
I20260130 02:58:07.462042 19198 tablet_service.cc:1552] Processing DeleteTablet for tablet 5b2976108584478a98fd90fe6594af66 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:57:58 UTC) from {username='slave'} at 127.0.0.1:42870
I20260130 02:58:07.462388 19199 tablet_service.cc:1552] Processing DeleteTablet for tablet b7b596f944be4daf953b82b65d5cec2a with delete_type TABLET_DATA_DELETED (replaced by tablet 3389db58493448aea3cf00aca411047c) from {username='slave'} at 127.0.0.1:42870
W20260130 02:58:07.434417 18869 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.18.100.66:59691 (request call id 23) took 3147 ms (client timeout 1000 ms). Trace:
W20260130 02:58:07.492089 18869 rpcz_store.cc:269] 0130 02:58:04.286985 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:58:04.371639 (+ 84654us) service_pool.cc:225] Handling call
0130 02:58:04.401386 (+ 29747us) spinlock_profiling.cc:243] Waited 18.7 ms on lock 0x7b5400030554. stack: 00007fb0bd3f7373 00007fb0bd3f7270 00007fb0bcf6c574 0000000000508744 00000000005086d9 00007fb0c44490c0 00007fb0c44e7bba 00007fb0c61e6eac 00007fb0c62e110e 00007fb0c3efe4a4 00007fb0c3efe432 00007fb0c3efe361 00007fb0c3efe2dc 00007fb0c3efd592 00007fb0c21dd34c 00007fb0c21dc786
0130 02:58:07.434342 (+3032956us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":5296,"reported_tablets":29,"spinlock_wait_cycles":43063168}
W20260130 02:58:07.493677 18874 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.18.100.67:47213 (request call id 21) took 2709 ms (client timeout 1000 ms). Trace:
I20260130 02:58:07.500167 19807 tablet_bootstrap.cc:654] T 3b148691090e4ee581d46f9845a6e49a P 371a36d219164974abdf848841528152: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:07.504024 20186 leader_election.cc:290] T 341041cb0be2456e88403367c0814704 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 election: Requested vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767)
W20260130 02:58:07.506618 18871 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.18.100.65:51021 (request call id 26) took 3749 ms (client timeout 1000 ms). Trace:
I20260130 02:58:07.509137 20256 ts_tablet_manager.cc:1916] T 7d76046ee6c8420fa4964decbe8209c1 P 371a36d219164974abdf848841528152: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260130 02:58:07.525919 19521 raft_consensus.cc:3060] T e47b22aafe5247c4b62c4193c24f427c P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:07.532737 19625 tablet_bootstrap.cc:654] T 3cf5154d8cc44f2bb12978202699c193 P d870164263bb44b6b780a557daea3656: Neither blocks nor log segments found. Creating new log.
W20260130 02:58:07.508882 18860 catalog_manager.cc:5058] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): alter failed for tablet 7d76046ee6c8420fa4964decbe8209c1 (table default.Table_55e0ba23f44146339c7c6928440d4948 [id=4c70005d1005479fa6c3a6cb6f5ce658]),no further retry: Illegal state: Tablet not RUNNING: STOPPED
I20260130 02:58:07.544818 19199 tablet_service.cc:1552] Processing DeleteTablet for tablet c3fce8cc99fb451ebd532fa37390e862 with delete_type TABLET_DATA_DELETED (replaced by tablet 78e9bc3e55fe4e4a98e8eb01da59d365) from {username='slave'} at 127.0.0.1:42870
I20260130 02:58:07.546589 20244 raft_consensus.cc:493] T 3b4908bd1a634ca0907234dd3c39e1c3 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:07.547084 20244 raft_consensus.cc:515] T 3b4908bd1a634ca0907234dd3c39e1c3 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:07.548868 20244 leader_election.cc:290] T 3b4908bd1a634ca0907234dd3c39e1c3 P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d870164263bb44b6b780a557daea3656 (127.18.100.67:39571), 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767)
I20260130 02:58:07.546536 19201 tablet_service.cc:1552] Processing DeleteTablet for tablet 481bd4cca6ee4c178873bf12392e3441 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:58:02 UTC) from {username='slave'} at 127.0.0.1:42870
I20260130 02:58:07.550523 19526 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3b4908bd1a634ca0907234dd3c39e1c3" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
I20260130 02:58:07.551851 19199 tablet_service.cc:1552] Processing DeleteTablet for tablet 2112b0402de5464cb1caf2c2abb67d78 with delete_type TABLET_DATA_DELETED (replaced by tablet 838e49e5e8874db393bdc15a1c92f87e) from {username='slave'} at 127.0.0.1:42870
W20260130 02:58:07.552378 19159 leader_election.cc:343] T 3b4908bd1a634ca0907234dd3c39e1c3 P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): Illegal state: must be running to vote when last-logged opid is not known
W20260130 02:58:07.498162 18874 rpcz_store.cc:269] 0130 02:58:04.784505 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:58:04.786232 (+ 1727us) service_pool.cc:225] Handling call
0130 02:58:07.493591 (+2707359us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0130 02:58:05.240824 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0130 02:58:05.241192 (+ 368us) write_op.cc:432] Acquiring schema lock in shared mode
0130 02:58:05.241214 (+ 22us) write_op.cc:435] Acquired schema lock
0130 02:58:05.241226 (+ 12us) tablet.cc:660] Decoding operations
0130 02:58:05.241468 (+ 242us) write_op.cc:620] Acquiring the partition lock for write op
0130 02:58:05.241504 (+ 36us) write_op.cc:641] Partition lock acquired for write op
0130 02:58:05.241525 (+ 21us) tablet.cc:683] Acquiring locks for 1 operations
0130 02:58:05.241677 (+ 152us) tablet.cc:699] Row locks acquired
0130 02:58:05.241689 (+ 12us) write_op.cc:260] PREPARE: finished
0130 02:58:05.241808 (+ 119us) write_op.cc:270] Start()
0130 02:58:05.241887 (+ 79us) write_op.cc:276] Timestamp: P: 1769741885241763 usec, L: 0
0130 02:58:05.241902 (+ 15us) op_driver.cc:348] REPLICATION: starting
0130 02:58:05.242362 (+ 460us) log.cc:844] Serialized 533 byte log entry
0130 02:58:05.323482 (+ 81120us) op_driver.cc:464] REPLICATION: finished
0130 02:58:05.324333 (+ 851us) write_op.cc:301] APPLY: starting
0130 02:58:05.324389 (+ 56us) tablet.cc:1365] starting BulkCheckPresence
0130 02:58:05.324482 (+ 93us) tablet.cc:1368] finished BulkCheckPresence
0130 02:58:05.324567 (+ 85us) tablet.cc:1370] starting ApplyRowOperation cycle
0130 02:58:05.324735 (+ 168us) tablet.cc:1381] finished ApplyRowOperation cycle
0130 02:58:05.324855 (+ 120us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0130 02:58:05.324904 (+ 49us) write_op.cc:312] APPLY: finished
0130 02:58:05.325078 (+ 174us) log.cc:844] Serialized 23 byte log entry
0130 02:58:05.325217 (+ 139us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:58:05.325274 (+ 57us) write_op.cc:454] Released schema lock
0130 02:58:05.325313 (+ 39us) write_op.cc:341] FINISH: Updating metrics
0130 02:58:05.325514 (+ 201us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:58:05.325539 (+ 25us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":29,"child_traces":[["op",{"apply.queue_time_us":695,"apply.run_cpu_time_us":1157,"apply.run_wall_time_us":1157,"num_ops":1,"prepare.queue_time_us":105,"prepare.run_cpu_time_us":1789,"prepare.run_wall_time_us":1790,"raft.queue_time_us":1139,"raft.run_cpu_time_us":13,"raft.run_wall_time_us":14,"replication_time_us":81487}]]}
I20260130 02:58:07.552645 19205 tablet_service.cc:1552] Processing DeleteTablet for tablet df2c8989d6c1480c97baa23db01141eb with delete_type TABLET_DATA_DELETED (replaced by tablet e2aaaf00f53d418b87d5c7af286e4d84) from {username='slave'} at 127.0.0.1:42870
W20260130 02:58:07.497153 18873 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.18.100.66:59691 (request call id 24) took 2090 ms (client timeout 1000 ms). Trace:
I20260130 02:58:07.557045 19201 tablet_service.cc:1552] Processing DeleteTablet for tablet 927c90b84d7449efa36bc5e2035bf0da with delete_type TABLET_DATA_DELETED (replaced by tablet 25fd48e693424537b91ba7c3fcc96878) from {username='slave'} at 127.0.0.1:42870
I20260130 02:58:07.567773 20243 consensus_queue.cc:1048] T 730bd1d7d15b4cbe945b3ef03b2455f7 P 371a36d219164974abdf848841528152 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260130 02:58:07.407938 19313 leader_election.cc:343] T 8bd3e1c06b89426da57e2b259e7983af P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:07.576840 19313 leader_election.cc:304] T 8bd3e1c06b89426da57e2b259e7983af P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78c2c46c7520492db35d51c244fb4f74; no voters: 371a36d219164974abdf848841528152, d870164263bb44b6b780a557daea3656
W20260130 02:58:07.577752 19313 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.100.67:39571, user_credentials={real_user=slave}} blocked reactor thread for 169841us
I20260130 02:58:07.581178 19375 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "341041cb0be2456e88403367c0814704" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74"
I20260130 02:58:07.581669 19375 raft_consensus.cc:3060] T 341041cb0be2456e88403367c0814704 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:07.588464 20180 raft_consensus.cc:2749] T 8bd3e1c06b89426da57e2b259e7983af P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260130 02:58:07.593891 19375 raft_consensus.cc:2468] T 341041cb0be2456e88403367c0814704 P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d870164263bb44b6b780a557daea3656 in term 1.
I20260130 02:58:07.413300 19352 tablet_service.cc:1552] Processing DeleteTablet for tablet 7d76046ee6c8420fa4964decbe8209c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:58:06 UTC) from {username='slave'} at 127.0.0.1:60110
I20260130 02:58:07.447306 19360 tablet_service.cc:1552] Processing DeleteTablet for tablet 481bd4cca6ee4c178873bf12392e3441 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:58:02 UTC) from {username='slave'} at 127.0.0.1:60110
I20260130 02:58:07.448289 19353 tablet_service.cc:1552] Processing DeleteTablet for tablet df2c8989d6c1480c97baa23db01141eb with delete_type TABLET_DATA_DELETED (replaced by tablet e2aaaf00f53d418b87d5c7af286e4d84) from {username='slave'} at 127.0.0.1:60110
I20260130 02:58:07.448781 19354 tablet_service.cc:1552] Processing DeleteTablet for tablet 534dc31a9b7e492c82f3672c75742e6c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:58:04 UTC) from {username='slave'} at 127.0.0.1:60110
I20260130 02:58:07.449245 19350 tablet_service.cc:1552] Processing DeleteTablet for tablet c3fce8cc99fb451ebd532fa37390e862 with delete_type TABLET_DATA_DELETED (replaced by tablet 78e9bc3e55fe4e4a98e8eb01da59d365) from {username='slave'} at 127.0.0.1:60110
I20260130 02:58:07.449692 19351 tablet_service.cc:1552] Processing DeleteTablet for tablet a743a82540014c85aeab0e9d0994b0dc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:58:03 UTC) from {username='slave'} at 127.0.0.1:60110
I20260130 02:58:07.450870 19356 tablet_service.cc:1552] Processing DeleteTablet for tablet b3f3177306914e3c8b253db54ee30fb6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:58:03 UTC) from {username='slave'} at 127.0.0.1:60110
I20260130 02:58:07.451323 19358 tablet_service.cc:1552] Processing DeleteTablet for tablet 730bd1d7d15b4cbe945b3ef03b2455f7 with delete_type TABLET_DATA_DELETED (replaced by tablet 3b02480d9c6f4328af83f7940716d698) from {username='slave'} at 127.0.0.1:60110
I20260130 02:58:07.550243 19377 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3b4908bd1a634ca0907234dd3c39e1c3" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74" is_pre_election: true
I20260130 02:58:07.595878 19807 tablet_bootstrap.cc:492] T 3b148691090e4ee581d46f9845a6e49a P 371a36d219164974abdf848841528152: No bootstrap required, opened a new log
I20260130 02:58:07.554459 19427 heartbeater.cc:344] Connected to a master server at 127.18.100.126:38191
I20260130 02:58:07.571501 20194 raft_consensus.cc:493] T 3b4908bd1a634ca0907234dd3c39e1c3 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:07.596439 20194 raft_consensus.cc:515] T 3b4908bd1a634ca0907234dd3c39e1c3 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:07.598259 20194 leader_election.cc:290] T 3b4908bd1a634ca0907234dd3c39e1c3 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), d870164263bb44b6b780a557daea3656 (127.18.100.67:39571)
I20260130 02:58:07.599093 19377 raft_consensus.cc:2468] T 3b4908bd1a634ca0907234dd3c39e1c3 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 371a36d219164974abdf848841528152 in term 0.
I20260130 02:58:07.599216 19528 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3b4908bd1a634ca0907234dd3c39e1c3" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
W20260130 02:58:07.600198 19313 leader_election.cc:343] T 3b4908bd1a634ca0907234dd3c39e1c3 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:07.601289 19223 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3b4908bd1a634ca0907234dd3c39e1c3" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152" is_pre_election: true
I20260130 02:58:07.601876 19223 raft_consensus.cc:2468] T 3b4908bd1a634ca0907234dd3c39e1c3 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78c2c46c7520492db35d51c244fb4f74 in term 0.
I20260130 02:58:07.602900 19314 leader_election.cc:304] T 3b4908bd1a634ca0907234dd3c39e1c3 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 371a36d219164974abdf848841528152, 78c2c46c7520492db35d51c244fb4f74; no voters: d870164263bb44b6b780a557daea3656
I20260130 02:58:07.604367 20194 raft_consensus.cc:2804] T 3b4908bd1a634ca0907234dd3c39e1c3 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260130 02:58:07.604748 20194 raft_consensus.cc:493] T 3b4908bd1a634ca0907234dd3c39e1c3 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260130 02:58:07.605929 20194 raft_consensus.cc:3060] T 3b4908bd1a634ca0907234dd3c39e1c3 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:07.607836 19947 ts_tablet_manager.cc:1929] T 7d76046ee6c8420fa4964decbe8209c1 P d870164263bb44b6b780a557daea3656: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260130 02:58:07.608443 19947 log.cc:1199] T 7d76046ee6c8420fa4964decbe8209c1 P d870164263bb44b6b780a557daea3656: Deleting WAL directory at /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-2/wal/wals/7d76046ee6c8420fa4964decbe8209c1
I20260130 02:58:07.553985 18874 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50284:
name: "default.Table_a675d7371dcd46b293f893ebe4939e7a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260130 02:58:07.612231 20194 raft_consensus.cc:515] T 3b4908bd1a634ca0907234dd3c39e1c3 P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
W20260130 02:58:07.613063 18874 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a675d7371dcd46b293f893ebe4939e7a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260130 02:58:07.613183 19159 leader_election.cc:304] T 3b4908bd1a634ca0907234dd3c39e1c3 P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 371a36d219164974abdf848841528152, 78c2c46c7520492db35d51c244fb4f74; no voters: d870164263bb44b6b780a557daea3656
W20260130 02:58:07.601912 18860 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.AlterSchema -> {remote=127.18.100.67:39571, user_credentials={real_user=slave}} blocked reactor thread for 93067.6us
I20260130 02:58:07.622853 20244 raft_consensus.cc:2804] T 3b4908bd1a634ca0907234dd3c39e1c3 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260130 02:58:07.623224 20244 raft_consensus.cc:493] T 3b4908bd1a634ca0907234dd3c39e1c3 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260130 02:58:07.623521 20244 raft_consensus.cc:3060] T 3b4908bd1a634ca0907234dd3c39e1c3 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Advancing to term 1
W20260130 02:58:07.543361 18871 rpcz_store.cc:269] 0130 02:58:03.756178 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:58:03.786841 (+ 30663us) service_pool.cc:225] Handling call
0130 02:58:07.505978 (+3719137us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0130 02:58:04.327884 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0130 02:58:04.328340 (+ 456us) write_op.cc:432] Acquiring schema lock in shared mode
0130 02:58:04.328360 (+ 20us) write_op.cc:435] Acquired schema lock
0130 02:58:04.328372 (+ 12us) tablet.cc:660] Decoding operations
0130 02:58:04.328602 (+ 230us) write_op.cc:620] Acquiring the partition lock for write op
0130 02:58:04.328636 (+ 34us) write_op.cc:641] Partition lock acquired for write op
0130 02:58:04.328656 (+ 20us) tablet.cc:683] Acquiring locks for 1 operations
0130 02:58:04.328799 (+ 143us) tablet.cc:699] Row locks acquired
0130 02:58:04.328812 (+ 13us) write_op.cc:260] PREPARE: finished
0130 02:58:04.328984 (+ 172us) write_op.cc:270] Start()
0130 02:58:04.329060 (+ 76us) write_op.cc:276] Timestamp: P: 1769741884328915 usec, L: 0
0130 02:58:04.329075 (+ 15us) op_driver.cc:348] REPLICATION: starting
0130 02:58:04.329594 (+ 519us) log.cc:844] Serialized 533 byte log entry
0130 02:58:04.417660 (+ 88066us) op_driver.cc:464] REPLICATION: finished
0130 02:58:04.418168 (+ 508us) write_op.cc:301] APPLY: starting
0130 02:58:04.418225 (+ 57us) tablet.cc:1365] starting BulkCheckPresence
0130 02:58:04.418304 (+ 79us) tablet.cc:1368] finished BulkCheckPresence
0130 02:58:04.418412 (+ 108us) tablet.cc:1370] starting ApplyRowOperation cycle
0130 02:58:04.418601 (+ 189us) tablet.cc:1381] finished ApplyRowOperation cycle
0130 02:58:04.418730 (+ 129us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0130 02:58:04.418766 (+ 36us) write_op.cc:312] APPLY: finished
0130 02:58:04.418925 (+ 159us) log.cc:844] Serialized 23 byte log entry
0130 02:58:04.419062 (+ 137us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:58:04.419116 (+ 54us) write_op.cc:454] Released schema lock
0130 02:58:04.419145 (+ 29us) write_op.cc:341] FINISH: Updating metrics
0130 02:58:04.419354 (+ 209us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:58:04.419379 (+ 25us) write_op.cc:454] Released schema lock
Related trace 'op':
0130 02:58:04.480827 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0130 02:58:04.481298 (+ 471us) write_op.cc:432] Acquiring schema lock in shared mode
0130 02:58:04.481321 (+ 23us) write_op.cc:435] Acquired schema lock
0130 02:58:04.481331 (+ 10us) tablet.cc:660] Decoding operations
0130 02:58:04.481573 (+ 242us) write_op.cc:620] Acquiring the partition lock for write op
0130 02:58:04.481608 (+ 35us) write_op.cc:641] Partition lock acquired for write op
0130 02:58:04.481628 (+ 20us) tablet.cc:683] Acquiring locks for 1 operations
0130 02:58:04.481774 (+ 146us) tablet.cc:699] Row locks acquired
0130 02:58:04.481786 (+ 12us) write_op.cc:260] PREPARE: finished
0130 02:58:04.481891 (+ 105us) write_op.cc:270] Start()
0130 02:58:04.481966 (+ 75us) write_op.cc:276] Timestamp: P: 1769741884481807 usec, L: 0
0130 02:58:04.481981 (+ 15us) op_driver.cc:348] REPLICATION: starting
0130 02:58:04.482411 (+ 430us) log.cc:844] Serialized 356 byte log entry
0130 02:58:04.597552 (+115141us) op_driver.cc:464] REPLICATION: finished
0130 02:58:04.598631 (+ 1079us) write_op.cc:301] APPLY: starting
0130 02:58:04.598686 (+ 55us) tablet.cc:1365] starting BulkCheckPresence
0130 02:58:04.598780 (+ 94us) tablet.cc:1368] finished BulkCheckPresence
0130 02:58:04.598871 (+ 91us) tablet.cc:1370] starting ApplyRowOperation cycle
0130 02:58:04.599024 (+ 153us) tablet.cc:1381] finished ApplyRowOperation cycle
0130 02:58:04.599145 (+ 121us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0130 02:58:04.599186 (+ 41us) write_op.cc:312] APPLY: finished
0130 02:58:04.599351 (+ 165us) log.cc:844] Serialized 23 byte log entry
0130 02:58:04.599491 (+ 140us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:58:04.599549 (+ 58us) write_op.cc:454] Released schema lock
0130 02:58:04.599579 (+ 30us) write_op.cc:341] FINISH: Updating metrics
0130 02:58:04.599794 (+ 215us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:58:04.599817 (+ 23us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":14998,"reported_tablets":26,"child_traces":[["op",{"apply.queue_time_us":354,"apply.run_cpu_time_us":1167,"apply.run_wall_time_us":1167,"num_ops":1,"prepare.queue_time_us":119,"prepare.run_cpu_time_us":1770,"prepare.run_wall_time_us":10916,"replication_time_us":88503}],["op",{"apply.queue_time_us":921,"apply.run_cpu_time_us":1136,"apply.run_wall_time_us":1136,"num_ops":1,"prepare.queue_time_us":38083,"prepare.run_cpu_time_us":1741,"prepare.run_wall_time_us":15854,"replication_time_us":115483}]]}
I20260130 02:58:07.629087 18871 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50284:
name: "default.Table_477fb92965284222aa1ecaa0107d1ec2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260130 02:58:07.513422 18870 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.18.100.67:47213 (request call id 22) took 1441 ms (client timeout 1000 ms). Trace:
W20260130 02:58:07.631629 18870 rpcz_store.cc:269] 0130 02:58:06.071763 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:58:06.081763 (+ 10000us) service_pool.cc:225] Handling call
0130 02:58:07.513352 (+1431589us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0130 02:58:06.530829 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0130 02:58:06.531226 (+ 397us) write_op.cc:432] Acquiring schema lock in shared mode
0130 02:58:06.531248 (+ 22us) write_op.cc:435] Acquired schema lock
0130 02:58:06.531259 (+ 11us) tablet.cc:660] Decoding operations
0130 02:58:06.531511 (+ 252us) write_op.cc:620] Acquiring the partition lock for write op
0130 02:58:06.531547 (+ 36us) write_op.cc:641] Partition lock acquired for write op
0130 02:58:06.531567 (+ 20us) tablet.cc:683] Acquiring locks for 1 operations
0130 02:58:06.531731 (+ 164us) tablet.cc:699] Row locks acquired
0130 02:58:06.531745 (+ 14us) write_op.cc:260] PREPARE: finished
0130 02:58:06.531856 (+ 111us) write_op.cc:270] Start()
0130 02:58:06.531936 (+ 80us) write_op.cc:276] Timestamp: P: 1769741886531770 usec, L: 0
0130 02:58:06.531950 (+ 14us) op_driver.cc:348] REPLICATION: starting
0130 02:58:06.532421 (+ 471us) log.cc:844] Serialized 533 byte log entry
0130 02:58:06.937742 (+405321us) op_driver.cc:464] REPLICATION: finished
0130 02:58:06.939992 (+ 2250us) write_op.cc:301] APPLY: starting
0130 02:58:06.940042 (+ 50us) tablet.cc:1365] starting BulkCheckPresence
0130 02:58:06.940134 (+ 92us) tablet.cc:1368] finished BulkCheckPresence
0130 02:58:06.940213 (+ 79us) tablet.cc:1370] starting ApplyRowOperation cycle
0130 02:58:06.940343 (+ 130us) tablet.cc:1381] finished ApplyRowOperation cycle
0130 02:58:06.940458 (+ 115us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0130 02:58:06.940497 (+ 39us) write_op.cc:312] APPLY: finished
0130 02:58:06.940661 (+ 164us) log.cc:844] Serialized 23 byte log entry
0130 02:58:06.940902 (+ 241us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:58:06.940958 (+ 56us) write_op.cc:454] Released schema lock
0130 02:58:06.940985 (+ 27us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"mutex_wait_us":26458,"reported_tablets":29,"child_traces":[["op",{"apply.queue_time_us":2109,"apply.run_cpu_time_us":1175,"apply.run_wall_time_us":1175,"num_ops":1,"prepare.queue_time_us":24556,"prepare.run_cpu_time_us":1804,"prepare.run_wall_time_us":1805,"replication_time_us":405727,"thread_start_us":24582,"threads_started":1}]]}
I20260130 02:58:07.645203 20244 raft_consensus.cc:515] T 3b4908bd1a634ca0907234dd3c39e1c3 P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:07.646301 18867 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:50284 (request call id 186) took 2526 ms. Trace:
I20260130 02:58:07.647574 20244 leader_election.cc:290] T 3b4908bd1a634ca0907234dd3c39e1c3 P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 election: Requested vote from peers d870164263bb44b6b780a557daea3656 (127.18.100.67:39571), 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767)
I20260130 02:58:07.646689 18867 rpcz_store.cc:276] 0130 02:58:05.120336 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:58:06.040537 (+920201us) service_pool.cc:225] Handling call
0130 02:58:06.051109 (+ 10572us) catalog_manager.cc:2393] Looking up, locking, and authorizing table
0130 02:58:06.065237 (+ 14128us) catalog_manager.cc:3708] Apply alter schema
0130 02:58:06.065792 (+ 555us) catalog_manager.cc:3740] Acquired catalog manager lock
0130 02:58:06.065993 (+ 201us) catalog_manager.cc:3898] Updating metadata on disk
0130 02:58:06.212222 (+146229us) catalog_manager.cc:3929] Committing alterations to in-memory state
0130 02:58:07.646234 (+1434012us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0130 02:58:06.118067 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0130 02:58:06.118436 (+ 369us) write_op.cc:432] Acquiring schema lock in shared mode
0130 02:58:06.118459 (+ 23us) write_op.cc:435] Acquired schema lock
0130 02:58:06.118471 (+ 12us) tablet.cc:660] Decoding operations
0130 02:58:06.119248 (+ 777us) write_op.cc:620] Acquiring the partition lock for write op
0130 02:58:06.119302 (+ 54us) write_op.cc:641] Partition lock acquired for write op
0130 02:58:06.119330 (+ 28us) tablet.cc:683] Acquiring locks for 1 operations
0130 02:58:06.119470 (+ 140us) tablet.cc:699] Row locks acquired
0130 02:58:06.119483 (+ 13us) write_op.cc:260] PREPARE: finished
0130 02:58:06.119592 (+ 109us) write_op.cc:270] Start()
0130 02:58:06.119667 (+ 75us) write_op.cc:276] Timestamp: P: 1769741886119556 usec, L: 0
0130 02:58:06.119682 (+ 15us) op_driver.cc:348] REPLICATION: starting
0130 02:58:06.120339 (+ 657us) log.cc:844] Serialized 405 byte log entry
0130 02:58:06.206137 (+ 85798us) op_driver.cc:464] REPLICATION: finished
0130 02:58:06.206673 (+ 536us) write_op.cc:301] APPLY: starting
0130 02:58:06.206737 (+ 64us) tablet.cc:1365] starting BulkCheckPresence
0130 02:58:06.206823 (+ 86us) tablet.cc:1368] finished BulkCheckPresence
0130 02:58:06.206912 (+ 89us) tablet.cc:1370] starting ApplyRowOperation cycle
0130 02:58:06.209200 (+ 2288us) tablet.cc:1381] finished ApplyRowOperation cycle
0130 02:58:06.209325 (+ 125us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0130 02:58:06.209366 (+ 41us) write_op.cc:312] APPLY: finished
0130 02:58:06.209532 (+ 166us) log.cc:844] Serialized 23 byte log entry
0130 02:58:06.209665 (+ 133us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:58:06.209723 (+ 58us) write_op.cc:454] Released schema lock
0130 02:58:06.209761 (+ 38us) write_op.cc:341] FINISH: Updating metrics
0130 02:58:06.211351 (+ 1590us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:58:06.211379 (+ 28us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":366,"apply.run_cpu_time_us":3255,"apply.run_wall_time_us":3255,"num_ops":1,"prepare.queue_time_us":785,"prepare.run_cpu_time_us":1919,"prepare.run_wall_time_us":2685,"raft.queue_time_us":80,"raft.run_cpu_time_us":10,"raft.run_wall_time_us":11,"replication_time_us":86363}]]}
I20260130 02:58:07.657115 19807 ts_tablet_manager.cc:1403] T 3b148691090e4ee581d46f9845a6e49a P 371a36d219164974abdf848841528152: Time spent bootstrapping tablet: real 0.217s user 0.004s sys 0.005s
I20260130 02:58:07.615665 19508 tablet_service.cc:1552] Processing DeleteTablet for tablet 2112b0402de5464cb1caf2c2abb67d78 with delete_type TABLET_DATA_DELETED (replaced by tablet 838e49e5e8874db393bdc15a1c92f87e) from {username='slave'} at 127.0.0.1:59068
I20260130 02:58:07.637992 19496 tablet_service.cc:1552] Processing DeleteTablet for tablet 927c90b84d7449efa36bc5e2035bf0da with delete_type TABLET_DATA_DELETED (replaced by tablet 25fd48e693424537b91ba7c3fcc96878) from {username='slave'} at 127.0.0.1:59068
I20260130 02:58:07.655571 19526 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3b4908bd1a634ca0907234dd3c39e1c3" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656"
I20260130 02:58:07.621412 19501 tablet_service.cc:1552] Processing DeleteTablet for tablet 481bd4cca6ee4c178873bf12392e3441 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:58:02 UTC) from {username='slave'} at 127.0.0.1:59068
I20260130 02:58:07.665140 19625 tablet_bootstrap.cc:492] T 3cf5154d8cc44f2bb12978202699c193 P d870164263bb44b6b780a557daea3656: No bootstrap required, opened a new log
I20260130 02:58:07.665547 19625 ts_tablet_manager.cc:1403] T 3cf5154d8cc44f2bb12978202699c193 P d870164263bb44b6b780a557daea3656: Time spent bootstrapping tablet: real 0.379s user 0.009s sys 0.000s
I20260130 02:58:07.668371 19625 raft_consensus.cc:359] T 3cf5154d8cc44f2bb12978202699c193 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:07.668992 19625 raft_consensus.cc:385] T 3cf5154d8cc44f2bb12978202699c193 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:07.669230 19625 raft_consensus.cc:740] T 3cf5154d8cc44f2bb12978202699c193 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d870164263bb44b6b780a557daea3656, State: Initialized, Role: FOLLOWER
I20260130 02:58:07.669785 19625 consensus_queue.cc:260] T 3cf5154d8cc44f2bb12978202699c193 P d870164263bb44b6b780a557daea3656 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
W20260130 02:58:07.603324 18875 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.18.100.65:51021 (request call id 28) took 1717 ms (client timeout 1000 ms). Trace:
W20260130 02:58:07.558872 18873 rpcz_store.cc:269] 0130 02:58:05.407138 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:58:05.411807 (+ 4669us) service_pool.cc:225] Handling call
0130 02:58:07.497087 (+2085280us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2,"reported_tablets":28}
I20260130 02:58:07.673727 18873 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50284:
table { table_name: "default.Table_003c5fa197e648259fe978a5ee4b44c8" } new_table_name: "default.Table_c1886dd9a7934e7797e0e7e0b065d692" modify_external_catalogs: true
I20260130 02:58:07.675300 20284 tablet.cc:1722] T c7c621b99deb457998c44ab1644d3939 P d870164263bb44b6b780a557daea3656: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260130 02:58:07.679916 19947 ts_tablet_manager.cc:1950] T 7d76046ee6c8420fa4964decbe8209c1 P d870164263bb44b6b780a557daea3656: Deleting consensus metadata
W20260130 02:58:07.672135 18875 rpcz_store.cc:269] 0130 02:58:05.885932 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:58:06.029127 (+143195us) service_pool.cc:225] Handling call
0130 02:58:07.096252 (+1067125us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:58:07.096282 (+ 30us) write_op.cc:454] Released schema lock
0130 02:58:07.603234 (+506952us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0130 02:58:07.067287 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0130 02:58:07.067737 (+ 450us) write_op.cc:432] Acquiring schema lock in shared mode
0130 02:58:07.067759 (+ 22us) write_op.cc:435] Acquired schema lock
0130 02:58:07.067776 (+ 17us) tablet.cc:660] Decoding operations
0130 02:58:07.068110 (+ 334us) write_op.cc:620] Acquiring the partition lock for write op
0130 02:58:07.068163 (+ 53us) write_op.cc:641] Partition lock acquired for write op
0130 02:58:07.068187 (+ 24us) tablet.cc:683] Acquiring locks for 2 operations
0130 02:58:07.068462 (+ 275us) tablet.cc:699] Row locks acquired
0130 02:58:07.068478 (+ 16us) write_op.cc:260] PREPARE: finished
0130 02:58:07.068596 (+ 118us) write_op.cc:270] Start()
0130 02:58:07.068681 (+ 85us) write_op.cc:276] Timestamp: P: 1769741887068511 usec, L: 0
0130 02:58:07.068702 (+ 21us) op_driver.cc:348] REPLICATION: starting
0130 02:58:07.069257 (+ 555us) log.cc:844] Serialized 913 byte log entry
0130 02:58:07.084153 (+ 14896us) op_driver.cc:464] REPLICATION: finished
0130 02:58:07.084384 (+ 231us) write_op.cc:301] APPLY: starting
0130 02:58:07.084436 (+ 52us) tablet.cc:1365] starting BulkCheckPresence
0130 02:58:07.084533 (+ 97us) tablet.cc:1368] finished BulkCheckPresence
0130 02:58:07.084640 (+ 107us) tablet.cc:1370] starting ApplyRowOperation cycle
0130 02:58:07.084889 (+ 249us) tablet.cc:1381] finished ApplyRowOperation cycle
0130 02:58:07.085040 (+ 151us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0130 02:58:07.085081 (+ 41us) write_op.cc:312] APPLY: finished
0130 02:58:07.085255 (+ 174us) log.cc:844] Serialized 29 byte log entry
0130 02:58:07.085390 (+ 135us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:58:07.085454 (+ 64us) write_op.cc:454] Released schema lock
0130 02:58:07.085484 (+ 30us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"mutex_wait_us":65535,"reported_tablets":25,"child_traces":[["op",{"apply.queue_time_us":73,"apply.run_cpu_time_us":1312,"apply.run_wall_time_us":1312,"mutex_wait_us":24,"num_ops":2,"prepare.queue_time_us":9442,"prepare.run_cpu_time_us":2325,"prepare.run_wall_time_us":2326,"raft.queue_time_us":368,"raft.run_cpu_time_us":581,"raft.run_wall_time_us":581,"replication_time_us":15370,"spinlock_wait_cycles":121088,"thread_start_us":38332,"threads_started":1}]]}
I20260130 02:58:07.650234 18867 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50284:
name: "default.Table_cda216a3bfc14ec0862b998bedd5b300"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260130 02:58:07.687733 18867 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cda216a3bfc14ec0862b998bedd5b300 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260130 02:58:07.690193 18875 master_service.cc:954] ReplaceTablet: received request to replace tablet 3389db58493448aea3cf00aca411047c from {username='slave'} at 127.0.0.1:50284
I20260130 02:58:07.695302 18860 catalog_manager.cc:4985] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): tablet 7d76046ee6c8420fa4964decbe8209c1 (table default.Table_55e0ba23f44146339c7c6928440d4948 [id=4c70005d1005479fa6c3a6cb6f5ce658]) successfully deleted
W20260130 02:58:07.602468 18866 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.18.100.65:51021 (request call id 27) took 2774 ms (client timeout 1000 ms). Trace:
W20260130 02:58:07.697307 18860 catalog_manager.cc:4952] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): delete failed for tablet 37470ce4356f4542b621e21698551209 because the tablet was not found. No further retry: Not found: Tablet not found: 37470ce4356f4542b621e21698551209
W20260130 02:58:07.697876 18860 catalog_manager.cc:4952] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): delete failed for tablet c3fce8cc99fb451ebd532fa37390e862 because the tablet was not found. No further retry: Not found: Tablet not found: c3fce8cc99fb451ebd532fa37390e862
W20260130 02:58:07.698858 18871 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_477fb92965284222aa1ecaa0107d1ec2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260130 02:58:07.699844 18860 catalog_manager.cc:4952] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): delete failed for tablet ceda64471a92476f9ba183d2beab30d1 because the tablet was not found. No further retry: Not found: Tablet not found: ceda64471a92476f9ba183d2beab30d1
W20260130 02:58:07.703817 18860 catalog_manager.cc:4952] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): delete failed for tablet 5b2976108584478a98fd90fe6594af66 because the tablet was not found. No further retry: Not found: Tablet not found: 5b2976108584478a98fd90fe6594af66
W20260130 02:58:07.702355 19159 outbound_call.cc:321] RPC callback for RPC call kudu.master.MasterService.TSHeartbeat -> {remote=127.18.100.125:39711, user_credentials={real_user=slave}} blocked reactor thread for 48135.9us
I20260130 02:58:07.705468 19625 ts_tablet_manager.cc:1434] T 3cf5154d8cc44f2bb12978202699c193 P d870164263bb44b6b780a557daea3656: Time spent starting tablet: real 0.039s user 0.005s sys 0.000s
I20260130 02:58:07.706338 19625 tablet_bootstrap.cc:492] T 6573198830ad45628e71952a7d9c00a0 P d870164263bb44b6b780a557daea3656: Bootstrap starting.
W20260130 02:58:07.711839 18860 catalog_manager.cc:4952] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): delete failed for tablet df2c8989d6c1480c97baa23db01141eb because the tablet was not found. No further retry: Not found: Tablet not found: df2c8989d6c1480c97baa23db01141eb
W20260130 02:58:07.712913 19159 consensus_peers.cc:597] T bd43c5d148514836869a8ac648c8ad11 P 371a36d219164974abdf848841528152 -> Peer d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): Couldn't send request to peer d870164263bb44b6b780a557daea3656. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: bd43c5d148514836869a8ac648c8ad11. This is attempt 1: this message will repeat every 5th retry.
W20260130 02:58:07.713441 19159 leader_election.cc:343] T 3b4908bd1a634ca0907234dd3c39e1c3 P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:07.635648 18870 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50284:
name: "default.Table_81553b46d7804fd99ef36f431793b001"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260130 02:58:07.653007 19377 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3b4908bd1a634ca0907234dd3c39e1c3" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74"
I20260130 02:58:07.715750 19377 raft_consensus.cc:2393] T 3b4908bd1a634ca0907234dd3c39e1c3 P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 371a36d219164974abdf848841528152 in current term 1: Already voted for candidate 78c2c46c7520492db35d51c244fb4f74 in this term.
W20260130 02:58:07.718475 18860 catalog_manager.cc:4952] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): delete failed for tablet d701152960b14396a6ac1bd10480d2b6 because the tablet was not found. No further retry: Not found: Tablet not found: d701152960b14396a6ac1bd10480d2b6
I20260130 02:58:07.720878 19222 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3b4908bd1a634ca0907234dd3c39e1c3" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152"
I20260130 02:58:07.721714 19222 raft_consensus.cc:2393] T 3b4908bd1a634ca0907234dd3c39e1c3 P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 78c2c46c7520492db35d51c244fb4f74 in current term 1: Already voted for candidate 371a36d219164974abdf848841528152 in this term.
W20260130 02:58:07.721731 18860 catalog_manager.cc:4952] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): delete failed for tablet b7b596f944be4daf953b82b65d5cec2a because the tablet was not found. No further retry: Not found: Tablet not found: b7b596f944be4daf953b82b65d5cec2a
I20260130 02:58:07.724853 19526 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3b4908bd1a634ca0907234dd3c39e1c3" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656"
I20260130 02:58:07.659471 19807 raft_consensus.cc:359] T 3b148691090e4ee581d46f9845a6e49a P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
W20260130 02:58:07.727380 18860 catalog_manager.cc:4952] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): delete failed for tablet a743a82540014c85aeab0e9d0994b0dc because the tablet was not found. No further retry: Not found: Tablet not found: a743a82540014c85aeab0e9d0994b0dc
W20260130 02:58:07.728053 19313 leader_election.cc:343] T 3b4908bd1a634ca0907234dd3c39e1c3 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:07.728477 19313 leader_election.cc:304] T 3b4908bd1a634ca0907234dd3c39e1c3 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78c2c46c7520492db35d51c244fb4f74; no voters: 371a36d219164974abdf848841528152, d870164263bb44b6b780a557daea3656
W20260130 02:58:07.728732 18860 catalog_manager.cc:4952] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): delete failed for tablet b3f3177306914e3c8b253db54ee30fb6 because the tablet was not found. No further retry: Not found: Tablet not found: b3f3177306914e3c8b253db54ee30fb6
I20260130 02:58:07.729207 20240 raft_consensus.cc:2749] T 3b4908bd1a634ca0907234dd3c39e1c3 P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260130 02:58:07.729941 20194 leader_election.cc:290] T 3b4908bd1a634ca0907234dd3c39e1c3 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 election: Requested vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), d870164263bb44b6b780a557daea3656 (127.18.100.67:39571)
W20260130 02:58:07.731985 18860 catalog_manager.cc:4952] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): delete failed for tablet bd43c5d148514836869a8ac648c8ad11 because the tablet was not found. No further retry: Not found: Tablet not found: bd43c5d148514836869a8ac648c8ad11
W20260130 02:58:07.733932 18860 catalog_manager.cc:4952] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): delete failed for tablet 2112b0402de5464cb1caf2c2abb67d78 because the tablet was not found. No further retry: Not found: Tablet not found: 2112b0402de5464cb1caf2c2abb67d78
I20260130 02:58:07.734354 19807 raft_consensus.cc:385] T 3b148691090e4ee581d46f9845a6e49a P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20260130 02:58:07.728787 18870 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_81553b46d7804fd99ef36f431793b001 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260130 02:58:07.736635 19807 raft_consensus.cc:740] T 3b148691090e4ee581d46f9845a6e49a P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 371a36d219164974abdf848841528152, State: Initialized, Role: FOLLOWER
I20260130 02:58:07.737269 19807 consensus_queue.cc:260] T 3b148691090e4ee581d46f9845a6e49a P 371a36d219164974abdf848841528152 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:07.739318 19807 ts_tablet_manager.cc:1434] T 3b148691090e4ee581d46f9845a6e49a P 371a36d219164974abdf848841528152: Time spent starting tablet: real 0.082s user 0.002s sys 0.003s
W20260130 02:58:07.741073 18862 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet 54be22352e404ff9b798a075693f7f21 (table default.Table_3a858a3cb886427cb8268f42c953ad4c [id=17150cb73e374ff9a1e92c43e0c3f2c1], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 54be22352e404ff9b798a075693f7f21 consensus state has no leader
W20260130 02:58:07.743988 18860 catalog_manager.cc:4952] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): delete failed for tablet 927c90b84d7449efa36bc5e2035bf0da because the tablet was not found. No further retry: Not found: Tablet not found: 927c90b84d7449efa36bc5e2035bf0da
I20260130 02:58:07.746973 20286 tablet.cc:1722] T c7c621b99deb457998c44ab1644d3939 P 371a36d219164974abdf848841528152: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260130 02:58:07.749809 20283 tablet.cc:1722] T c7c621b99deb457998c44ab1644d3939 P 78c2c46c7520492db35d51c244fb4f74: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20260130 02:58:07.754861 18860 catalog_manager.cc:4952] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): delete failed for tablet 481bd4cca6ee4c178873bf12392e3441 because the tablet was not found. No further retry: Not found: Tablet not found: 481bd4cca6ee4c178873bf12392e3441
I20260130 02:58:07.762198 20186 consensus_queue.cc:1048] T 91f954f3ef4244c9a575a7421a7e75eb P d870164263bb44b6b780a557daea3656 [LEADER]: Connected to new peer: Peer: permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260130 02:58:07.767167 19625 tablet_bootstrap.cc:654] T 6573198830ad45628e71952a7d9c00a0 P d870164263bb44b6b780a557daea3656: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:07.753789 19840 tablet_bootstrap.cc:654] T f54addb1a4304ce69c711ae0cc3b452c P 78c2c46c7520492db35d51c244fb4f74: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:07.791196 19625 tablet_bootstrap.cc:492] T 6573198830ad45628e71952a7d9c00a0 P d870164263bb44b6b780a557daea3656: No bootstrap required, opened a new log
I20260130 02:58:07.791592 19625 ts_tablet_manager.cc:1403] T 6573198830ad45628e71952a7d9c00a0 P d870164263bb44b6b780a557daea3656: Time spent bootstrapping tablet: real 0.085s user 0.008s sys 0.000s
I20260130 02:58:07.793748 19625 raft_consensus.cc:359] T 6573198830ad45628e71952a7d9c00a0 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:07.811326 19625 raft_consensus.cc:385] T 6573198830ad45628e71952a7d9c00a0 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:07.811623 19625 raft_consensus.cc:740] T 6573198830ad45628e71952a7d9c00a0 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d870164263bb44b6b780a557daea3656, State: Initialized, Role: FOLLOWER
I20260130 02:58:07.752451 19159 leader_election.cc:304] T 3b4908bd1a634ca0907234dd3c39e1c3 P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 371a36d219164974abdf848841528152; no voters: 78c2c46c7520492db35d51c244fb4f74, d870164263bb44b6b780a557daea3656
I20260130 02:58:07.815876 19840 tablet_bootstrap.cc:492] T f54addb1a4304ce69c711ae0cc3b452c P 78c2c46c7520492db35d51c244fb4f74: No bootstrap required, opened a new log
W20260130 02:58:07.816004 19159 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.100.66:40767, user_credentials={real_user=slave}} blocked reactor thread for 63625.5us
I20260130 02:58:07.816282 19840 ts_tablet_manager.cc:1403] T f54addb1a4304ce69c711ae0cc3b452c P 78c2c46c7520492db35d51c244fb4f74: Time spent bootstrapping tablet: real 0.421s user 0.010s sys 0.004s
I20260130 02:58:07.818210 20178 raft_consensus.cc:2749] T 3b4908bd1a634ca0907234dd3c39e1c3 P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260130 02:58:07.819007 19625 consensus_queue.cc:260] T 6573198830ad45628e71952a7d9c00a0 P d870164263bb44b6b780a557daea3656 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:07.819350 19840 raft_consensus.cc:359] T f54addb1a4304ce69c711ae0cc3b452c P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:07.819958 19840 raft_consensus.cc:385] T f54addb1a4304ce69c711ae0cc3b452c P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:07.820214 19840 raft_consensus.cc:740] T f54addb1a4304ce69c711ae0cc3b452c P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78c2c46c7520492db35d51c244fb4f74, State: Initialized, Role: FOLLOWER
I20260130 02:58:07.820832 19840 consensus_queue.cc:260] T f54addb1a4304ce69c711ae0cc3b452c P 78c2c46c7520492db35d51c244fb4f74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:07.822911 19840 ts_tablet_manager.cc:1434] T f54addb1a4304ce69c711ae0cc3b452c P 78c2c46c7520492db35d51c244fb4f74: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260130 02:58:07.823736 19840 tablet_bootstrap.cc:492] T b2ac91da01954c2390b6b7dcc06c394f P 78c2c46c7520492db35d51c244fb4f74: Bootstrap starting.
I20260130 02:58:07.763718 19807 tablet_bootstrap.cc:492] T 54be22352e404ff9b798a075693f7f21 P 371a36d219164974abdf848841528152: Bootstrap starting.
I20260130 02:58:07.826201 20186 consensus_queue.cc:1048] T 91f954f3ef4244c9a575a7421a7e75eb P d870164263bb44b6b780a557daea3656 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260130 02:58:07.829685 19807 tablet_bootstrap.cc:654] T 54be22352e404ff9b798a075693f7f21 P 371a36d219164974abdf848841528152: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:07.792956 20256 ts_tablet_manager.cc:1929] T 7d76046ee6c8420fa4964decbe8209c1 P 371a36d219164974abdf848841528152: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260130 02:58:07.840210 20256 log.cc:1199] T 7d76046ee6c8420fa4964decbe8209c1 P 371a36d219164974abdf848841528152: Deleting WAL directory at /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-0/wal/wals/7d76046ee6c8420fa4964decbe8209c1
I20260130 02:58:07.841112 20256 ts_tablet_manager.cc:1950] T 7d76046ee6c8420fa4964decbe8209c1 P 371a36d219164974abdf848841528152: Deleting consensus metadata
I20260130 02:58:07.785387 20208 ts_tablet_manager.cc:1929] T 3cf5154d8cc44f2bb12978202699c193 P 78c2c46c7520492db35d51c244fb4f74: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260130 02:58:07.842747 20208 log.cc:1199] T 3cf5154d8cc44f2bb12978202699c193 P 78c2c46c7520492db35d51c244fb4f74: Deleting WAL directory at /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-1/wal/wals/3cf5154d8cc44f2bb12978202699c193
I20260130 02:58:07.836175 19625 ts_tablet_manager.cc:1434] T 6573198830ad45628e71952a7d9c00a0 P d870164263bb44b6b780a557daea3656: Time spent starting tablet: real 0.044s user 0.007s sys 0.000s
I20260130 02:58:07.843138 19840 tablet_bootstrap.cc:654] T b2ac91da01954c2390b6b7dcc06c394f P 78c2c46c7520492db35d51c244fb4f74: Neither blocks nor log segments found. Creating new log.
W20260130 02:58:07.697561 18866 rpcz_store.cc:269] 0130 02:58:04.828047 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:58:04.978010 (+149963us) service_pool.cc:225] Handling call
0130 02:58:07.602397 (+2624387us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0130 02:58:05.817727 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0130 02:58:05.818097 (+ 370us) write_op.cc:432] Acquiring schema lock in shared mode
0130 02:58:05.818122 (+ 25us) write_op.cc:435] Acquired schema lock
0130 02:58:05.818136 (+ 14us) tablet.cc:660] Decoding operations
0130 02:58:05.818464 (+ 328us) write_op.cc:620] Acquiring the partition lock for write op
0130 02:58:05.818504 (+ 40us) write_op.cc:641] Partition lock acquired for write op
0130 02:58:05.818531 (+ 27us) tablet.cc:683] Acquiring locks for 2 operations
0130 02:58:05.818810 (+ 279us) tablet.cc:699] Row locks acquired
0130 02:58:05.818824 (+ 14us) write_op.cc:260] PREPARE: finished
0130 02:58:05.818937 (+ 113us) write_op.cc:270] Start()
0130 02:58:05.819020 (+ 83us) write_op.cc:276] Timestamp: P: 1769741885818879 usec, L: 0
0130 02:58:05.819035 (+ 15us) op_driver.cc:348] REPLICATION: starting
0130 02:58:05.819539 (+ 504us) log.cc:844] Serialized 913 byte log entry
0130 02:58:05.877833 (+ 58294us) op_driver.cc:464] REPLICATION: finished
0130 02:58:05.878412 (+ 579us) write_op.cc:301] APPLY: starting
0130 02:58:05.878470 (+ 58us) tablet.cc:1365] starting BulkCheckPresence
0130 02:58:05.878567 (+ 97us) tablet.cc:1368] finished BulkCheckPresence
0130 02:58:05.878836 (+ 269us) tablet.cc:1370] starting ApplyRowOperation cycle
0130 02:58:05.879068 (+ 232us) tablet.cc:1381] finished ApplyRowOperation cycle
0130 02:58:05.879204 (+ 136us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0130 02:58:05.879244 (+ 40us) write_op.cc:312] APPLY: finished
0130 02:58:05.879417 (+ 173us) log.cc:844] Serialized 29 byte log entry
0130 02:58:05.879552 (+ 135us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:58:05.879619 (+ 67us) write_op.cc:454] Released schema lock
0130 02:58:05.879650 (+ 31us) write_op.cc:341] FINISH: Updating metrics
0130 02:58:05.879890 (+ 240us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:58:05.879918 (+ 28us) write_op.cc:454] Released schema lock
Related trace 'op':
0130 02:58:06.017579 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0130 02:58:06.017993 (+ 414us) write_op.cc:432] Acquiring schema lock in shared mode
0130 02:58:06.018017 (+ 24us) write_op.cc:435] Acquired schema lock
0130 02:58:06.018029 (+ 12us) tablet.cc:660] Decoding operations
0130 02:58:06.026903 (+ 8874us) write_op.cc:620] Acquiring the partition lock for write op
0130 02:58:06.026945 (+ 42us) write_op.cc:641] Partition lock acquired for write op
0130 02:58:06.026966 (+ 21us) tablet.cc:683] Acquiring locks for 1 operations
0130 02:58:06.027142 (+ 176us) tablet.cc:699] Row locks acquired
0130 02:58:06.027155 (+ 13us) write_op.cc:260] PREPARE: finished
0130 02:58:06.027268 (+ 113us) write_op.cc:270] Start()
0130 02:58:06.027347 (+ 79us) write_op.cc:276] Timestamp: P: 1769741886027190 usec, L: 0
0130 02:58:06.027362 (+ 15us) op_driver.cc:348] REPLICATION: starting
0130 02:58:06.027821 (+ 459us) log.cc:844] Serialized 356 byte log entry
0130 02:58:06.040867 (+ 13046us) op_driver.cc:464] REPLICATION: finished
0130 02:58:06.041182 (+ 315us) write_op.cc:301] APPLY: starting
0130 02:58:06.041236 (+ 54us) tablet.cc:1365] starting BulkCheckPresence
0130 02:58:06.041322 (+ 86us) tablet.cc:1368] finished BulkCheckPresence
0130 02:58:06.041401 (+ 79us) tablet.cc:1370] starting ApplyRowOperation cycle
0130 02:58:06.041547 (+ 146us) tablet.cc:1381] finished ApplyRowOperation cycle
0130 02:58:06.041671 (+ 124us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0130 02:58:06.041710 (+ 39us) write_op.cc:312] APPLY: finished
0130 02:58:06.041875 (+ 165us) log.cc:844] Serialized 23 byte log entry
0130 02:58:06.042007 (+ 132us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:58:06.042076 (+ 69us) write_op.cc:454] Released schema lock
0130 02:58:06.042104 (+ 28us) write_op.cc:341] FINISH: Updating metrics
0130 02:58:06.042301 (+ 197us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:58:06.042324 (+ 23us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":27,"child_traces":[["op",{"apply.queue_time_us":429,"apply.run_cpu_time_us":1299,"apply.run_wall_time_us":1448,"num_ops":2,"prepare.queue_time_us":52256,"prepare.run_cpu_time_us":2085,"prepare.run_wall_time_us":2085,"raft.queue_time_us":1167,"raft.run_cpu_time_us":560,"raft.run_wall_time_us":561,"replication_time_us":58691}],["op",{"apply.queue_time_us":183,"apply.run_cpu_time_us":1091,"apply.run_wall_time_us":1089,"num_ops":1,"prepare.queue_time_us":72069,"prepare.run_cpu_time_us":2546,"prepare.run_wall_time_us":10542,"raft.queue_time_us":2254,"raft.run_cpu_time_us":596,"raft.run_wall_time_us":598,"replication_time_us":13410}]]}
I20260130 02:58:07.847496 20208 ts_tablet_manager.cc:1950] T 3cf5154d8cc44f2bb12978202699c193 P 78c2c46c7520492db35d51c244fb4f74: Deleting consensus metadata
I20260130 02:58:07.849195 20256 tablet_replica.cc:333] T bd43c5d148514836869a8ac648c8ad11 P 371a36d219164974abdf848841528152: stopping tablet replica
I20260130 02:58:07.850035 20256 raft_consensus.cc:2243] T bd43c5d148514836869a8ac648c8ad11 P 371a36d219164974abdf848841528152 [term 1 LEADER]: Raft consensus shutting down.
I20260130 02:58:07.854856 19625 tablet_bootstrap.cc:492] T 3b4908bd1a634ca0907234dd3c39e1c3 P d870164263bb44b6b780a557daea3656: Bootstrap starting.
I20260130 02:58:07.857483 18860 catalog_manager.cc:4985] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767): tablet 3cf5154d8cc44f2bb12978202699c193 (table default.Table_bb334df0beee420e883c3225d93892c0 [id=b7b1d250632741a2ad0ed194aae9b197]) successfully deleted
I20260130 02:58:07.861300 18861 catalog_manager.cc:4985] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021): tablet 7d76046ee6c8420fa4964decbe8209c1 (table default.Table_55e0ba23f44146339c7c6928440d4948 [id=4c70005d1005479fa6c3a6cb6f5ce658]) successfully deleted
I20260130 02:58:07.861404 20256 raft_consensus.cc:2272] T bd43c5d148514836869a8ac648c8ad11 P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260130 02:58:07.861897 18861 catalog_manager.cc:4952] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021): delete failed for tablet d701152960b14396a6ac1bd10480d2b6 because the tablet was not found. No further retry: Not found: Tablet not found: d701152960b14396a6ac1bd10480d2b6
I20260130 02:58:07.859539 19807 tablet_bootstrap.cc:492] T 54be22352e404ff9b798a075693f7f21 P 371a36d219164974abdf848841528152: No bootstrap required, opened a new log
I20260130 02:58:07.863431 19807 ts_tablet_manager.cc:1403] T 54be22352e404ff9b798a075693f7f21 P 371a36d219164974abdf848841528152: Time spent bootstrapping tablet: real 0.100s user 0.010s sys 0.000s
W20260130 02:58:07.862413 18861 catalog_manager.cc:4952] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021): delete failed for tablet a743a82540014c85aeab0e9d0994b0dc because the tablet was not found. No further retry: Not found: Tablet not found: a743a82540014c85aeab0e9d0994b0dc
I20260130 02:58:07.865681 20208 tablet_replica.cc:333] T 7d76046ee6c8420fa4964decbe8209c1 P 78c2c46c7520492db35d51c244fb4f74: stopping tablet replica
I20260130 02:58:07.865806 19807 raft_consensus.cc:359] T 54be22352e404ff9b798a075693f7f21 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:07.866446 19807 raft_consensus.cc:385] T 54be22352e404ff9b798a075693f7f21 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:07.866801 19807 raft_consensus.cc:740] T 54be22352e404ff9b798a075693f7f21 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 371a36d219164974abdf848841528152, State: Initialized, Role: FOLLOWER
I20260130 02:58:07.867632 18866 master_service.cc:954] ReplaceTablet: received request to replace tablet 5ca9919f80954e74a4d91c94b74de0e4 from {username='slave'} at 127.0.0.1:50284
I20260130 02:58:07.867429 19807 consensus_queue.cc:260] T 54be22352e404ff9b798a075693f7f21 P 371a36d219164974abdf848841528152 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:07.880537 20208 raft_consensus.cc:2243] T 7d76046ee6c8420fa4964decbe8209c1 P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260130 02:58:07.881753 19625 tablet_bootstrap.cc:654] T 3b4908bd1a634ca0907234dd3c39e1c3 P d870164263bb44b6b780a557daea3656: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:07.881731 20208 raft_consensus.cc:2272] T 7d76046ee6c8420fa4964decbe8209c1 P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260130 02:58:07.890163 20208 ts_tablet_manager.cc:1916] T 7d76046ee6c8420fa4964decbe8209c1 P 78c2c46c7520492db35d51c244fb4f74: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260130 02:58:07.910298 20256 ts_tablet_manager.cc:1916] T bd43c5d148514836869a8ac648c8ad11 P 371a36d219164974abdf848841528152: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260130 02:58:07.913125 19625 tablet_bootstrap.cc:492] T 3b4908bd1a634ca0907234dd3c39e1c3 P d870164263bb44b6b780a557daea3656: No bootstrap required, opened a new log
I20260130 02:58:07.913533 19625 ts_tablet_manager.cc:1403] T 3b4908bd1a634ca0907234dd3c39e1c3 P d870164263bb44b6b780a557daea3656: Time spent bootstrapping tablet: real 0.059s user 0.003s sys 0.005s
I20260130 02:58:07.919679 19807 ts_tablet_manager.cc:1434] T 54be22352e404ff9b798a075693f7f21 P 371a36d219164974abdf848841528152: Time spent starting tablet: real 0.056s user 0.006s sys 0.000s
I20260130 02:58:07.897599 20180 consensus_queue.cc:1048] T b8c579ff5fa84651ba9125457d927162 P 78c2c46c7520492db35d51c244fb4f74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260130 02:58:07.908450 19840 tablet_bootstrap.cc:492] T b2ac91da01954c2390b6b7dcc06c394f P 78c2c46c7520492db35d51c244fb4f74: No bootstrap required, opened a new log
I20260130 02:58:07.923197 19840 ts_tablet_manager.cc:1403] T b2ac91da01954c2390b6b7dcc06c394f P 78c2c46c7520492db35d51c244fb4f74: Time spent bootstrapping tablet: real 0.100s user 0.013s sys 0.006s
I20260130 02:58:07.925418 19840 raft_consensus.cc:359] T b2ac91da01954c2390b6b7dcc06c394f P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:07.926061 19840 raft_consensus.cc:385] T b2ac91da01954c2390b6b7dcc06c394f P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:07.926311 19840 raft_consensus.cc:740] T b2ac91da01954c2390b6b7dcc06c394f P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78c2c46c7520492db35d51c244fb4f74, State: Initialized, Role: FOLLOWER
I20260130 02:58:07.926966 19840 consensus_queue.cc:260] T b2ac91da01954c2390b6b7dcc06c394f P 78c2c46c7520492db35d51c244fb4f74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:07.928588 19807 tablet_bootstrap.cc:492] T f54addb1a4304ce69c711ae0cc3b452c P 371a36d219164974abdf848841528152: Bootstrap starting.
I20260130 02:58:07.936158 19840 ts_tablet_manager.cc:1434] T b2ac91da01954c2390b6b7dcc06c394f P 78c2c46c7520492db35d51c244fb4f74: Time spent starting tablet: real 0.013s user 0.005s sys 0.000s
I20260130 02:58:07.937707 19625 raft_consensus.cc:359] T 3b4908bd1a634ca0907234dd3c39e1c3 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:07.938373 19625 raft_consensus.cc:385] T 3b4908bd1a634ca0907234dd3c39e1c3 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:07.938678 19625 raft_consensus.cc:740] T 3b4908bd1a634ca0907234dd3c39e1c3 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d870164263bb44b6b780a557daea3656, State: Initialized, Role: FOLLOWER
I20260130 02:58:07.941643 19807 tablet_bootstrap.cc:654] T f54addb1a4304ce69c711ae0cc3b452c P 371a36d219164974abdf848841528152: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:07.941963 19625 consensus_queue.cc:260] T 3b4908bd1a634ca0907234dd3c39e1c3 P d870164263bb44b6b780a557daea3656 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:07.943269 19840 tablet_bootstrap.cc:492] T 3593f8b89da24ff9b6cf549d5ac58bc6 P 78c2c46c7520492db35d51c244fb4f74: Bootstrap starting.
I20260130 02:58:07.957048 19625 ts_tablet_manager.cc:1434] T 3b4908bd1a634ca0907234dd3c39e1c3 P d870164263bb44b6b780a557daea3656: Time spent starting tablet: real 0.043s user 0.006s sys 0.000s
I20260130 02:58:07.967912 19625 tablet_bootstrap.cc:492] T b8c579ff5fa84651ba9125457d927162 P d870164263bb44b6b780a557daea3656: Bootstrap starting.
W20260130 02:58:07.979840 19313 consensus_peers.cc:597] T b8c579ff5fa84651ba9125457d927162 P 78c2c46c7520492db35d51c244fb4f74 -> Peer d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): Couldn't send request to peer d870164263bb44b6b780a557daea3656. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20260130 02:58:07.993481 20208 ts_tablet_manager.cc:1929] T 7d76046ee6c8420fa4964decbe8209c1 P 78c2c46c7520492db35d51c244fb4f74: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260130 02:58:07.994118 20208 log.cc:1199] T 7d76046ee6c8420fa4964decbe8209c1 P 78c2c46c7520492db35d51c244fb4f74: Deleting WAL directory at /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-1/wal/wals/7d76046ee6c8420fa4964decbe8209c1
I20260130 02:58:07.996078 20208 ts_tablet_manager.cc:1950] T 7d76046ee6c8420fa4964decbe8209c1 P 78c2c46c7520492db35d51c244fb4f74: Deleting consensus metadata
I20260130 02:58:07.998881 18860 catalog_manager.cc:4985] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767): tablet 7d76046ee6c8420fa4964decbe8209c1 (table default.Table_55e0ba23f44146339c7c6928440d4948 [id=4c70005d1005479fa6c3a6cb6f5ce658]) successfully deleted
I20260130 02:58:07.999894 20208 tablet_replica.cc:333] T 481bd4cca6ee4c178873bf12392e3441 P 78c2c46c7520492db35d51c244fb4f74: stopping tablet replica
I20260130 02:58:08.000705 20186 raft_consensus.cc:493] T 6573198830ad45628e71952a7d9c00a0 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:08.000808 20208 raft_consensus.cc:2243] T 481bd4cca6ee4c178873bf12392e3441 P 78c2c46c7520492db35d51c244fb4f74 [term 1 LEADER]: Raft consensus shutting down.
I20260130 02:58:08.001152 20186 raft_consensus.cc:515] T 6573198830ad45628e71952a7d9c00a0 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:08.001716 20208 raft_consensus.cc:2272] T 481bd4cca6ee4c178873bf12392e3441 P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260130 02:58:08.003186 20186 leader_election.cc:290] T 6573198830ad45628e71952a7d9c00a0 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767)
I20260130 02:58:08.004457 19840 tablet_bootstrap.cc:654] T 3593f8b89da24ff9b6cf549d5ac58bc6 P 78c2c46c7520492db35d51c244fb4f74: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:08.004794 19377 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6573198830ad45628e71952a7d9c00a0" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74" is_pre_election: true
I20260130 02:58:08.005470 19377 raft_consensus.cc:2393] T 6573198830ad45628e71952a7d9c00a0 P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d870164263bb44b6b780a557daea3656 in current term 1: Already voted for candidate 371a36d219164974abdf848841528152 in this term.
I20260130 02:58:08.006244 20244 raft_consensus.cc:493] T 3b148691090e4ee581d46f9845a6e49a P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:08.006690 20244 raft_consensus.cc:515] T 3b148691090e4ee581d46f9845a6e49a P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:08.008481 20244 leader_election.cc:290] T 3b148691090e4ee581d46f9845a6e49a P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767), d870164263bb44b6b780a557daea3656 (127.18.100.67:39571)
I20260130 02:58:08.010145 19377 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3b148691090e4ee581d46f9845a6e49a" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74" is_pre_election: true
I20260130 02:58:08.010161 19521 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3b148691090e4ee581d46f9845a6e49a" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
I20260130 02:58:08.010718 19377 raft_consensus.cc:2468] T 3b148691090e4ee581d46f9845a6e49a P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 371a36d219164974abdf848841528152 in term 0.
W20260130 02:58:08.011373 19159 leader_election.cc:343] T 3b148691090e4ee581d46f9845a6e49a P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:08.011618 19223 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6573198830ad45628e71952a7d9c00a0" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152" is_pre_election: true
I20260130 02:58:08.011977 19159 leader_election.cc:304] T 3b148691090e4ee581d46f9845a6e49a P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 371a36d219164974abdf848841528152, 78c2c46c7520492db35d51c244fb4f74; no voters: d870164263bb44b6b780a557daea3656
I20260130 02:58:08.012701 20244 raft_consensus.cc:2804] T 3b148691090e4ee581d46f9845a6e49a P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260130 02:58:08.013304 20244 raft_consensus.cc:493] T 3b148691090e4ee581d46f9845a6e49a P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260130 02:58:08.014900 19840 tablet_bootstrap.cc:492] T 3593f8b89da24ff9b6cf549d5ac58bc6 P 78c2c46c7520492db35d51c244fb4f74: No bootstrap required, opened a new log
I20260130 02:58:08.013669 20244 raft_consensus.cc:3060] T 3b148691090e4ee581d46f9845a6e49a P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:08.015287 19840 ts_tablet_manager.cc:1403] T 3593f8b89da24ff9b6cf549d5ac58bc6 P 78c2c46c7520492db35d51c244fb4f74: Time spent bootstrapping tablet: real 0.072s user 0.010s sys 0.005s
I20260130 02:58:08.017441 19840 raft_consensus.cc:359] T 3593f8b89da24ff9b6cf549d5ac58bc6 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:08.017992 19840 raft_consensus.cc:385] T 3593f8b89da24ff9b6cf549d5ac58bc6 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:08.018235 19840 raft_consensus.cc:740] T 3593f8b89da24ff9b6cf549d5ac58bc6 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78c2c46c7520492db35d51c244fb4f74, State: Initialized, Role: FOLLOWER
I20260130 02:58:08.018929 19840 consensus_queue.cc:260] T 3593f8b89da24ff9b6cf549d5ac58bc6 P 78c2c46c7520492db35d51c244fb4f74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:08.019687 19625 tablet_bootstrap.cc:654] T b8c579ff5fa84651ba9125457d927162 P d870164263bb44b6b780a557daea3656: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:08.020854 19840 ts_tablet_manager.cc:1434] T 3593f8b89da24ff9b6cf549d5ac58bc6 P 78c2c46c7520492db35d51c244fb4f74: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260130 02:58:08.023471 19463 leader_election.cc:304] T 6573198830ad45628e71952a7d9c00a0 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d870164263bb44b6b780a557daea3656; no voters: 371a36d219164974abdf848841528152, 78c2c46c7520492db35d51c244fb4f74
I20260130 02:58:08.024435 20186 raft_consensus.cc:3060] T 6573198830ad45628e71952a7d9c00a0 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:08.027357 19840 tablet_bootstrap.cc:492] T 7d96e93d95f14af79cfb268626559b1e P 78c2c46c7520492db35d51c244fb4f74: Bootstrap starting.
I20260130 02:58:08.027338 20244 raft_consensus.cc:515] T 3b148691090e4ee581d46f9845a6e49a P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:08.029861 19377 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3b148691090e4ee581d46f9845a6e49a" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74"
I20260130 02:58:08.030478 19377 raft_consensus.cc:3060] T 3b148691090e4ee581d46f9845a6e49a P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:08.033479 19840 tablet_bootstrap.cc:654] T 7d96e93d95f14af79cfb268626559b1e P 78c2c46c7520492db35d51c244fb4f74: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:08.034185 20186 raft_consensus.cc:2749] T 6573198830ad45628e71952a7d9c00a0 P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260130 02:58:08.036875 19377 raft_consensus.cc:2468] T 3b148691090e4ee581d46f9845a6e49a P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 371a36d219164974abdf848841528152 in term 1.
I20260130 02:58:08.037894 19159 leader_election.cc:304] T 3b148691090e4ee581d46f9845a6e49a P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 371a36d219164974abdf848841528152, 78c2c46c7520492db35d51c244fb4f74; no voters:
I20260130 02:58:08.038602 20178 raft_consensus.cc:2804] T 3b148691090e4ee581d46f9845a6e49a P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Leader election won for term 1
I20260130 02:58:08.039043 20178 raft_consensus.cc:697] T 3b148691090e4ee581d46f9845a6e49a P 371a36d219164974abdf848841528152 [term 1 LEADER]: Becoming Leader. State: Replica: 371a36d219164974abdf848841528152, State: Running, Role: LEADER
I20260130 02:58:08.039819 20178 consensus_queue.cc:237] T 3b148691090e4ee581d46f9845a6e49a P 371a36d219164974abdf848841528152 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:08.044968 19840 tablet_bootstrap.cc:492] T 7d96e93d95f14af79cfb268626559b1e P 78c2c46c7520492db35d51c244fb4f74: No bootstrap required, opened a new log
I20260130 02:58:08.004805 20208 ts_tablet_manager.cc:1916] T 481bd4cca6ee4c178873bf12392e3441 P 78c2c46c7520492db35d51c244fb4f74: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260130 02:58:08.045377 19840 ts_tablet_manager.cc:1403] T 7d96e93d95f14af79cfb268626559b1e P 78c2c46c7520492db35d51c244fb4f74: Time spent bootstrapping tablet: real 0.018s user 0.006s sys 0.009s
I20260130 02:58:08.049908 19840 raft_consensus.cc:359] T 7d96e93d95f14af79cfb268626559b1e P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:08.050905 19840 raft_consensus.cc:385] T 7d96e93d95f14af79cfb268626559b1e P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:08.051651 19840 raft_consensus.cc:740] T 7d96e93d95f14af79cfb268626559b1e P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78c2c46c7520492db35d51c244fb4f74, State: Initialized, Role: FOLLOWER
I20260130 02:58:08.053539 19840 consensus_queue.cc:260] T 7d96e93d95f14af79cfb268626559b1e P 78c2c46c7520492db35d51c244fb4f74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:08.055305 19625 tablet_bootstrap.cc:492] T b8c579ff5fa84651ba9125457d927162 P d870164263bb44b6b780a557daea3656: No bootstrap required, opened a new log
I20260130 02:58:08.057960 19625 ts_tablet_manager.cc:1403] T b8c579ff5fa84651ba9125457d927162 P d870164263bb44b6b780a557daea3656: Time spent bootstrapping tablet: real 0.090s user 0.009s sys 0.000s
I20260130 02:58:08.059366 18874 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:50284 (request call id 191) took 2589 ms. Trace:
I20260130 02:58:08.059677 18874 rpcz_store.cc:276] 0130 02:58:05.470296 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:58:07.553650 (+2083354us) service_pool.cc:225] Handling call
0130 02:58:07.657365 (+103715us) catalog_manager.cc:2101] Acquired catalog manager lock
0130 02:58:07.659172 (+ 1807us) catalog_manager.cc:2147] Created new table and tablet info
0130 02:58:08.058758 (+399586us) catalog_manager.cc:2207] Wrote table and tablets to system table
0130 02:58:08.059044 (+ 286us) catalog_manager.cc:2241] Inserted table and tablets into CatalogManager maps
0130 02:58:08.059301 (+ 257us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0130 02:58:07.660461 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0130 02:58:07.660912 (+ 451us) write_op.cc:432] Acquiring schema lock in shared mode
0130 02:58:07.660937 (+ 25us) write_op.cc:435] Acquired schema lock
0130 02:58:07.660949 (+ 12us) tablet.cc:660] Decoding operations
0130 02:58:07.661257 (+ 308us) write_op.cc:620] Acquiring the partition lock for write op
0130 02:58:07.661293 (+ 36us) write_op.cc:641] Partition lock acquired for write op
0130 02:58:07.661318 (+ 25us) tablet.cc:683] Acquiring locks for 2 operations
0130 02:58:07.661525 (+ 207us) tablet.cc:699] Row locks acquired
0130 02:58:07.661537 (+ 12us) write_op.cc:260] PREPARE: finished
0130 02:58:07.661658 (+ 121us) write_op.cc:270] Start()
0130 02:58:07.661746 (+ 88us) write_op.cc:276] Timestamp: P: 1769741887661608 usec, L: 0
0130 02:58:07.661764 (+ 18us) op_driver.cc:348] REPLICATION: starting
0130 02:58:07.662303 (+ 539us) log.cc:844] Serialized 433 byte log entry
0130 02:58:08.056428 (+394125us) op_driver.cc:464] REPLICATION: finished
0130 02:58:08.057279 (+ 851us) write_op.cc:301] APPLY: starting
0130 02:58:08.057339 (+ 60us) tablet.cc:1365] starting BulkCheckPresence
0130 02:58:08.057438 (+ 99us) tablet.cc:1368] finished BulkCheckPresence
0130 02:58:08.057539 (+ 101us) tablet.cc:1370] starting ApplyRowOperation cycle
0130 02:58:08.057924 (+ 385us) tablet.cc:1381] finished ApplyRowOperation cycle
0130 02:58:08.058054 (+ 130us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0130 02:58:08.058094 (+ 40us) write_op.cc:312] APPLY: finished
0130 02:58:08.058268 (+ 174us) log.cc:844] Serialized 29 byte log entry
0130 02:58:08.058441 (+ 173us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:58:08.058513 (+ 72us) write_op.cc:454] Released schema lock
0130 02:58:08.058546 (+ 33us) write_op.cc:341] FINISH: Updating metrics
0130 02:58:08.058774 (+ 228us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:58:08.058801 (+ 27us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":705,"apply.run_cpu_time_us":1472,"apply.run_wall_time_us":1472,"num_ops":2,"prepare.queue_time_us":119,"prepare.run_cpu_time_us":2049,"prepare.run_wall_time_us":2050,"replication_time_us":394579}]]}
W20260130 02:58:08.062415 18873 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 345eaaa5625c48d099280b7fbb2e514b consensus state has no leader
I20260130 02:58:08.062894 18873 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:50284 (request call id 195) took 1777 ms. Trace:
I20260130 02:58:08.067368 19840 ts_tablet_manager.cc:1434] T 7d96e93d95f14af79cfb268626559b1e P 78c2c46c7520492db35d51c244fb4f74: Time spent starting tablet: real 0.020s user 0.005s sys 0.000s
I20260130 02:58:08.068222 19840 tablet_bootstrap.cc:492] T b89ed298e3dc49d789d166c613ad7fb8 P 78c2c46c7520492db35d51c244fb4f74: Bootstrap starting.
I20260130 02:58:08.063156 18873 rpcz_store.cc:276] 0130 02:58:06.285509 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:58:07.673426 (+1387917us) service_pool.cc:225] Handling call
0130 02:58:07.674257 (+ 831us) catalog_manager.cc:2393] Looking up, locking, and authorizing table
0130 02:58:07.674966 (+ 709us) catalog_manager.cc:3708] Apply alter schema
0130 02:58:07.675520 (+ 554us) catalog_manager.cc:3740] Acquired catalog manager lock
0130 02:58:07.675720 (+ 200us) catalog_manager.cc:3898] Updating metadata on disk
0130 02:58:08.061970 (+386250us) catalog_manager.cc:3929] Committing alterations to in-memory state
0130 02:58:08.062840 (+ 870us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0130 02:58:07.685594 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0130 02:58:07.686081 (+ 487us) write_op.cc:432] Acquiring schema lock in shared mode
0130 02:58:07.686106 (+ 25us) write_op.cc:435] Acquired schema lock
0130 02:58:07.686121 (+ 15us) tablet.cc:660] Decoding operations
0130 02:58:07.686407 (+ 286us) write_op.cc:620] Acquiring the partition lock for write op
0130 02:58:07.686444 (+ 37us) write_op.cc:641] Partition lock acquired for write op
0130 02:58:07.686470 (+ 26us) tablet.cc:683] Acquiring locks for 1 operations
0130 02:58:07.686790 (+ 320us) tablet.cc:699] Row locks acquired
0130 02:58:07.686818 (+ 28us) write_op.cc:260] PREPARE: finished
0130 02:58:07.686936 (+ 118us) write_op.cc:270] Start()
0130 02:58:07.687023 (+ 87us) write_op.cc:276] Timestamp: P: 1769741887686885 usec, L: 0
0130 02:58:07.687041 (+ 18us) op_driver.cc:348] REPLICATION: starting
0130 02:58:07.687678 (+ 637us) log.cc:844] Serialized 405 byte log entry
0130 02:58:08.056618 (+368940us) op_driver.cc:464] REPLICATION: finished
0130 02:58:08.059025 (+ 2407us) write_op.cc:301] APPLY: starting
0130 02:58:08.059075 (+ 50us) tablet.cc:1365] starting BulkCheckPresence
0130 02:58:08.059150 (+ 75us) tablet.cc:1368] finished BulkCheckPresence
0130 02:58:08.059237 (+ 87us) tablet.cc:1370] starting ApplyRowOperation cycle
0130 02:58:08.059368 (+ 131us) tablet.cc:1381] finished ApplyRowOperation cycle
0130 02:58:08.059481 (+ 113us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0130 02:58:08.059521 (+ 40us) write_op.cc:312] APPLY: finished
0130 02:58:08.059663 (+ 142us) log.cc:844] Serialized 23 byte log entry
0130 02:58:08.059779 (+ 116us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:58:08.059841 (+ 62us) write_op.cc:454] Released schema lock
0130 02:58:08.059873 (+ 32us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":2303,"apply.run_cpu_time_us":1009,"apply.run_wall_time_us":1009,"num_ops":1,"prepare.queue_time_us":8803,"prepare.run_cpu_time_us":2127,"prepare.run_wall_time_us":3333,"replication_time_us":369516,"threads_started":1}]]}
I20260130 02:58:08.071708 20244 leader_election.cc:290] T 3b148691090e4ee581d46f9845a6e49a P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 election: Requested vote from peers 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767), d870164263bb44b6b780a557daea3656 (127.18.100.67:39571)
I20260130 02:58:08.072316 19521 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3b148691090e4ee581d46f9845a6e49a" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656"
W20260130 02:58:08.073951 19159 leader_election.cc:343] T 3b148691090e4ee581d46f9845a6e49a P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:08.084287 19625 raft_consensus.cc:359] T b8c579ff5fa84651ba9125457d927162 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:08.095461 20256 ts_tablet_manager.cc:1929] T bd43c5d148514836869a8ac648c8ad11 P 371a36d219164974abdf848841528152: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260130 02:58:08.098316 19840 tablet_bootstrap.cc:654] T b89ed298e3dc49d789d166c613ad7fb8 P 78c2c46c7520492db35d51c244fb4f74: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:08.099066 20256 log.cc:1199] T bd43c5d148514836869a8ac648c8ad11 P 371a36d219164974abdf848841528152: Deleting WAL directory at /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-0/wal/wals/bd43c5d148514836869a8ac648c8ad11
I20260130 02:58:08.103327 19625 raft_consensus.cc:385] T b8c579ff5fa84651ba9125457d927162 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:08.103621 19625 raft_consensus.cc:740] T b8c579ff5fa84651ba9125457d927162 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d870164263bb44b6b780a557daea3656, State: Initialized, Role: FOLLOWER
I20260130 02:58:08.106196 19625 consensus_queue.cc:260] T b8c579ff5fa84651ba9125457d927162 P d870164263bb44b6b780a557daea3656 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:08.108366 19807 tablet_bootstrap.cc:492] T f54addb1a4304ce69c711ae0cc3b452c P 371a36d219164974abdf848841528152: No bootstrap required, opened a new log
I20260130 02:58:08.108839 19807 ts_tablet_manager.cc:1403] T f54addb1a4304ce69c711ae0cc3b452c P 371a36d219164974abdf848841528152: Time spent bootstrapping tablet: real 0.180s user 0.007s sys 0.007s
I20260130 02:58:08.109659 18875 catalog_manager.cc:5941] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4: Sending DeleteTablet for 3 replicas of tablet 3389db58493448aea3cf00aca411047c
I20260130 02:58:08.110978 18875 catalog_manager.cc:6724] ReplaceTablet: tablet 3389db58493448aea3cf00aca411047c deleted and replaced by tablet af0f90071a334016aa17fae2e3a97fc8
I20260130 02:58:08.103415 20256 ts_tablet_manager.cc:1950] T bd43c5d148514836869a8ac648c8ad11 P 371a36d219164974abdf848841528152: Deleting consensus metadata
I20260130 02:58:08.111320 18875 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:50284 (request call id 196) took 1676 ms. Trace:
I20260130 02:58:08.113059 19807 raft_consensus.cc:359] T f54addb1a4304ce69c711ae0cc3b452c P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:08.114696 19807 raft_consensus.cc:385] T f54addb1a4304ce69c711ae0cc3b452c P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:08.117964 19807 raft_consensus.cc:740] T f54addb1a4304ce69c711ae0cc3b452c P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 371a36d219164974abdf848841528152, State: Initialized, Role: FOLLOWER
I20260130 02:58:08.119251 18861 catalog_manager.cc:4985] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021): tablet bd43c5d148514836869a8ac648c8ad11 (table default.Table_a9476edf8a5445c883f8d6ce786554a2 [id=7e29f588b41b491e8a9d1ad61e50b033]) successfully deleted
W20260130 02:58:08.119906 18861 catalog_manager.cc:4952] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021): delete failed for tablet b3f3177306914e3c8b253db54ee30fb6 because the tablet was not found. No further retry: Not found: Tablet not found: b3f3177306914e3c8b253db54ee30fb6
I20260130 02:58:08.111595 18875 rpcz_store.cc:276] 0130 02:58:06.434940 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:58:07.690070 (+1255130us) service_pool.cc:225] Handling call
0130 02:58:08.111261 (+421191us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0130 02:58:07.739403 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0130 02:58:07.740073 (+ 670us) write_op.cc:432] Acquiring schema lock in shared mode
0130 02:58:07.740099 (+ 26us) write_op.cc:435] Acquired schema lock
0130 02:58:07.740116 (+ 17us) tablet.cc:660] Decoding operations
0130 02:58:07.740438 (+ 322us) write_op.cc:620] Acquiring the partition lock for write op
0130 02:58:07.740479 (+ 41us) write_op.cc:641] Partition lock acquired for write op
0130 02:58:07.740508 (+ 29us) tablet.cc:683] Acquiring locks for 2 operations
0130 02:58:07.740711 (+ 203us) tablet.cc:699] Row locks acquired
0130 02:58:07.740724 (+ 13us) write_op.cc:260] PREPARE: finished
0130 02:58:07.740852 (+ 128us) write_op.cc:270] Start()
0130 02:58:07.740943 (+ 91us) write_op.cc:276] Timestamp: P: 1769741887740797 usec, L: 0
0130 02:58:07.740957 (+ 14us) op_driver.cc:348] REPLICATION: starting
0130 02:58:07.741465 (+ 508us) log.cc:844] Serialized 628 byte log entry
0130 02:58:08.105765 (+364300us) op_driver.cc:464] REPLICATION: finished
0130 02:58:08.106188 (+ 423us) write_op.cc:301] APPLY: starting
0130 02:58:08.106243 (+ 55us) tablet.cc:1365] starting BulkCheckPresence
0130 02:58:08.106345 (+ 102us) tablet.cc:1368] finished BulkCheckPresence
0130 02:58:08.106443 (+ 98us) tablet.cc:1370] starting ApplyRowOperation cycle
0130 02:58:08.106781 (+ 338us) tablet.cc:1381] finished ApplyRowOperation cycle
0130 02:58:08.106981 (+ 200us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0130 02:58:08.107022 (+ 41us) write_op.cc:312] APPLY: finished
0130 02:58:08.107194 (+ 172us) log.cc:844] Serialized 29 byte log entry
0130 02:58:08.108984 (+ 1790us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:58:08.109050 (+ 66us) write_op.cc:454] Released schema lock
0130 02:58:08.109083 (+ 33us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":263,"num_ops":2,"prepare.queue_time_us":22968,"prepare.run_cpu_time_us":2111,"prepare.run_wall_time_us":3364,"replication_time_us":364721}]]}
W20260130 02:58:08.120434 18861 catalog_manager.cc:4952] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021): delete failed for tablet 7a902d0d571a42bb82ebffd48f7eca3f because the tablet was not found. No further retry: Not found: Tablet not found: 7a902d0d571a42bb82ebffd48f7eca3f
I20260130 02:58:08.120440 19201 tablet_service.cc:1552] Processing DeleteTablet for tablet 3389db58493448aea3cf00aca411047c with delete_type TABLET_DATA_DELETED (replaced by tablet af0f90071a334016aa17fae2e3a97fc8) from {username='slave'} at 127.0.0.1:42870
W20260130 02:58:08.120923 18861 catalog_manager.cc:4952] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021): delete failed for tablet 5b2976108584478a98fd90fe6594af66 because the tablet was not found. No further retry: Not found: Tablet not found: 5b2976108584478a98fd90fe6594af66
W20260130 02:58:08.121417 18861 catalog_manager.cc:4952] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021): delete failed for tablet b7b596f944be4daf953b82b65d5cec2a because the tablet was not found. No further retry: Not found: Tablet not found: b7b596f944be4daf953b82b65d5cec2a
W20260130 02:58:08.121915 18861 catalog_manager.cc:4952] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021): delete failed for tablet c3fce8cc99fb451ebd532fa37390e862 because the tablet was not found. No further retry: Not found: Tablet not found: c3fce8cc99fb451ebd532fa37390e862
I20260130 02:58:08.122380 19625 ts_tablet_manager.cc:1434] T b8c579ff5fa84651ba9125457d927162 P d870164263bb44b6b780a557daea3656: Time spent starting tablet: real 0.054s user 0.005s sys 0.000s
I20260130 02:58:08.122900 19501 tablet_service.cc:1552] Processing DeleteTablet for tablet 3389db58493448aea3cf00aca411047c with delete_type TABLET_DATA_DELETED (replaced by tablet af0f90071a334016aa17fae2e3a97fc8) from {username='slave'} at 127.0.0.1:59068
I20260130 02:58:08.122920 19355 tablet_service.cc:1552] Processing DeleteTablet for tablet 3389db58493448aea3cf00aca411047c with delete_type TABLET_DATA_DELETED (replaced by tablet af0f90071a334016aa17fae2e3a97fc8) from {username='slave'} at 127.0.0.1:60110
I20260130 02:58:08.120175 19807 consensus_queue.cc:260] T f54addb1a4304ce69c711ae0cc3b452c P 371a36d219164974abdf848841528152 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:08.129798 18871 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:50284 (request call id 192) took 2462 ms. Trace:
I20260130 02:58:08.130126 18871 rpcz_store.cc:276] 0130 02:58:05.667820 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:58:07.626741 (+1958921us) service_pool.cc:225] Handling call
0130 02:58:07.699291 (+ 72550us) catalog_manager.cc:2101] Acquired catalog manager lock
0130 02:58:07.744746 (+ 45455us) catalog_manager.cc:2147] Created new table and tablet info
0130 02:58:08.129248 (+384502us) catalog_manager.cc:2207] Wrote table and tablets to system table
0130 02:58:08.129508 (+ 260us) catalog_manager.cc:2241] Inserted table and tablets into CatalogManager maps
0130 02:58:08.129745 (+ 237us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0130 02:58:07.865575 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0130 02:58:07.875107 (+ 9532us) write_op.cc:432] Acquiring schema lock in shared mode
0130 02:58:07.875138 (+ 31us) write_op.cc:435] Acquired schema lock
0130 02:58:07.875153 (+ 15us) tablet.cc:660] Decoding operations
0130 02:58:07.875451 (+ 298us) write_op.cc:620] Acquiring the partition lock for write op
0130 02:58:07.875491 (+ 40us) write_op.cc:641] Partition lock acquired for write op
0130 02:58:07.875519 (+ 28us) tablet.cc:683] Acquiring locks for 2 operations
0130 02:58:07.875729 (+ 210us) tablet.cc:699] Row locks acquired
0130 02:58:07.875744 (+ 15us) write_op.cc:260] PREPARE: finished
0130 02:58:07.875870 (+ 126us) write_op.cc:270] Start()
0130 02:58:07.875963 (+ 93us) write_op.cc:276] Timestamp: P: 1769741887875789 usec, L: 0
0130 02:58:07.875981 (+ 18us) op_driver.cc:348] REPLICATION: starting
0130 02:58:07.876532 (+ 551us) log.cc:844] Serialized 433 byte log entry
0130 02:58:08.106006 (+229474us) op_driver.cc:464] REPLICATION: finished
0130 02:58:08.127154 (+ 21148us) write_op.cc:301] APPLY: starting
0130 02:58:08.127205 (+ 51us) tablet.cc:1365] starting BulkCheckPresence
0130 02:58:08.127302 (+ 97us) tablet.cc:1368] finished BulkCheckPresence
0130 02:58:08.127405 (+ 103us) tablet.cc:1370] starting ApplyRowOperation cycle
0130 02:58:08.127799 (+ 394us) tablet.cc:1381] finished ApplyRowOperation cycle
0130 02:58:08.127934 (+ 135us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0130 02:58:08.127976 (+ 42us) write_op.cc:312] APPLY: finished
0130 02:58:08.128144 (+ 168us) log.cc:844] Serialized 29 byte log entry
0130 02:58:08.128984 (+ 840us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:58:08.129047 (+ 63us) write_op.cc:454] Released schema lock
0130 02:58:08.129077 (+ 30us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":21030,"num_ops":2,"prepare.queue_time_us":40142,"prepare.run_cpu_time_us":2212,"prepare.run_wall_time_us":31331,"replication_time_us":229956}]]}
I20260130 02:58:08.132695 19807 ts_tablet_manager.cc:1434] T f54addb1a4304ce69c711ae0cc3b452c P 371a36d219164974abdf848841528152: Time spent starting tablet: real 0.024s user 0.004s sys 0.000s
I20260130 02:58:08.136056 20244 consensus_queue.cc:1048] T 6573198830ad45628e71952a7d9c00a0 P 371a36d219164974abdf848841528152 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260130 02:58:08.117705 20256 tablet_replica.cc:333] T 481bd4cca6ee4c178873bf12392e3441 P 371a36d219164974abdf848841528152: stopping tablet replica
W20260130 02:58:08.137775 18860 catalog_manager.cc:4965] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): delete failed for tablet 3389db58493448aea3cf00aca411047c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3389db58493448aea3cf00aca411047c already in progress: creating tablet
I20260130 02:58:08.141654 20208 ts_tablet_manager.cc:1929] T 481bd4cca6ee4c178873bf12392e3441 P 78c2c46c7520492db35d51c244fb4f74: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260130 02:58:08.138031 20256 raft_consensus.cc:2243] T 481bd4cca6ee4c178873bf12392e3441 P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260130 02:58:08.142458 19807 tablet_bootstrap.cc:492] T 8bd3e1c06b89426da57e2b259e7983af P 371a36d219164974abdf848841528152: Bootstrap starting.
I20260130 02:58:08.150738 20244 raft_consensus.cc:493] T 54be22352e404ff9b798a075693f7f21 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:08.151326 20244 raft_consensus.cc:515] T 54be22352e404ff9b798a075693f7f21 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:08.153302 20244 leader_election.cc:290] T 54be22352e404ff9b798a075693f7f21 P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767), d870164263bb44b6b780a557daea3656 (127.18.100.67:39571)
I20260130 02:58:08.143070 20256 raft_consensus.cc:2272] T 481bd4cca6ee4c178873bf12392e3441 P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260130 02:58:08.154740 19526 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "54be22352e404ff9b798a075693f7f21" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
W20260130 02:58:08.156044 19159 leader_election.cc:343] T 54be22352e404ff9b798a075693f7f21 P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:08.157855 20256 ts_tablet_manager.cc:1916] T 481bd4cca6ee4c178873bf12392e3441 P 371a36d219164974abdf848841528152: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260130 02:58:08.156782 19377 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "54be22352e404ff9b798a075693f7f21" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74" is_pre_election: true
I20260130 02:58:08.167680 19377 raft_consensus.cc:2468] T 54be22352e404ff9b798a075693f7f21 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 371a36d219164974abdf848841528152 in term 0.
I20260130 02:58:08.167866 18870 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:50284 (request call id 193) took 2499 ms. Trace:
I20260130 02:58:08.169143 20208 log.cc:1199] T 481bd4cca6ee4c178873bf12392e3441 P 78c2c46c7520492db35d51c244fb4f74: Deleting WAL directory at /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-1/wal/wals/481bd4cca6ee4c178873bf12392e3441
I20260130 02:58:08.169900 20208 ts_tablet_manager.cc:1950] T 481bd4cca6ee4c178873bf12392e3441 P 78c2c46c7520492db35d51c244fb4f74: Deleting consensus metadata
I20260130 02:58:08.171236 18867 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:50284 (request call id 194) took 2503 ms. Trace:
I20260130 02:58:08.172289 18860 catalog_manager.cc:4985] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767): tablet 481bd4cca6ee4c178873bf12392e3441 (table default.Table_e3b63d7d5d874900aac850654d98d62e [id=bc8cc55916fa4a41b78019f59e6fbe39]) successfully deleted
I20260130 02:58:08.168207 18870 rpcz_store.cc:276] 0130 02:58:05.668055 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:58:07.635299 (+1967244us) service_pool.cc:225] Handling call
0130 02:58:07.735833 (+100534us) catalog_manager.cc:2101] Acquired catalog manager lock
0130 02:58:07.866997 (+131164us) catalog_manager.cc:2147] Created new table and tablet info
0130 02:58:08.167298 (+300301us) catalog_manager.cc:2207] Wrote table and tablets to system table
0130 02:58:08.167566 (+ 268us) catalog_manager.cc:2241] Inserted table and tablets into CatalogManager maps
0130 02:58:08.167803 (+ 237us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0130 02:58:07.884969 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0130 02:58:07.885417 (+ 448us) write_op.cc:432] Acquiring schema lock in shared mode
0130 02:58:07.885442 (+ 25us) write_op.cc:435] Acquired schema lock
0130 02:58:07.885457 (+ 15us) tablet.cc:660] Decoding operations
0130 02:58:07.885732 (+ 275us) write_op.cc:620] Acquiring the partition lock for write op
0130 02:58:07.885769 (+ 37us) write_op.cc:641] Partition lock acquired for write op
0130 02:58:07.885815 (+ 46us) tablet.cc:683] Acquiring locks for 2 operations
0130 02:58:07.886024 (+ 209us) tablet.cc:699] Row locks acquired
0130 02:58:07.886041 (+ 17us) write_op.cc:260] PREPARE: finished
0130 02:58:07.886159 (+ 118us) write_op.cc:270] Start()
0130 02:58:07.886245 (+ 86us) write_op.cc:276] Timestamp: P: 1769741887886087 usec, L: 0
0130 02:58:07.886265 (+ 20us) op_driver.cc:348] REPLICATION: starting
0130 02:58:07.893029 (+ 6764us) log.cc:844] Serialized 433 byte log entry
0130 02:58:07.908467 (+ 15438us) spinlock_profiling.cc:243] Waited 6.25 ms on lock 0x7b5400030554. stack: 00007fb0bd3f7373 00007fb0bd3f7270 00007fb0bcf6c574 0000000000508744 00000000005086d9 00007fb0c44485a1 00007fb0c44daaa8 00007fb0c494c4d1 00007fb0c494bca8 00007fb0c4950941 00007fb0c49508f9 00007fb0c4950889 00007fb0c4950851 00007fb0c494fa7d 00000000005aa2b4 00000000005aa0e9
0130 02:58:08.158609 (+250142us) op_driver.cc:464] REPLICATION: finished
0130 02:58:08.159225 (+ 616us) write_op.cc:301] APPLY: starting
0130 02:58:08.159280 (+ 55us) tablet.cc:1365] starting BulkCheckPresence
0130 02:58:08.159393 (+ 113us) tablet.cc:1368] finished BulkCheckPresence
0130 02:58:08.159412 (+ 19us) tablet.cc:1370] starting ApplyRowOperation cycle
0130 02:58:08.159785 (+ 373us) tablet.cc:1381] finished ApplyRowOperation cycle
0130 02:58:08.159926 (+ 141us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0130 02:58:08.159971 (+ 45us) write_op.cc:312] APPLY: finished
0130 02:58:08.160143 (+ 172us) log.cc:844] Serialized 29 byte log entry
0130 02:58:08.167022 (+ 6879us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:58:08.167091 (+ 69us) write_op.cc:454] Released schema lock
0130 02:58:08.167125 (+ 34us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":472,"apply.run_cpu_time_us":1397,"apply.run_wall_time_us":8092,"num_ops":2,"prepare.queue_time_us":115,"prepare.run_cpu_time_us":3518,"prepare.run_wall_time_us":23630,"replication_time_us":272225,"spinlock_wait_cycles":14373632}]]}
I20260130 02:58:08.173151 18866 catalog_manager.cc:5941] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4: Sending DeleteTablet for 3 replicas of tablet 5ca9919f80954e74a4d91c94b74de0e4
I20260130 02:58:08.174433 18866 catalog_manager.cc:6724] ReplaceTablet: tablet 5ca9919f80954e74a4d91c94b74de0e4 deleted and replaced by tablet 13f0ace4d9de43abab08eca32cdb30c1
I20260130 02:58:08.171547 18867 rpcz_store.cc:276] 0130 02:58:05.668264 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:58:07.649858 (+1981594us) service_pool.cc:225] Handling call
0130 02:58:07.688124 (+ 38266us) catalog_manager.cc:2101] Acquired catalog manager lock
0130 02:58:07.845890 (+157766us) catalog_manager.cc:2147] Created new table and tablet info
0130 02:58:08.170657 (+324767us) catalog_manager.cc:2207] Wrote table and tablets to system table
0130 02:58:08.170931 (+ 274us) catalog_manager.cc:2241] Inserted table and tablets into CatalogManager maps
0130 02:58:08.171182 (+ 251us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0130 02:58:07.908756 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0130 02:58:07.909264 (+ 508us) write_op.cc:432] Acquiring schema lock in shared mode
0130 02:58:07.909292 (+ 28us) write_op.cc:435] Acquired schema lock
0130 02:58:07.909308 (+ 16us) tablet.cc:660] Decoding operations
0130 02:58:07.909632 (+ 324us) write_op.cc:620] Acquiring the partition lock for write op
0130 02:58:07.909670 (+ 38us) write_op.cc:641] Partition lock acquired for write op
0130 02:58:07.909697 (+ 27us) tablet.cc:683] Acquiring locks for 2 operations
0130 02:58:07.909943 (+ 246us) tablet.cc:699] Row locks acquired
0130 02:58:07.909960 (+ 17us) write_op.cc:260] PREPARE: finished
0130 02:58:07.910086 (+ 126us) write_op.cc:270] Start()
0130 02:58:07.910184 (+ 98us) write_op.cc:276] Timestamp: P: 1769741887910005 usec, L: 0
0130 02:58:07.910209 (+ 25us) op_driver.cc:348] REPLICATION: starting
0130 02:58:07.910820 (+ 611us) log.cc:844] Serialized 433 byte log entry
0130 02:58:08.167626 (+256806us) op_driver.cc:464] REPLICATION: finished
0130 02:58:08.168717 (+ 1091us) write_op.cc:301] APPLY: starting
0130 02:58:08.168772 (+ 55us) tablet.cc:1365] starting BulkCheckPresence
0130 02:58:08.168884 (+ 112us) tablet.cc:1368] finished BulkCheckPresence
0130 02:58:08.168986 (+ 102us) tablet.cc:1370] starting ApplyRowOperation cycle
0130 02:58:08.169368 (+ 382us) tablet.cc:1381] finished ApplyRowOperation cycle
0130 02:58:08.169499 (+ 131us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0130 02:58:08.169540 (+ 41us) write_op.cc:312] APPLY: finished
0130 02:58:08.169716 (+ 176us) log.cc:844] Serialized 29 byte log entry
0130 02:58:08.170334 (+ 618us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:58:08.170398 (+ 64us) write_op.cc:454] Released schema lock
0130 02:58:08.170429 (+ 31us) write_op.cc:341] FINISH: Updating metrics
0130 02:58:08.170672 (+ 243us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:58:08.170697 (+ 25us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":958,"apply.run_cpu_time_us":1457,"apply.run_wall_time_us":1921,"num_ops":2,"prepare.queue_time_us":20621,"prepare.run_cpu_time_us":2360,"prepare.run_wall_time_us":6434,"replication_time_us":257342}]]}
I20260130 02:58:08.174753 18866 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:50284 (request call id 197) took 1114 ms. Trace:
I20260130 02:58:08.168545 19159 leader_election.cc:304] T 54be22352e404ff9b798a075693f7f21 P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 371a36d219164974abdf848841528152, 78c2c46c7520492db35d51c244fb4f74; no voters: d870164263bb44b6b780a557daea3656
I20260130 02:58:08.175602 20244 raft_consensus.cc:2804] T 54be22352e404ff9b798a075693f7f21 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260130 02:58:08.175972 19355 tablet_service.cc:1552] Processing DeleteTablet for tablet 5ca9919f80954e74a4d91c94b74de0e4 with delete_type TABLET_DATA_DELETED (replaced by tablet 13f0ace4d9de43abab08eca32cdb30c1) from {username='slave'} at 127.0.0.1:60110
I20260130 02:58:08.175998 20244 raft_consensus.cc:493] T 54be22352e404ff9b798a075693f7f21 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260130 02:58:08.176349 20244 raft_consensus.cc:3060] T 54be22352e404ff9b798a075693f7f21 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:08.175050 18866 rpcz_store.cc:276] 0130 02:58:07.060569 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:58:07.845481 (+784912us) service_pool.cc:225] Handling call
0130 02:58:08.174704 (+329223us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0130 02:58:07.915345 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0130 02:58:07.915836 (+ 491us) write_op.cc:432] Acquiring schema lock in shared mode
0130 02:58:07.915873 (+ 37us) write_op.cc:435] Acquired schema lock
0130 02:58:07.915891 (+ 18us) tablet.cc:660] Decoding operations
0130 02:58:07.916250 (+ 359us) write_op.cc:620] Acquiring the partition lock for write op
0130 02:58:07.916301 (+ 51us) write_op.cc:641] Partition lock acquired for write op
0130 02:58:07.916328 (+ 27us) tablet.cc:683] Acquiring locks for 2 operations
0130 02:58:07.916558 (+ 230us) tablet.cc:699] Row locks acquired
0130 02:58:07.916572 (+ 14us) write_op.cc:260] PREPARE: finished
0130 02:58:07.916692 (+ 120us) write_op.cc:270] Start()
0130 02:58:07.916793 (+ 101us) write_op.cc:276] Timestamp: P: 1769741887916616 usec, L: 0
0130 02:58:07.916813 (+ 20us) op_driver.cc:348] REPLICATION: starting
0130 02:58:07.917403 (+ 590us) log.cc:844] Serialized 662 byte log entry
0130 02:58:08.167821 (+250418us) op_driver.cc:464] REPLICATION: finished
0130 02:58:08.170923 (+ 3102us) write_op.cc:301] APPLY: starting
0130 02:58:08.170970 (+ 47us) tablet.cc:1365] starting BulkCheckPresence
0130 02:58:08.171057 (+ 87us) tablet.cc:1368] finished BulkCheckPresence
0130 02:58:08.171147 (+ 90us) tablet.cc:1370] starting ApplyRowOperation cycle
0130 02:58:08.171450 (+ 303us) tablet.cc:1381] finished ApplyRowOperation cycle
0130 02:58:08.171758 (+ 308us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0130 02:58:08.171804 (+ 46us) write_op.cc:312] APPLY: finished
0130 02:58:08.171988 (+ 184us) log.cc:844] Serialized 29 byte log entry
0130 02:58:08.172559 (+ 571us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:58:08.172617 (+ 58us) write_op.cc:454] Released schema lock
0130 02:58:08.172645 (+ 28us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":2976,"num_ops":2,"prepare.queue_time_us":15362,"prepare.run_cpu_time_us":2296,"prepare.run_wall_time_us":3641,"replication_time_us":250936}]]}
I20260130 02:58:08.171130 20186 raft_consensus.cc:493] T 3b4908bd1a634ca0907234dd3c39e1c3 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:08.177522 20186 raft_consensus.cc:515] T 3b4908bd1a634ca0907234dd3c39e1c3 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:08.179419 20186 leader_election.cc:290] T 3b4908bd1a634ca0907234dd3c39e1c3 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767)
I20260130 02:58:08.179672 20178 raft_consensus.cc:493] T f54addb1a4304ce69c711ae0cc3b452c P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:08.179979 19201 tablet_service.cc:1552] Processing DeleteTablet for tablet 5ca9919f80954e74a4d91c94b74de0e4 with delete_type TABLET_DATA_DELETED (replaced by tablet 13f0ace4d9de43abab08eca32cdb30c1) from {username='slave'} at 127.0.0.1:42870
I20260130 02:58:08.180086 20178 raft_consensus.cc:515] T f54addb1a4304ce69c711ae0cc3b452c P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:08.174998 19501 tablet_service.cc:1552] Processing DeleteTablet for tablet 5ca9919f80954e74a4d91c94b74de0e4 with delete_type TABLET_DATA_DELETED (replaced by tablet 13f0ace4d9de43abab08eca32cdb30c1) from {username='slave'} at 127.0.0.1:59068
I20260130 02:58:08.181056 19947 tablet_replica.cc:333] T 5ca9919f80954e74a4d91c94b74de0e4 P d870164263bb44b6b780a557daea3656: stopping tablet replica
I20260130 02:58:08.181742 19947 raft_consensus.cc:2243] T 5ca9919f80954e74a4d91c94b74de0e4 P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260130 02:58:08.181763 20178 leader_election.cc:290] T f54addb1a4304ce69c711ae0cc3b452c P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767), d870164263bb44b6b780a557daea3656 (127.18.100.67:39571)
I20260130 02:58:08.182191 19947 raft_consensus.cc:2272] T 5ca9919f80954e74a4d91c94b74de0e4 P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260130 02:58:08.183960 19947 ts_tablet_manager.cc:1916] T 5ca9919f80954e74a4d91c94b74de0e4 P d870164263bb44b6b780a557daea3656: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260130 02:58:08.188282 20299 raft_consensus.cc:493] T f54addb1a4304ce69c711ae0cc3b452c P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:08.188735 20299 raft_consensus.cc:515] T f54addb1a4304ce69c711ae0cc3b452c P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:08.192000 20299 leader_election.cc:290] T f54addb1a4304ce69c711ae0cc3b452c P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), d870164263bb44b6b780a557daea3656 (127.18.100.67:39571)
I20260130 02:58:08.192194 19222 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f54addb1a4304ce69c711ae0cc3b452c" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152" is_pre_election: true
I20260130 02:58:08.192744 19222 raft_consensus.cc:2468] T f54addb1a4304ce69c711ae0cc3b452c P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78c2c46c7520492db35d51c244fb4f74 in term 0.
I20260130 02:58:08.193100 19223 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3b4908bd1a634ca0907234dd3c39e1c3" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152" is_pre_election: true
I20260130 02:58:08.193724 19223 raft_consensus.cc:2393] T 3b4908bd1a634ca0907234dd3c39e1c3 P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d870164263bb44b6b780a557daea3656 in current term 1: Already voted for candidate 371a36d219164974abdf848841528152 in this term.
I20260130 02:58:08.194078 19526 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f54addb1a4304ce69c711ae0cc3b452c" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
I20260130 02:58:08.194437 19377 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3b4908bd1a634ca0907234dd3c39e1c3" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74" is_pre_election: true
I20260130 02:58:08.195065 19377 raft_consensus.cc:2393] T 3b4908bd1a634ca0907234dd3c39e1c3 P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d870164263bb44b6b780a557daea3656 in current term 1: Already voted for candidate 78c2c46c7520492db35d51c244fb4f74 in this term.
I20260130 02:58:08.196030 19314 leader_election.cc:304] T f54addb1a4304ce69c711ae0cc3b452c P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 371a36d219164974abdf848841528152, 78c2c46c7520492db35d51c244fb4f74; no voters:
I20260130 02:58:08.197808 19807 tablet_bootstrap.cc:654] T 8bd3e1c06b89426da57e2b259e7983af P 371a36d219164974abdf848841528152: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:08.199579 19840 tablet_bootstrap.cc:492] T b89ed298e3dc49d789d166c613ad7fb8 P 78c2c46c7520492db35d51c244fb4f74: No bootstrap required, opened a new log
I20260130 02:58:08.207340 19840 ts_tablet_manager.cc:1403] T b89ed298e3dc49d789d166c613ad7fb8 P 78c2c46c7520492db35d51c244fb4f74: Time spent bootstrapping tablet: real 0.139s user 0.009s sys 0.003s
I20260130 02:58:08.207275 18875 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50284:
table { table_name: "default.Table_0ad5d205dfea4344a759522f45a55475" } modify_external_catalogs: true
I20260130 02:58:08.207778 18875 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50284:
table { table_name: "default.Table_0ad5d205dfea4344a759522f45a55475" } modify_external_catalogs: true
I20260130 02:58:08.209802 19840 raft_consensus.cc:359] T b89ed298e3dc49d789d166c613ad7fb8 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:08.211225 18873 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50284:
table { table_name: "default.Table_1d9d1f3b2b6048f8949fdba26f5fd3d7" } modify_external_catalogs: true
I20260130 02:58:08.211654 18873 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50284:
table { table_name: "default.Table_1d9d1f3b2b6048f8949fdba26f5fd3d7" } modify_external_catalogs: true
I20260130 02:58:08.215260 18871 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50284:
table { table_name: "default.Table_5c724fbcf445490089992ba7fe40dc3d" } new_table_name: "default.Table_e4d07634d84a4678b1c94374e3adc476" modify_external_catalogs: true
I20260130 02:58:08.217674 19462 leader_election.cc:304] T 3b4908bd1a634ca0907234dd3c39e1c3 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d870164263bb44b6b780a557daea3656; no voters: 371a36d219164974abdf848841528152, 78c2c46c7520492db35d51c244fb4f74
I20260130 02:58:08.218954 20186 raft_consensus.cc:3060] T 3b4908bd1a634ca0907234dd3c39e1c3 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:08.218798 20244 raft_consensus.cc:515] T 54be22352e404ff9b798a075693f7f21 P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:08.222929 20299 raft_consensus.cc:2804] T f54addb1a4304ce69c711ae0cc3b452c P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260130 02:58:08.223333 20299 raft_consensus.cc:493] T f54addb1a4304ce69c711ae0cc3b452c P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260130 02:58:08.223696 20299 raft_consensus.cc:3060] T f54addb1a4304ce69c711ae0cc3b452c P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:08.224061 20244 leader_election.cc:290] T 54be22352e404ff9b798a075693f7f21 P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 election: Requested vote from peers 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767), d870164263bb44b6b780a557daea3656 (127.18.100.67:39571)
W20260130 02:58:08.225610 18860 catalog_manager.cc:4952] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767): delete failed for tablet df2c8989d6c1480c97baa23db01141eb because the tablet was not found. No further retry: Not found: Tablet not found: df2c8989d6c1480c97baa23db01141eb
W20260130 02:58:08.227432 18860 catalog_manager.cc:4952] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767): delete failed for tablet 534dc31a9b7e492c82f3672c75742e6c because the tablet was not found. No further retry: Not found: Tablet not found: 534dc31a9b7e492c82f3672c75742e6c
W20260130 02:58:08.228802 18860 catalog_manager.cc:4952] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767): delete failed for tablet c3fce8cc99fb451ebd532fa37390e862 because the tablet was not found. No further retry: Not found: Tablet not found: c3fce8cc99fb451ebd532fa37390e862
W20260130 02:58:08.230082 18860 catalog_manager.cc:4952] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767): delete failed for tablet a743a82540014c85aeab0e9d0994b0dc because the tablet was not found. No further retry: Not found: Tablet not found: a743a82540014c85aeab0e9d0994b0dc
I20260130 02:58:08.210407 19840 raft_consensus.cc:385] T b89ed298e3dc49d789d166c613ad7fb8 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:08.231192 19840 raft_consensus.cc:740] T b89ed298e3dc49d789d166c613ad7fb8 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78c2c46c7520492db35d51c244fb4f74, State: Initialized, Role: FOLLOWER
W20260130 02:58:08.231565 19313 leader_election.cc:343] T f54addb1a4304ce69c711ae0cc3b452c P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:08.231950 19840 consensus_queue.cc:260] T b89ed298e3dc49d789d166c613ad7fb8 P 78c2c46c7520492db35d51c244fb4f74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:08.239409 19840 ts_tablet_manager.cc:1434] T b89ed298e3dc49d789d166c613ad7fb8 P 78c2c46c7520492db35d51c244fb4f74: Time spent starting tablet: real 0.032s user 0.005s sys 0.000s
I20260130 02:58:08.240759 20208 tablet_replica.cc:333] T 730bd1d7d15b4cbe945b3ef03b2455f7 P 78c2c46c7520492db35d51c244fb4f74: stopping tablet replica
I20260130 02:58:08.241641 20208 raft_consensus.cc:2243] T 730bd1d7d15b4cbe945b3ef03b2455f7 P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260130 02:58:08.242219 20208 raft_consensus.cc:2272] T 730bd1d7d15b4cbe945b3ef03b2455f7 P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260130 02:58:08.243095 18860 catalog_manager.cc:4952] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767): delete failed for tablet b3f3177306914e3c8b253db54ee30fb6 because the tablet was not found. No further retry: Not found: Tablet not found: b3f3177306914e3c8b253db54ee30fb6
I20260130 02:58:08.249634 18872 catalog_manager.cc:5746] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571) does not have the latest schema for tablet c7c621b99deb457998c44ab1644d3939 (table default.Table_3413dae2efac41488448e6e5975cf710 [id=7a08fa3986d0410d81de0235d2fb0f92]). Expected version 1 got 0
I20260130 02:58:08.266036 19840 tablet_bootstrap.cc:492] T ee7c8ff92ca9465fa18be6eae63d78ca P 78c2c46c7520492db35d51c244fb4f74: Bootstrap starting.
I20260130 02:58:08.267674 18872 catalog_manager.cc:5654] T 91f954f3ef4244c9a575a7421a7e75eb P d870164263bb44b6b780a557daea3656 reported cstate change: term changed from 0 to 1, leader changed from <none> to d870164263bb44b6b780a557daea3656 (127.18.100.67). New cstate: current_term: 1 leader_uuid: "d870164263bb44b6b780a557daea3656" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } health_report { overall_health: UNKNOWN } } }
I20260130 02:58:08.270133 18872 catalog_manager.cc:5746] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571) does not have the latest schema for tablet 91f954f3ef4244c9a575a7421a7e75eb (table default.Table_9f83f87711d94e8394abe1925ac84ad1 [id=db23c1be42c44b6f9eb774f14d2b8fd6]). Expected version 1 got 0
I20260130 02:58:08.271399 18873 catalog_manager.cc:5941] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4: Sending DeleteTablet for 3 replicas of tablet 8bd3e1c06b89426da57e2b259e7983af
I20260130 02:58:08.272680 20208 ts_tablet_manager.cc:1916] T 730bd1d7d15b4cbe945b3ef03b2455f7 P 78c2c46c7520492db35d51c244fb4f74: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260130 02:58:08.272787 18873 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:50284 (request call id 198) took 1010 ms. Trace:
I20260130 02:58:08.276443 19355 tablet_service.cc:1552] Processing DeleteTablet for tablet 8bd3e1c06b89426da57e2b259e7983af with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:58:08 UTC) from {username='slave'} at 127.0.0.1:60110
I20260130 02:58:08.277173 19501 tablet_service.cc:1552] Processing DeleteTablet for tablet 8bd3e1c06b89426da57e2b259e7983af with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:58:08 UTC) from {username='slave'} at 127.0.0.1:59068
I20260130 02:58:08.277824 19201 tablet_service.cc:1552] Processing DeleteTablet for tablet 8bd3e1c06b89426da57e2b259e7983af with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:58:08 UTC) from {username='slave'} at 127.0.0.1:42870
I20260130 02:58:08.279335 19526 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f54addb1a4304ce69c711ae0cc3b452c" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
I20260130 02:58:08.279335 19376 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "54be22352e404ff9b798a075693f7f21" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74"
I20260130 02:58:08.279868 19376 raft_consensus.cc:3060] T 54be22352e404ff9b798a075693f7f21 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:08.278515 19377 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f54addb1a4304ce69c711ae0cc3b452c" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74" is_pre_election: true
I20260130 02:58:08.280897 19840 tablet_bootstrap.cc:654] T ee7c8ff92ca9465fa18be6eae63d78ca P 78c2c46c7520492db35d51c244fb4f74: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:08.282145 18866 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50284:
name: "default.Table_7c48a0d054354501947d11c5fd7b1864"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260130 02:58:08.283721 18866 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7c48a0d054354501947d11c5fd7b1864 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260130 02:58:08.286046 18867 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50284:
table { table_name: "default.Table_1cb6165b7dda4b659c1c379902ee4930" } new_table_name: "default.Table_d0b4516d24384e5386c1debaf2543adb" modify_external_catalogs: true
I20260130 02:58:08.286266 19807 tablet_bootstrap.cc:492] T 8bd3e1c06b89426da57e2b259e7983af P 371a36d219164974abdf848841528152: No bootstrap required, opened a new log
I20260130 02:58:08.286674 19807 ts_tablet_manager.cc:1403] T 8bd3e1c06b89426da57e2b259e7983af P 371a36d219164974abdf848841528152: Time spent bootstrapping tablet: real 0.148s user 0.010s sys 0.000s
I20260130 02:58:08.287851 19376 raft_consensus.cc:2468] T 54be22352e404ff9b798a075693f7f21 P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 371a36d219164974abdf848841528152 in term 1.
I20260130 02:58:08.288618 18875 catalog_manager.cc:5941] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4: Sending DeleteTablet for 3 replicas of tablet 3b4908bd1a634ca0907234dd3c39e1c3
I20260130 02:58:08.288782 19807 raft_consensus.cc:359] T 8bd3e1c06b89426da57e2b259e7983af P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:08.289376 19807 raft_consensus.cc:385] T 8bd3e1c06b89426da57e2b259e7983af P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:08.289606 19807 raft_consensus.cc:740] T 8bd3e1c06b89426da57e2b259e7983af P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 371a36d219164974abdf848841528152, State: Initialized, Role: FOLLOWER
I20260130 02:58:08.289526 20180 raft_consensus.cc:493] T b89ed298e3dc49d789d166c613ad7fb8 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:08.289964 20180 raft_consensus.cc:515] T b89ed298e3dc49d789d166c613ad7fb8 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:08.290552 19807 consensus_queue.cc:260] T 8bd3e1c06b89426da57e2b259e7983af P 371a36d219164974abdf848841528152 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:08.292188 19355 tablet_service.cc:1552] Processing DeleteTablet for tablet 3b4908bd1a634ca0907234dd3c39e1c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:58:08 UTC) from {username='slave'} at 127.0.0.1:60110
I20260130 02:58:08.292650 19807 ts_tablet_manager.cc:1434] T 8bd3e1c06b89426da57e2b259e7983af P 371a36d219164974abdf848841528152: Time spent starting tablet: real 0.006s user 0.001s sys 0.003s
I20260130 02:58:08.292577 20180 leader_election.cc:290] T b89ed298e3dc49d789d166c613ad7fb8 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), d870164263bb44b6b780a557daea3656 (127.18.100.67:39571)
I20260130 02:58:08.293430 19807 tablet_bootstrap.cc:492] T b2ac91da01954c2390b6b7dcc06c394f P 371a36d219164974abdf848841528152: Bootstrap starting.
I20260130 02:58:08.294636 19201 tablet_service.cc:1552] Processing DeleteTablet for tablet 3b4908bd1a634ca0907234dd3c39e1c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:58:08 UTC) from {username='slave'} at 127.0.0.1:42870
I20260130 02:58:08.295409 19223 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b89ed298e3dc49d789d166c613ad7fb8" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152" is_pre_election: true
W20260130 02:58:08.296499 19314 leader_election.cc:343] T b89ed298e3dc49d789d166c613ad7fb8 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:08.299304 20180 raft_consensus.cc:493] T b2ac91da01954c2390b6b7dcc06c394f P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:08.299747 20180 raft_consensus.cc:515] T b2ac91da01954c2390b6b7dcc06c394f P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:08.300935 19807 tablet_bootstrap.cc:654] T b2ac91da01954c2390b6b7dcc06c394f P 371a36d219164974abdf848841528152: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:08.302489 20180 leader_election.cc:290] T b2ac91da01954c2390b6b7dcc06c394f P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), d870164263bb44b6b780a557daea3656 (127.18.100.67:39571)
I20260130 02:58:08.303833 19223 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b2ac91da01954c2390b6b7dcc06c394f" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152" is_pre_election: true
W20260130 02:58:08.304980 19314 leader_election.cc:343] T b2ac91da01954c2390b6b7dcc06c394f P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:08.273075 18873 rpcz_store.cc:276] 0130 02:58:07.262747 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:58:08.071388 (+808641us) service_pool.cc:225] Handling call
0130 02:58:08.211142 (+139754us) spinlock_profiling.cc:243] Waited 139 ms on lock 0x7b5400030554. stack: 00007fb0bd3f7373 00007fb0bd3f7270 00007fb0bcf6c574 0000000000508744 00000000005086d9 00007fb0c44490c0 00007fb0c44e7bba 00007fb0c61e6eac 00007fb0c62e2966 00007fb0c3f0a024 00007fb0c3f09fb2 00007fb0c3f09ee1 00007fb0c3f09e5c 00007fb0c3f09112 00007fb0c21dd34c 00007fb0c21dc786
0130 02:58:08.212001 (+ 859us) catalog_manager.cc:2393] Looking up, locking, and authorizing table
0130 02:58:08.212458 (+ 457us) catalog_manager.cc:2815] Modifying in-memory table state
0130 02:58:08.212560 (+ 102us) catalog_manager.cc:2824] Locking tablets
0130 02:58:08.213088 (+ 528us) catalog_manager.cc:2838] Removing table and tablets from system table
0130 02:58:08.243248 (+ 30160us) catalog_manager.cc:2856] Removing table from by-name map
0130 02:58:08.271243 (+ 27995us) catalog_manager.cc:2872] Committing in-memory state
0130 02:58:08.271326 (+ 83us) catalog_manager.cc:2876] Aborting table tasks
0130 02:58:08.272731 (+ 1405us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0130 02:58:08.227023 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0130 02:58:08.227479 (+ 456us) write_op.cc:432] Acquiring schema lock in shared mode
0130 02:58:08.227509 (+ 30us) write_op.cc:435] Acquired schema lock
0130 02:58:08.227525 (+ 16us) tablet.cc:660] Decoding operations
0130 02:58:08.227885 (+ 360us) write_op.cc:620] Acquiring the partition lock for write op
0130 02:58:08.227933 (+ 48us) write_op.cc:641] Partition lock acquired for write op
0130 02:58:08.227956 (+ 23us) tablet.cc:683] Acquiring locks for 2 operations
0130 02:58:08.228275 (+ 319us) tablet.cc:699] Row locks acquired
0130 02:58:08.228288 (+ 13us) write_op.cc:260] PREPARE: finished
0130 02:58:08.228409 (+ 121us) write_op.cc:270] Start()
0130 02:58:08.228508 (+ 99us) write_op.cc:276] Timestamp: P: 1769741888228357 usec, L: 0
0130 02:58:08.228523 (+ 15us) op_driver.cc:348] REPLICATION: starting
0130 02:58:08.229093 (+ 570us) log.cc:844] Serialized 731 byte log entry
0130 02:58:08.238815 (+ 9722us) op_driver.cc:464] REPLICATION: finished
0130 02:58:08.239208 (+ 393us) write_op.cc:301] APPLY: starting
0130 02:58:08.239260 (+ 52us) tablet.cc:1365] starting BulkCheckPresence
0130 02:58:08.239356 (+ 96us) tablet.cc:1368] finished BulkCheckPresence
0130 02:58:08.239471 (+ 115us) tablet.cc:1370] starting ApplyRowOperation cycle
0130 02:58:08.239704 (+ 233us) tablet.cc:1381] finished ApplyRowOperation cycle
0130 02:58:08.239836 (+ 132us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0130 02:58:08.239885 (+ 49us) write_op.cc:312] APPLY: finished
0130 02:58:08.240084 (+ 199us) log.cc:844] Serialized 29 byte log entry
0130 02:58:08.240215 (+ 131us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:58:08.240279 (+ 64us) write_op.cc:454] Released schema lock
0130 02:58:08.240307 (+ 28us) write_op.cc:341] FINISH: Updating metrics
0130 02:58:08.240510 (+ 203us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:58:08.240539 (+ 29us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":319792000,"child_traces":[["op",{"apply.queue_time_us":229,"apply.run_cpu_time_us":1295,"apply.run_wall_time_us":1294,"num_ops":2,"prepare.queue_time_us":12367,"prepare.run_cpu_time_us":2385,"prepare.run_wall_time_us":2386,"raft.queue_time_us":1280,"raft.run_cpu_time_us":539,"raft.run_wall_time_us":539,"replication_time_us":10211}]]}
I20260130 02:58:08.307904 19525 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b2ac91da01954c2390b6b7dcc06c394f" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
W20260130 02:58:08.310842 19313 leader_election.cc:343] T b2ac91da01954c2390b6b7dcc06c394f P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:08.311303 19313 leader_election.cc:304] T b2ac91da01954c2390b6b7dcc06c394f P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78c2c46c7520492db35d51c244fb4f74; no voters: 371a36d219164974abdf848841528152, d870164263bb44b6b780a557daea3656
W20260130 02:58:08.313925 18871 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4c139aa8cd184f488cec4e05ef7dcada consensus state has no leader
I20260130 02:58:08.312536 20180 raft_consensus.cc:2749] T b2ac91da01954c2390b6b7dcc06c394f P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260130 02:58:08.316756 19501 tablet_service.cc:1552] Processing DeleteTablet for tablet 3b4908bd1a634ca0907234dd3c39e1c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:58:08 UTC) from {username='slave'} at 127.0.0.1:59068
I20260130 02:58:08.317462 19526 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b89ed298e3dc49d789d166c613ad7fb8" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
W20260130 02:58:08.320936 19313 leader_election.cc:343] T b89ed298e3dc49d789d166c613ad7fb8 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:08.321305 19313 leader_election.cc:304] T b89ed298e3dc49d789d166c613ad7fb8 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78c2c46c7520492db35d51c244fb4f74; no voters: 371a36d219164974abdf848841528152, d870164263bb44b6b780a557daea3656
I20260130 02:58:08.321907 20180 raft_consensus.cc:2749] T b89ed298e3dc49d789d166c613ad7fb8 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260130 02:58:08.340525 18871 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50284:
name: "default.Table_e03bdbb221e94bd6b535ddc02c863bad"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260130 02:58:08.345000 19807 tablet_bootstrap.cc:492] T b2ac91da01954c2390b6b7dcc06c394f P 371a36d219164974abdf848841528152: No bootstrap required, opened a new log
I20260130 02:58:08.345427 19807 ts_tablet_manager.cc:1403] T b2ac91da01954c2390b6b7dcc06c394f P 371a36d219164974abdf848841528152: Time spent bootstrapping tablet: real 0.052s user 0.008s sys 0.012s
W20260130 02:58:08.346879 18871 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e03bdbb221e94bd6b535ddc02c863bad in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260130 02:58:08.348209 19807 raft_consensus.cc:359] T b2ac91da01954c2390b6b7dcc06c394f P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:08.348827 19807 raft_consensus.cc:385] T b2ac91da01954c2390b6b7dcc06c394f P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:08.349092 19807 raft_consensus.cc:740] T b2ac91da01954c2390b6b7dcc06c394f P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 371a36d219164974abdf848841528152, State: Initialized, Role: FOLLOWER
I20260130 02:58:08.349610 19807 consensus_queue.cc:260] T b2ac91da01954c2390b6b7dcc06c394f P 371a36d219164974abdf848841528152 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:08.352568 19807 ts_tablet_manager.cc:1434] T b2ac91da01954c2390b6b7dcc06c394f P 371a36d219164974abdf848841528152: Time spent starting tablet: real 0.007s user 0.002s sys 0.005s
I20260130 02:58:08.353332 19807 tablet_bootstrap.cc:492] T ee7c8ff92ca9465fa18be6eae63d78ca P 371a36d219164974abdf848841528152: Bootstrap starting.
I20260130 02:58:08.350878 20256 ts_tablet_manager.cc:1929] T 481bd4cca6ee4c178873bf12392e3441 P 371a36d219164974abdf848841528152: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260130 02:58:08.354804 20256 log.cc:1199] T 481bd4cca6ee4c178873bf12392e3441 P 371a36d219164974abdf848841528152: Deleting WAL directory at /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-0/wal/wals/481bd4cca6ee4c178873bf12392e3441
I20260130 02:58:08.355654 20256 ts_tablet_manager.cc:1950] T 481bd4cca6ee4c178873bf12392e3441 P 371a36d219164974abdf848841528152: Deleting consensus metadata
I20260130 02:58:08.358099 19840 tablet_bootstrap.cc:492] T ee7c8ff92ca9465fa18be6eae63d78ca P 78c2c46c7520492db35d51c244fb4f74: No bootstrap required, opened a new log
I20260130 02:58:08.359143 19840 ts_tablet_manager.cc:1403] T ee7c8ff92ca9465fa18be6eae63d78ca P 78c2c46c7520492db35d51c244fb4f74: Time spent bootstrapping tablet: real 0.093s user 0.009s sys 0.000s
I20260130 02:58:08.362030 19840 raft_consensus.cc:359] T ee7c8ff92ca9465fa18be6eae63d78ca P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:08.364817 19807 tablet_bootstrap.cc:654] T ee7c8ff92ca9465fa18be6eae63d78ca P 371a36d219164974abdf848841528152: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:08.370627 20244 raft_consensus.cc:493] T b2ac91da01954c2390b6b7dcc06c394f P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:08.370728 19840 raft_consensus.cc:385] T ee7c8ff92ca9465fa18be6eae63d78ca P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:08.370998 19840 raft_consensus.cc:740] T ee7c8ff92ca9465fa18be6eae63d78ca P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78c2c46c7520492db35d51c244fb4f74, State: Initialized, Role: FOLLOWER
I20260130 02:58:08.371205 20244 raft_consensus.cc:515] T b2ac91da01954c2390b6b7dcc06c394f P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:08.373581 20244 leader_election.cc:290] T b2ac91da01954c2390b6b7dcc06c394f P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767), d870164263bb44b6b780a557daea3656 (127.18.100.67:39571)
I20260130 02:58:08.379760 19840 consensus_queue.cc:260] T ee7c8ff92ca9465fa18be6eae63d78ca P 78c2c46c7520492db35d51c244fb4f74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:08.391629 20186 raft_consensus.cc:2749] T 3b4908bd1a634ca0907234dd3c39e1c3 P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260130 02:58:08.394357 19807 tablet_bootstrap.cc:492] T ee7c8ff92ca9465fa18be6eae63d78ca P 371a36d219164974abdf848841528152: No bootstrap required, opened a new log
I20260130 02:58:08.394804 19807 ts_tablet_manager.cc:1403] T ee7c8ff92ca9465fa18be6eae63d78ca P 371a36d219164974abdf848841528152: Time spent bootstrapping tablet: real 0.042s user 0.007s sys 0.005s
W20260130 02:58:08.362470 18867 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b1e8b78eff0d4a0b92059adb2de4a3ce consensus state has no leader
I20260130 02:58:08.396986 19807 raft_consensus.cc:359] T ee7c8ff92ca9465fa18be6eae63d78ca P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:08.363937 18861 catalog_manager.cc:4985] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021): tablet 481bd4cca6ee4c178873bf12392e3441 (table default.Table_e3b63d7d5d874900aac850654d98d62e [id=bc8cc55916fa4a41b78019f59e6fbe39]) successfully deleted
I20260130 02:58:08.397588 19807 raft_consensus.cc:385] T ee7c8ff92ca9465fa18be6eae63d78ca P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:08.397820 19807 raft_consensus.cc:740] T ee7c8ff92ca9465fa18be6eae63d78ca P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 371a36d219164974abdf848841528152, State: Initialized, Role: FOLLOWER
W20260130 02:58:08.397994 18861 catalog_manager.cc:4952] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021): delete failed for tablet 2112b0402de5464cb1caf2c2abb67d78 because the tablet was not found. No further retry: Not found: Tablet not found: 2112b0402de5464cb1caf2c2abb67d78
I20260130 02:58:08.398345 19807 consensus_queue.cc:260] T ee7c8ff92ca9465fa18be6eae63d78ca P 371a36d219164974abdf848841528152 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
W20260130 02:58:08.398558 18861 catalog_manager.cc:4952] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021): delete failed for tablet df2c8989d6c1480c97baa23db01141eb because the tablet was not found. No further retry: Not found: Tablet not found: df2c8989d6c1480c97baa23db01141eb
W20260130 02:58:08.399717 18861 catalog_manager.cc:4952] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021): delete failed for tablet 927c90b84d7449efa36bc5e2035bf0da because the tablet was not found. No further retry: Not found: Tablet not found: 927c90b84d7449efa36bc5e2035bf0da
W20260130 02:58:08.400241 18861 catalog_manager.cc:4965] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021): delete failed for tablet 3389db58493448aea3cf00aca411047c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3389db58493448aea3cf00aca411047c already in progress: creating tablet
I20260130 02:58:08.400362 20256 tablet_replica.cc:333] T 5ca9919f80954e74a4d91c94b74de0e4 P 371a36d219164974abdf848841528152: stopping tablet replica
I20260130 02:58:08.401228 20256 raft_consensus.cc:2243] T 5ca9919f80954e74a4d91c94b74de0e4 P 371a36d219164974abdf848841528152 [term 1 LEADER]: Raft consensus shutting down.
I20260130 02:58:08.404291 19807 ts_tablet_manager.cc:1434] T ee7c8ff92ca9465fa18be6eae63d78ca P 371a36d219164974abdf848841528152: Time spent starting tablet: real 0.009s user 0.004s sys 0.000s
I20260130 02:58:08.405143 19807 tablet_bootstrap.cc:492] T 3593f8b89da24ff9b6cf549d5ac58bc6 P 371a36d219164974abdf848841528152: Bootstrap starting.
I20260130 02:58:08.405901 20256 raft_consensus.cc:2272] T 5ca9919f80954e74a4d91c94b74de0e4 P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260130 02:58:08.387328 18873 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50284:
name: "default.Table_9e52337cb913484caafe98b1e08c642e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260130 02:58:08.406417 19840 ts_tablet_manager.cc:1434] T ee7c8ff92ca9465fa18be6eae63d78ca P 78c2c46c7520492db35d51c244fb4f74: Time spent starting tablet: real 0.047s user 0.006s sys 0.000s
I20260130 02:58:08.407326 19840 tablet_bootstrap.cc:492] T 127a84346b9f473d9e7f739b1e65feb8 P 78c2c46c7520492db35d51c244fb4f74: Bootstrap starting.
W20260130 02:58:08.407366 18873 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9e52337cb913484caafe98b1e08c642e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260130 02:58:08.410925 20256 ts_tablet_manager.cc:1916] T 5ca9919f80954e74a4d91c94b74de0e4 P 371a36d219164974abdf848841528152: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260130 02:58:08.387974 18866 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50284:
name: "default.Table_829d1e187e8b4db5b8f4f835e84246e5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260130 02:58:08.421314 18866 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_829d1e187e8b4db5b8f4f835e84246e5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260130 02:58:08.430904 20208 ts_tablet_manager.cc:1929] T 730bd1d7d15b4cbe945b3ef03b2455f7 P 78c2c46c7520492db35d51c244fb4f74: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260130 02:58:08.431384 20208 log.cc:1199] T 730bd1d7d15b4cbe945b3ef03b2455f7 P 78c2c46c7520492db35d51c244fb4f74: Deleting WAL directory at /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-1/wal/wals/730bd1d7d15b4cbe945b3ef03b2455f7
I20260130 02:58:08.432183 20208 ts_tablet_manager.cc:1950] T 730bd1d7d15b4cbe945b3ef03b2455f7 P 78c2c46c7520492db35d51c244fb4f74: Deleting consensus metadata
I20260130 02:58:08.436758 19840 tablet_bootstrap.cc:654] T 127a84346b9f473d9e7f739b1e65feb8 P 78c2c46c7520492db35d51c244fb4f74: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:08.420796 19807 tablet_bootstrap.cc:654] T 3593f8b89da24ff9b6cf549d5ac58bc6 P 371a36d219164974abdf848841528152: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:08.442375 19526 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "54be22352e404ff9b798a075693f7f21" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656"
I20260130 02:58:08.427227 19159 leader_election.cc:304] T 54be22352e404ff9b798a075693f7f21 P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 371a36d219164974abdf848841528152, 78c2c46c7520492db35d51c244fb4f74; no voters:
I20260130 02:58:08.449733 20244 raft_consensus.cc:2804] T 54be22352e404ff9b798a075693f7f21 P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Leader election won for term 1
I20260130 02:58:08.450194 20244 raft_consensus.cc:697] T 54be22352e404ff9b798a075693f7f21 P 371a36d219164974abdf848841528152 [term 1 LEADER]: Becoming Leader. State: Replica: 371a36d219164974abdf848841528152, State: Running, Role: LEADER
I20260130 02:58:08.450947 20244 consensus_queue.cc:237] T 54be22352e404ff9b798a075693f7f21 P 371a36d219164974abdf848841528152 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:08.452999 20313 consensus_queue.cc:1048] T 341041cb0be2456e88403367c0814704 P d870164263bb44b6b780a557daea3656 [LEADER]: Connected to new peer: Peer: permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260130 02:58:08.457975 18869 catalog_manager.cc:5746] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021) does not have the latest schema for tablet c7c621b99deb457998c44ab1644d3939 (table default.Table_3413dae2efac41488448e6e5975cf710 [id=7a08fa3986d0410d81de0235d2fb0f92]). Expected version 1 got 0
I20260130 02:58:08.460199 18869 catalog_manager.cc:5654] T 111287b5557d4abda1c5cd560fb0d019 P 371a36d219164974abdf848841528152 reported cstate change: term changed from 0 to 1, leader changed from <none> to 371a36d219164974abdf848841528152 (127.18.100.65). New cstate: current_term: 1 leader_uuid: "371a36d219164974abdf848841528152" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } health_report { overall_health: UNKNOWN } } }
I20260130 02:58:08.466071 20243 consensus_queue.cc:1048] T 6573198830ad45628e71952a7d9c00a0 P 371a36d219164974abdf848841528152 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260130 02:58:08.468447 18869 catalog_manager.cc:5746] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021) does not have the latest schema for tablet 91f954f3ef4244c9a575a7421a7e75eb (table default.Table_9f83f87711d94e8394abe1925ac84ad1 [id=db23c1be42c44b6f9eb774f14d2b8fd6]). Expected version 1 got 0
I20260130 02:58:08.468447 20243 consensus_queue.cc:1048] T 111287b5557d4abda1c5cd560fb0d019 P 371a36d219164974abdf848841528152 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260130 02:58:08.469916 19159 leader_election.cc:343] T f54addb1a4304ce69c711ae0cc3b452c P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:08.479295 18875 master_service.cc:954] ReplaceTablet: received request to replace tablet 1bbc55bbf2254d8bb6b721d7e8f69d85 from {username='slave'} at 127.0.0.1:50284
I20260130 02:58:08.480641 20299 raft_consensus.cc:515] T f54addb1a4304ce69c711ae0cc3b452c P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:08.481719 19377 raft_consensus.cc:2393] T f54addb1a4304ce69c711ae0cc3b452c P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 371a36d219164974abdf848841528152 in current term 1: Already voted for candidate 78c2c46c7520492db35d51c244fb4f74 in this term.
I20260130 02:58:08.483114 20208 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60110 (request call id 196) took 1036 ms. Trace:
I20260130 02:58:08.483541 20208 rpcz_store.cc:276] 0130 02:58:07.446851 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:58:07.451200 (+ 4349us) service_pool.cc:225] Handling call
0130 02:58:08.240615 (+789415us) ts_tablet_manager.cc:1151] Deleting tablet 730bd1d7d15b4cbe945b3ef03b2455f7
0130 02:58:08.240652 (+ 37us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0130 02:58:08.319704 (+ 79052us) tablet_metadata.cc:641] Metadata flushed
0130 02:58:08.430645 (+110941us) tablet_metadata.cc:641] Metadata flushed
0130 02:58:08.483052 (+ 52407us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":644599}
I20260130 02:58:08.485018 18860 catalog_manager.cc:4985] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767): tablet 730bd1d7d15b4cbe945b3ef03b2455f7 (table default.Table_1e9e20a1bfec4bcf81f9eb6219c33d59 [id=f675b7a3ec744533903d8834eeed51d7]) successfully deleted
I20260130 02:58:08.485085 19521 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f54addb1a4304ce69c711ae0cc3b452c" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656"
W20260130 02:58:08.485314 19159 leader_election.cc:343] T 54be22352e404ff9b798a075693f7f21 P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): Illegal state: must be running to vote when last-logged opid is not known
W20260130 02:58:08.485955 18860 catalog_manager.cc:4965] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767): delete failed for tablet 3389db58493448aea3cf00aca411047c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3389db58493448aea3cf00aca411047c already in progress: creating tablet
I20260130 02:58:08.484076 20299 leader_election.cc:290] T f54addb1a4304ce69c711ae0cc3b452c P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 election: Requested vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), d870164263bb44b6b780a557daea3656 (127.18.100.67:39571)
I20260130 02:58:08.486194 20208 tablet_replica.cc:333] T 5ca9919f80954e74a4d91c94b74de0e4 P 78c2c46c7520492db35d51c244fb4f74: stopping tablet replica
W20260130 02:58:08.486630 19313 leader_election.cc:343] T f54addb1a4304ce69c711ae0cc3b452c P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:08.487332 20208 raft_consensus.cc:2243] T 5ca9919f80954e74a4d91c94b74de0e4 P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260130 02:58:08.488056 20208 raft_consensus.cc:2272] T 5ca9919f80954e74a4d91c94b74de0e4 P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260130 02:58:08.488832 20312 raft_consensus.cc:493] T 3cf5154d8cc44f2bb12978202699c193 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:08.489537 19807 tablet_bootstrap.cc:492] T 3593f8b89da24ff9b6cf549d5ac58bc6 P 371a36d219164974abdf848841528152: No bootstrap required, opened a new log
I20260130 02:58:08.489929 19807 ts_tablet_manager.cc:1403] T 3593f8b89da24ff9b6cf549d5ac58bc6 P 371a36d219164974abdf848841528152: Time spent bootstrapping tablet: real 0.085s user 0.017s sys 0.003s
I20260130 02:58:08.489905 20208 ts_tablet_manager.cc:1916] T 5ca9919f80954e74a4d91c94b74de0e4 P 78c2c46c7520492db35d51c244fb4f74: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260130 02:58:08.489893 20312 raft_consensus.cc:515] T 3cf5154d8cc44f2bb12978202699c193 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:08.492122 20243 consensus_queue.cc:1048] T 111287b5557d4abda1c5cd560fb0d019 P 371a36d219164974abdf848841528152 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260130 02:58:08.168280 19625 tablet_bootstrap.cc:492] T 3b148691090e4ee581d46f9845a6e49a P d870164263bb44b6b780a557daea3656: Bootstrap starting.
I20260130 02:58:08.493924 18870 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50284:
table { table_name: "default.Table_e84c6277dfc944059f62cbe0d6b937ee" } modify_external_catalogs: true
I20260130 02:58:08.494405 18870 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50284:
table { table_name: "default.Table_e84c6277dfc944059f62cbe0d6b937ee" } modify_external_catalogs: true
I20260130 02:58:08.497020 19377 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b2ac91da01954c2390b6b7dcc06c394f" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74" is_pre_election: true
I20260130 02:58:08.497530 19377 raft_consensus.cc:2468] T b2ac91da01954c2390b6b7dcc06c394f P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 371a36d219164974abdf848841528152 in term 0.
I20260130 02:58:08.501338 20194 raft_consensus.cc:493] T ee7c8ff92ca9465fa18be6eae63d78ca P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:08.501778 20194 raft_consensus.cc:515] T ee7c8ff92ca9465fa18be6eae63d78ca P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
W20260130 02:58:08.503604 19462 leader_election.cc:343] T 3cf5154d8cc44f2bb12978202699c193 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767): Not found: Tablet not found: 3cf5154d8cc44f2bb12978202699c193
I20260130 02:58:08.503690 20194 leader_election.cc:290] T ee7c8ff92ca9465fa18be6eae63d78ca P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), d870164263bb44b6b780a557daea3656 (127.18.100.67:39571)
I20260130 02:58:08.505215 19840 tablet_bootstrap.cc:492] T 127a84346b9f473d9e7f739b1e65feb8 P 78c2c46c7520492db35d51c244fb4f74: No bootstrap required, opened a new log
I20260130 02:58:08.505867 19840 ts_tablet_manager.cc:1403] T 127a84346b9f473d9e7f739b1e65feb8 P 78c2c46c7520492db35d51c244fb4f74: Time spent bootstrapping tablet: real 0.099s user 0.009s sys 0.001s
I20260130 02:58:08.508277 19840 raft_consensus.cc:359] T 127a84346b9f473d9e7f739b1e65feb8 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:08.508939 19840 raft_consensus.cc:385] T 127a84346b9f473d9e7f739b1e65feb8 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:08.509210 19840 raft_consensus.cc:740] T 127a84346b9f473d9e7f739b1e65feb8 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78c2c46c7520492db35d51c244fb4f74, State: Initialized, Role: FOLLOWER
I20260130 02:58:08.510964 19521 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ee7c8ff92ca9465fa18be6eae63d78ca" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
I20260130 02:58:08.514714 19521 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b2ac91da01954c2390b6b7dcc06c394f" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
I20260130 02:58:08.516526 19807 raft_consensus.cc:359] T 3593f8b89da24ff9b6cf549d5ac58bc6 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:08.517144 19807 raft_consensus.cc:385] T 3593f8b89da24ff9b6cf549d5ac58bc6 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:08.517390 19807 raft_consensus.cc:740] T 3593f8b89da24ff9b6cf549d5ac58bc6 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 371a36d219164974abdf848841528152, State: Initialized, Role: FOLLOWER
W20260130 02:58:08.517338 19313 leader_election.cc:343] T ee7c8ff92ca9465fa18be6eae63d78ca P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:08.517904 19807 consensus_queue.cc:260] T 3593f8b89da24ff9b6cf549d5ac58bc6 P 371a36d219164974abdf848841528152 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:08.518971 19840 consensus_queue.cc:260] T 127a84346b9f473d9e7f739b1e65feb8 P 78c2c46c7520492db35d51c244fb4f74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:08.521915 19222 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f54addb1a4304ce69c711ae0cc3b452c" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152"
I20260130 02:58:08.522439 19222 raft_consensus.cc:3060] T f54addb1a4304ce69c711ae0cc3b452c P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:08.523417 20317 raft_consensus.cc:493] T b8c579ff5fa84651ba9125457d927162 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:08.523864 20317 raft_consensus.cc:515] T b8c579ff5fa84651ba9125457d927162 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:08.525699 20317 leader_election.cc:290] T b8c579ff5fa84651ba9125457d927162 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767)
I20260130 02:58:08.528522 19840 ts_tablet_manager.cc:1434] T 127a84346b9f473d9e7f739b1e65feb8 P 78c2c46c7520492db35d51c244fb4f74: Time spent starting tablet: real 0.022s user 0.001s sys 0.004s
I20260130 02:58:08.529362 19840 tablet_bootstrap.cc:492] T 3ae5472cba1e4613abe4cda42ccb2496 P 78c2c46c7520492db35d51c244fb4f74: Bootstrap starting.
I20260130 02:58:08.530269 20194 raft_consensus.cc:493] T 3593f8b89da24ff9b6cf549d5ac58bc6 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:08.530692 20194 raft_consensus.cc:515] T 3593f8b89da24ff9b6cf549d5ac58bc6 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:08.531798 19225 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ee7c8ff92ca9465fa18be6eae63d78ca" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152" is_pre_election: true
I20260130 02:58:08.532382 19225 raft_consensus.cc:2468] T ee7c8ff92ca9465fa18be6eae63d78ca P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78c2c46c7520492db35d51c244fb4f74 in term 0.
I20260130 02:58:08.532452 20194 leader_election.cc:290] T 3593f8b89da24ff9b6cf549d5ac58bc6 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), d870164263bb44b6b780a557daea3656 (127.18.100.67:39571)
I20260130 02:58:08.533741 19314 leader_election.cc:304] T ee7c8ff92ca9465fa18be6eae63d78ca P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 371a36d219164974abdf848841528152, 78c2c46c7520492db35d51c244fb4f74; no voters: d870164263bb44b6b780a557daea3656
I20260130 02:58:08.534735 19526 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3593f8b89da24ff9b6cf549d5ac58bc6" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
I20260130 02:58:08.535750 20194 raft_consensus.cc:2804] T ee7c8ff92ca9465fa18be6eae63d78ca P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260130 02:58:08.536072 20194 raft_consensus.cc:493] T ee7c8ff92ca9465fa18be6eae63d78ca P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260130 02:58:08.536348 20194 raft_consensus.cc:3060] T ee7c8ff92ca9465fa18be6eae63d78ca P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:08.537613 19222 raft_consensus.cc:2468] T f54addb1a4304ce69c711ae0cc3b452c P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78c2c46c7520492db35d51c244fb4f74 in term 1.
W20260130 02:58:08.539383 19313 leader_election.cc:343] T 3593f8b89da24ff9b6cf549d5ac58bc6 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:08.543951 19807 ts_tablet_manager.cc:1434] T 3593f8b89da24ff9b6cf549d5ac58bc6 P 371a36d219164974abdf848841528152: Time spent starting tablet: real 0.054s user 0.000s sys 0.006s
I20260130 02:58:08.545163 19314 leader_election.cc:304] T f54addb1a4304ce69c711ae0cc3b452c P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 371a36d219164974abdf848841528152, 78c2c46c7520492db35d51c244fb4f74; no voters: d870164263bb44b6b780a557daea3656
I20260130 02:58:08.546152 20180 raft_consensus.cc:2804] T f54addb1a4304ce69c711ae0cc3b452c P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Leader election won for term 1
I20260130 02:58:08.546561 20180 raft_consensus.cc:697] T f54addb1a4304ce69c711ae0cc3b452c P 78c2c46c7520492db35d51c244fb4f74 [term 1 LEADER]: Becoming Leader. State: Replica: 78c2c46c7520492db35d51c244fb4f74, State: Running, Role: LEADER
I20260130 02:58:08.546530 19159 leader_election.cc:304] T f54addb1a4304ce69c711ae0cc3b452c P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 371a36d219164974abdf848841528152; no voters: 78c2c46c7520492db35d51c244fb4f74, d870164263bb44b6b780a557daea3656
I20260130 02:58:08.547339 20180 consensus_queue.cc:237] T f54addb1a4304ce69c711ae0cc3b452c P 78c2c46c7520492db35d51c244fb4f74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:08.548794 20243 raft_consensus.cc:2749] T f54addb1a4304ce69c711ae0cc3b452c P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260130 02:58:08.556641 19807 tablet_bootstrap.cc:492] T b89ed298e3dc49d789d166c613ad7fb8 P 371a36d219164974abdf848841528152: Bootstrap starting.
I20260130 02:58:08.556573 19225 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3593f8b89da24ff9b6cf549d5ac58bc6" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152" is_pre_election: true
I20260130 02:58:08.557232 19225 raft_consensus.cc:2468] T 3593f8b89da24ff9b6cf549d5ac58bc6 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78c2c46c7520492db35d51c244fb4f74 in term 0.
I20260130 02:58:08.558338 19314 leader_election.cc:304] T 3593f8b89da24ff9b6cf549d5ac58bc6 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 371a36d219164974abdf848841528152, 78c2c46c7520492db35d51c244fb4f74; no voters: d870164263bb44b6b780a557daea3656
I20260130 02:58:08.559367 20240 raft_consensus.cc:2804] T 3593f8b89da24ff9b6cf549d5ac58bc6 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260130 02:58:08.559686 20240 raft_consensus.cc:493] T 3593f8b89da24ff9b6cf549d5ac58bc6 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260130 02:58:08.559988 20240 raft_consensus.cc:3060] T 3593f8b89da24ff9b6cf549d5ac58bc6 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Advancing to term 1
W20260130 02:58:08.560393 19463 leader_election.cc:343] T 3cf5154d8cc44f2bb12978202699c193 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Not found: Tablet not found: 3cf5154d8cc44f2bb12978202699c193
I20260130 02:58:08.560748 19463 leader_election.cc:304] T 3cf5154d8cc44f2bb12978202699c193 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d870164263bb44b6b780a557daea3656; no voters: 371a36d219164974abdf848841528152, 78c2c46c7520492db35d51c244fb4f74
I20260130 02:58:08.561378 20325 raft_consensus.cc:2749] T 3cf5154d8cc44f2bb12978202699c193 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260130 02:58:08.562849 20312 leader_election.cc:290] T 3cf5154d8cc44f2bb12978202699c193 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767)
I20260130 02:58:08.567315 19807 tablet_bootstrap.cc:654] T b89ed298e3dc49d789d166c613ad7fb8 P 371a36d219164974abdf848841528152: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:08.567256 20240 raft_consensus.cc:515] T 3593f8b89da24ff9b6cf549d5ac58bc6 P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:08.569602 20240 leader_election.cc:290] T 3593f8b89da24ff9b6cf549d5ac58bc6 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 election: Requested vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), d870164263bb44b6b780a557daea3656 (127.18.100.67:39571)
I20260130 02:58:08.570544 19159 leader_election.cc:304] T b2ac91da01954c2390b6b7dcc06c394f P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 371a36d219164974abdf848841528152, 78c2c46c7520492db35d51c244fb4f74; no voters:
I20260130 02:58:08.571647 19625 tablet_bootstrap.cc:654] T 3b148691090e4ee581d46f9845a6e49a P d870164263bb44b6b780a557daea3656: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:08.572077 20243 raft_consensus.cc:2804] T b2ac91da01954c2390b6b7dcc06c394f P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260130 02:58:08.572422 20243 raft_consensus.cc:493] T b2ac91da01954c2390b6b7dcc06c394f P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260130 02:58:08.572116 20194 raft_consensus.cc:515] T ee7c8ff92ca9465fa18be6eae63d78ca P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:08.572718 20243 raft_consensus.cc:3060] T b2ac91da01954c2390b6b7dcc06c394f P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:08.574242 20194 leader_election.cc:290] T ee7c8ff92ca9465fa18be6eae63d78ca P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 election: Requested vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), d870164263bb44b6b780a557daea3656 (127.18.100.67:39571)
I20260130 02:58:08.576563 19225 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3593f8b89da24ff9b6cf549d5ac58bc6" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152"
I20260130 02:58:08.577096 19225 raft_consensus.cc:3060] T 3593f8b89da24ff9b6cf549d5ac58bc6 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:08.581318 20243 raft_consensus.cc:515] T b2ac91da01954c2390b6b7dcc06c394f P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:08.584486 20243 leader_election.cc:290] T b2ac91da01954c2390b6b7dcc06c394f P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 election: Requested vote from peers 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767), d870164263bb44b6b780a557daea3656 (127.18.100.67:39571)
I20260130 02:58:08.590502 19526 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ee7c8ff92ca9465fa18be6eae63d78ca" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656"
I20260130 02:58:08.592129 19521 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3593f8b89da24ff9b6cf549d5ac58bc6" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656"
I20260130 02:58:08.596024 19625 tablet_bootstrap.cc:492] T 3b148691090e4ee581d46f9845a6e49a P d870164263bb44b6b780a557daea3656: No bootstrap required, opened a new log
I20260130 02:58:08.584729 20256 ts_tablet_manager.cc:1929] T 5ca9919f80954e74a4d91c94b74de0e4 P 371a36d219164974abdf848841528152: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260130 02:58:08.599344 20256 log.cc:1199] T 5ca9919f80954e74a4d91c94b74de0e4 P 371a36d219164974abdf848841528152: Deleting WAL directory at /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-0/wal/wals/5ca9919f80954e74a4d91c94b74de0e4
I20260130 02:58:08.600142 20256 ts_tablet_manager.cc:1950] T 5ca9919f80954e74a4d91c94b74de0e4 P 371a36d219164974abdf848841528152: Deleting consensus metadata
I20260130 02:58:08.601428 20256 tablet_replica.cc:333] T 8bd3e1c06b89426da57e2b259e7983af P 371a36d219164974abdf848841528152: stopping tablet replica
I20260130 02:58:08.602370 20256 raft_consensus.cc:2243] T 8bd3e1c06b89426da57e2b259e7983af P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260130 02:58:08.602923 20256 raft_consensus.cc:2272] T 8bd3e1c06b89426da57e2b259e7983af P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260130 02:58:08.604844 18861 catalog_manager.cc:4985] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021): tablet 5ca9919f80954e74a4d91c94b74de0e4 (table default.Table_9d0cd9756483406081c8bdb18565a42f [id=5f6488df8ba94ef293b181328cc3b680]) successfully deleted
I20260130 02:58:08.606766 19625 ts_tablet_manager.cc:1403] T 3b148691090e4ee581d46f9845a6e49a P d870164263bb44b6b780a557daea3656: Time spent bootstrapping tablet: real 0.435s user 0.010s sys 0.000s
W20260130 02:58:08.608188 19313 leader_election.cc:343] T ee7c8ff92ca9465fa18be6eae63d78ca P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:08.609614 19376 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b2ac91da01954c2390b6b7dcc06c394f" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74"
I20260130 02:58:08.610157 19376 raft_consensus.cc:3060] T b2ac91da01954c2390b6b7dcc06c394f P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:08.616194 19625 raft_consensus.cc:359] T 3b148691090e4ee581d46f9845a6e49a P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:08.617152 19376 raft_consensus.cc:2468] T b2ac91da01954c2390b6b7dcc06c394f P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 371a36d219164974abdf848841528152 in term 1.
I20260130 02:58:08.618299 19840 tablet_bootstrap.cc:654] T 3ae5472cba1e4613abe4cda42ccb2496 P 78c2c46c7520492db35d51c244fb4f74: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:08.587809 19223 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b8c579ff5fa84651ba9125457d927162" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152" is_pre_election: true
I20260130 02:58:08.604282 19224 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ee7c8ff92ca9465fa18be6eae63d78ca" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152"
I20260130 02:58:08.619935 19224 raft_consensus.cc:3060] T ee7c8ff92ca9465fa18be6eae63d78ca P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:08.623566 19625 raft_consensus.cc:385] T 3b148691090e4ee581d46f9845a6e49a P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:08.624202 19625 raft_consensus.cc:740] T 3b148691090e4ee581d46f9845a6e49a P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d870164263bb44b6b780a557daea3656, State: Initialized, Role: FOLLOWER
I20260130 02:58:08.625120 19625 consensus_queue.cc:260] T 3b148691090e4ee581d46f9845a6e49a P d870164263bb44b6b780a557daea3656 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:08.626194 19224 raft_consensus.cc:2468] T ee7c8ff92ca9465fa18be6eae63d78ca P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78c2c46c7520492db35d51c244fb4f74 in term 1.
I20260130 02:58:08.587150 19225 raft_consensus.cc:2468] T 3593f8b89da24ff9b6cf549d5ac58bc6 P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78c2c46c7520492db35d51c244fb4f74 in term 1.
W20260130 02:58:08.629276 19159 leader_election.cc:343] T b2ac91da01954c2390b6b7dcc06c394f P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:08.633752 19314 leader_election.cc:304] T ee7c8ff92ca9465fa18be6eae63d78ca P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 371a36d219164974abdf848841528152, 78c2c46c7520492db35d51c244fb4f74; no voters: d870164263bb44b6b780a557daea3656
I20260130 02:58:08.634099 19625 ts_tablet_manager.cc:1434] T 3b148691090e4ee581d46f9845a6e49a P d870164263bb44b6b780a557daea3656: Time spent starting tablet: real 0.027s user 0.006s sys 0.000s
I20260130 02:58:08.634642 19314 leader_election.cc:304] T 3593f8b89da24ff9b6cf549d5ac58bc6 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 371a36d219164974abdf848841528152, 78c2c46c7520492db35d51c244fb4f74; no voters:
W20260130 02:58:08.632390 19159 consensus_peers.cc:597] T 730bd1d7d15b4cbe945b3ef03b2455f7 P 371a36d219164974abdf848841528152 -> Peer 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767): Couldn't send request to peer 78c2c46c7520492db35d51c244fb4f74. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 730bd1d7d15b4cbe945b3ef03b2455f7. This is attempt 1: this message will repeat every 5th retry.
W20260130 02:58:08.635488 19313 leader_election.cc:343] T 3593f8b89da24ff9b6cf549d5ac58bc6 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:08.637650 19159 leader_election.cc:304] T b2ac91da01954c2390b6b7dcc06c394f P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 371a36d219164974abdf848841528152, 78c2c46c7520492db35d51c244fb4f74; no voters:
I20260130 02:58:08.638831 20244 raft_consensus.cc:2804] T b2ac91da01954c2390b6b7dcc06c394f P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Leader election won for term 1
I20260130 02:58:08.639281 20244 raft_consensus.cc:697] T b2ac91da01954c2390b6b7dcc06c394f P 371a36d219164974abdf848841528152 [term 1 LEADER]: Becoming Leader. State: Replica: 371a36d219164974abdf848841528152, State: Running, Role: LEADER
I20260130 02:58:08.640162 20244 consensus_queue.cc:237] T b2ac91da01954c2390b6b7dcc06c394f P 371a36d219164974abdf848841528152 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:08.642256 20180 raft_consensus.cc:2804] T 3593f8b89da24ff9b6cf549d5ac58bc6 P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Leader election won for term 1
I20260130 02:58:08.642730 20180 raft_consensus.cc:697] T 3593f8b89da24ff9b6cf549d5ac58bc6 P 78c2c46c7520492db35d51c244fb4f74 [term 1 LEADER]: Becoming Leader. State: Replica: 78c2c46c7520492db35d51c244fb4f74, State: Running, Role: LEADER
I20260130 02:58:08.643481 20180 consensus_queue.cc:237] T 3593f8b89da24ff9b6cf549d5ac58bc6 P 78c2c46c7520492db35d51c244fb4f74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:08.645541 19526 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b2ac91da01954c2390b6b7dcc06c394f" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656"
I20260130 02:58:08.634625 20194 raft_consensus.cc:2804] T ee7c8ff92ca9465fa18be6eae63d78ca P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Leader election won for term 1
I20260130 02:58:08.648063 20194 raft_consensus.cc:697] T ee7c8ff92ca9465fa18be6eae63d78ca P 78c2c46c7520492db35d51c244fb4f74 [term 1 LEADER]: Becoming Leader. State: Replica: 78c2c46c7520492db35d51c244fb4f74, State: Running, Role: LEADER
I20260130 02:58:08.648962 20194 consensus_queue.cc:237] T ee7c8ff92ca9465fa18be6eae63d78ca P 78c2c46c7520492db35d51c244fb4f74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:08.657686 20256 ts_tablet_manager.cc:1916] T 8bd3e1c06b89426da57e2b259e7983af P 371a36d219164974abdf848841528152: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260130 02:58:08.663652 20208 ts_tablet_manager.cc:1929] T 5ca9919f80954e74a4d91c94b74de0e4 P 78c2c46c7520492db35d51c244fb4f74: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260130 02:58:08.664116 20208 log.cc:1199] T 5ca9919f80954e74a4d91c94b74de0e4 P 78c2c46c7520492db35d51c244fb4f74: Deleting WAL directory at /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-1/wal/wals/5ca9919f80954e74a4d91c94b74de0e4
I20260130 02:58:08.666404 19807 tablet_bootstrap.cc:492] T b89ed298e3dc49d789d166c613ad7fb8 P 371a36d219164974abdf848841528152: No bootstrap required, opened a new log
I20260130 02:58:08.666893 19807 ts_tablet_manager.cc:1403] T b89ed298e3dc49d789d166c613ad7fb8 P 371a36d219164974abdf848841528152: Time spent bootstrapping tablet: real 0.110s user 0.011s sys 0.001s
I20260130 02:58:08.669142 19807 raft_consensus.cc:359] T b89ed298e3dc49d789d166c613ad7fb8 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:08.674446 19807 raft_consensus.cc:385] T b89ed298e3dc49d789d166c613ad7fb8 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:08.674760 19807 raft_consensus.cc:740] T b89ed298e3dc49d789d166c613ad7fb8 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 371a36d219164974abdf848841528152, State: Initialized, Role: FOLLOWER
I20260130 02:58:08.675335 19807 consensus_queue.cc:260] T b89ed298e3dc49d789d166c613ad7fb8 P 371a36d219164974abdf848841528152 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:08.679937 19807 ts_tablet_manager.cc:1434] T b89ed298e3dc49d789d166c613ad7fb8 P 371a36d219164974abdf848841528152: Time spent starting tablet: real 0.013s user 0.004s sys 0.000s
I20260130 02:58:08.680030 18873 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50284:
table { table_name: "default.Table_5d29239ce1984d2fad02607a2bdd4dd0" } modify_external_catalogs: true
I20260130 02:58:08.680545 18873 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50284:
table { table_name: "default.Table_5d29239ce1984d2fad02607a2bdd4dd0" } modify_external_catalogs: true
I20260130 02:58:08.681375 20208 ts_tablet_manager.cc:1950] T 5ca9919f80954e74a4d91c94b74de0e4 P 78c2c46c7520492db35d51c244fb4f74: Deleting consensus metadata
I20260130 02:58:08.687947 19625 tablet_bootstrap.cc:492] T 54be22352e404ff9b798a075693f7f21 P d870164263bb44b6b780a557daea3656: Bootstrap starting.
W20260130 02:58:08.689733 19159 leader_election.cc:343] T b2ac91da01954c2390b6b7dcc06c394f P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:08.701197 19807 tablet_bootstrap.cc:492] T 04f4ac27800c4ac3a79ae12d35479757 P 371a36d219164974abdf848841528152: Bootstrap starting.
I20260130 02:58:08.707821 18866 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50284:
table { table_name: "default.Table_b2d2da7aae8b430aba32bac2944aa931" } new_table_name: "default.Table_1703c69112744c0e951c1ac9667bbac6" modify_external_catalogs: true
I20260130 02:58:08.715718 19625 tablet_bootstrap.cc:654] T 54be22352e404ff9b798a075693f7f21 P d870164263bb44b6b780a557daea3656: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:08.727690 19947 ts_tablet_manager.cc:1929] T 5ca9919f80954e74a4d91c94b74de0e4 P d870164263bb44b6b780a557daea3656: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260130 02:58:08.737035 19947 log.cc:1199] T 5ca9919f80954e74a4d91c94b74de0e4 P d870164263bb44b6b780a557daea3656: Deleting WAL directory at /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-2/wal/wals/5ca9919f80954e74a4d91c94b74de0e4
I20260130 02:58:08.737850 19807 tablet_bootstrap.cc:654] T 04f4ac27800c4ac3a79ae12d35479757 P 371a36d219164974abdf848841528152: Neither blocks nor log segments found. Creating new log.
W20260130 02:58:08.754102 18860 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet b1e8b78eff0d4a0b92059adb2de4a3ce (table default.Table_d0b4516d24384e5386c1debaf2543adb [id=f1342f9497b74e7a9ecad1de8082578d], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet b1e8b78eff0d4a0b92059adb2de4a3ce consensus state has no leader
I20260130 02:58:08.753934 19375 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b8c579ff5fa84651ba9125457d927162" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74" is_pre_election: true
I20260130 02:58:08.756973 20208 tablet_replica.cc:333] T 8bd3e1c06b89426da57e2b259e7983af P 78c2c46c7520492db35d51c244fb4f74: stopping tablet replica
I20260130 02:58:08.757735 20208 raft_consensus.cc:2243] T 8bd3e1c06b89426da57e2b259e7983af P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260130 02:58:08.761654 20256 ts_tablet_manager.cc:1929] T 8bd3e1c06b89426da57e2b259e7983af P 371a36d219164974abdf848841528152: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20260130 02:58:08.738046 20342 log.cc:927] Time spent T ee7c8ff92ca9465fa18be6eae63d78ca P 78c2c46c7520492db35d51c244fb4f74: Append to log took a long time: real 0.081s user 0.001s sys 0.000s
I20260130 02:58:08.766862 20208 raft_consensus.cc:2272] T 8bd3e1c06b89426da57e2b259e7983af P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260130 02:58:08.768524 18860 catalog_manager.cc:4985] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767): tablet 5ca9919f80954e74a4d91c94b74de0e4 (table default.Table_9d0cd9756483406081c8bdb18565a42f [id=5f6488df8ba94ef293b181328cc3b680]) successfully deleted
I20260130 02:58:08.768594 20208 ts_tablet_manager.cc:1916] T 8bd3e1c06b89426da57e2b259e7983af P 78c2c46c7520492db35d51c244fb4f74: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260130 02:58:08.770857 20256 log.cc:1199] T 8bd3e1c06b89426da57e2b259e7983af P 371a36d219164974abdf848841528152: Deleting WAL directory at /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-0/wal/wals/8bd3e1c06b89426da57e2b259e7983af
I20260130 02:58:08.771524 20256 ts_tablet_manager.cc:1950] T 8bd3e1c06b89426da57e2b259e7983af P 371a36d219164974abdf848841528152: Deleting consensus metadata
I20260130 02:58:08.782501 18870 catalog_manager.cc:5941] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4: Sending DeleteTablet for 3 replicas of tablet e2aaaf00f53d418b87d5c7af286e4d84
I20260130 02:58:08.784485 18870 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50284:
name: "default.Table_d3f82845a18f4a4a830460cb1cbbb2de"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260130 02:58:08.786005 18870 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d3f82845a18f4a4a830460cb1cbbb2de in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260130 02:58:08.785581 20313 consensus_queue.cc:1048] T 341041cb0be2456e88403367c0814704 P d870164263bb44b6b780a557daea3656 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260130 02:58:08.787078 18861 catalog_manager.cc:4985] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021): tablet 8bd3e1c06b89426da57e2b259e7983af (table default.Table_1d9d1f3b2b6048f8949fdba26f5fd3d7 [id=1b409bcef4ca4d248418597c1aa69fdb]) successfully deleted
I20260130 02:58:08.787508 19201 tablet_service.cc:1552] Processing DeleteTablet for tablet e2aaaf00f53d418b87d5c7af286e4d84 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:58:08 UTC) from {username='slave'} at 127.0.0.1:42870
I20260130 02:58:08.788404 20256 tablet_replica.cc:333] T 3b4908bd1a634ca0907234dd3c39e1c3 P 371a36d219164974abdf848841528152: stopping tablet replica
I20260130 02:58:08.789848 19947 ts_tablet_manager.cc:1950] T 5ca9919f80954e74a4d91c94b74de0e4 P d870164263bb44b6b780a557daea3656: Deleting consensus metadata
I20260130 02:58:08.790463 20256 raft_consensus.cc:2243] T 3b4908bd1a634ca0907234dd3c39e1c3 P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260130 02:58:08.792512 20256 raft_consensus.cc:2272] T 3b4908bd1a634ca0907234dd3c39e1c3 P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260130 02:58:08.793154 19501 tablet_service.cc:1552] Processing DeleteTablet for tablet e2aaaf00f53d418b87d5c7af286e4d84 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:58:08 UTC) from {username='slave'} at 127.0.0.1:59068
I20260130 02:58:08.793404 18860 catalog_manager.cc:4985] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): tablet 5ca9919f80954e74a4d91c94b74de0e4 (table default.Table_9d0cd9756483406081c8bdb18565a42f [id=5f6488df8ba94ef293b181328cc3b680]) successfully deleted
W20260130 02:58:08.795092 18860 catalog_manager.cc:4965] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): delete failed for tablet 8bd3e1c06b89426da57e2b259e7983af because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8bd3e1c06b89426da57e2b259e7983af already in progress: creating tablet
I20260130 02:58:08.796583 20256 ts_tablet_manager.cc:1916] T 3b4908bd1a634ca0907234dd3c39e1c3 P 371a36d219164974abdf848841528152: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260130 02:58:08.797335 19355 tablet_service.cc:1552] Processing DeleteTablet for tablet e2aaaf00f53d418b87d5c7af286e4d84 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:58:08 UTC) from {username='slave'} at 127.0.0.1:60110
I20260130 02:58:08.807382 20344 tablet.cc:1722] T 91f954f3ef4244c9a575a7421a7e75eb P 371a36d219164974abdf848841528152: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260130 02:58:08.809213 20343 tablet.cc:1722] T 91f954f3ef4244c9a575a7421a7e75eb P d870164263bb44b6b780a557daea3656: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260130 02:58:08.809814 19840 tablet_bootstrap.cc:492] T 3ae5472cba1e4613abe4cda42ccb2496 P 78c2c46c7520492db35d51c244fb4f74: No bootstrap required, opened a new log
I20260130 02:58:08.810223 19840 ts_tablet_manager.cc:1403] T 3ae5472cba1e4613abe4cda42ccb2496 P 78c2c46c7520492db35d51c244fb4f74: Time spent bootstrapping tablet: real 0.281s user 0.004s sys 0.010s
I20260130 02:58:08.810793 19625 tablet_bootstrap.cc:492] T 54be22352e404ff9b798a075693f7f21 P d870164263bb44b6b780a557daea3656: No bootstrap required, opened a new log
I20260130 02:58:08.812628 19625 ts_tablet_manager.cc:1403] T 54be22352e404ff9b798a075693f7f21 P d870164263bb44b6b780a557daea3656: Time spent bootstrapping tablet: real 0.125s user 0.008s sys 0.003s
I20260130 02:58:08.812570 19840 raft_consensus.cc:359] T 3ae5472cba1e4613abe4cda42ccb2496 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:08.813189 19840 raft_consensus.cc:385] T 3ae5472cba1e4613abe4cda42ccb2496 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:08.813447 19840 raft_consensus.cc:740] T 3ae5472cba1e4613abe4cda42ccb2496 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78c2c46c7520492db35d51c244fb4f74, State: Initialized, Role: FOLLOWER
I20260130 02:58:08.814004 19840 consensus_queue.cc:260] T 3ae5472cba1e4613abe4cda42ccb2496 P 78c2c46c7520492db35d51c244fb4f74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:08.815359 19625 raft_consensus.cc:359] T 54be22352e404ff9b798a075693f7f21 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:08.812557 19807 tablet_bootstrap.cc:492] T 04f4ac27800c4ac3a79ae12d35479757 P 371a36d219164974abdf848841528152: No bootstrap required, opened a new log
I20260130 02:58:08.816951 19807 ts_tablet_manager.cc:1403] T 04f4ac27800c4ac3a79ae12d35479757 P 371a36d219164974abdf848841528152: Time spent bootstrapping tablet: real 0.116s user 0.019s sys 0.002s
I20260130 02:58:08.818022 19462 leader_election.cc:304] T b8c579ff5fa84651ba9125457d927162 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d870164263bb44b6b780a557daea3656; no voters: 371a36d219164974abdf848841528152, 78c2c46c7520492db35d51c244fb4f74
I20260130 02:58:08.820464 20313 raft_consensus.cc:2749] T b8c579ff5fa84651ba9125457d927162 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260130 02:58:08.821502 19625 raft_consensus.cc:385] T 54be22352e404ff9b798a075693f7f21 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:08.829473 19625 raft_consensus.cc:740] T 54be22352e404ff9b798a075693f7f21 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d870164263bb44b6b780a557daea3656, State: Initialized, Role: FOLLOWER
I20260130 02:58:08.835721 19807 raft_consensus.cc:359] T 04f4ac27800c4ac3a79ae12d35479757 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:08.836463 19625 consensus_queue.cc:260] T 54be22352e404ff9b798a075693f7f21 P d870164263bb44b6b780a557daea3656 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:08.838709 19807 raft_consensus.cc:385] T 04f4ac27800c4ac3a79ae12d35479757 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:08.845007 19807 raft_consensus.cc:740] T 04f4ac27800c4ac3a79ae12d35479757 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 371a36d219164974abdf848841528152, State: Initialized, Role: FOLLOWER
I20260130 02:58:08.846228 19625 ts_tablet_manager.cc:1434] T 54be22352e404ff9b798a075693f7f21 P d870164263bb44b6b780a557daea3656: Time spent starting tablet: real 0.033s user 0.001s sys 0.004s
I20260130 02:58:08.846702 19807 consensus_queue.cc:260] T 04f4ac27800c4ac3a79ae12d35479757 P 371a36d219164974abdf848841528152 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:08.814831 20208 ts_tablet_manager.cc:1929] T 8bd3e1c06b89426da57e2b259e7983af P 78c2c46c7520492db35d51c244fb4f74: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260130 02:58:08.850785 19625 tablet_bootstrap.cc:492] T 8bd3e1c06b89426da57e2b259e7983af P d870164263bb44b6b780a557daea3656: Bootstrap starting.
I20260130 02:58:08.850944 20208 log.cc:1199] T 8bd3e1c06b89426da57e2b259e7983af P 78c2c46c7520492db35d51c244fb4f74: Deleting WAL directory at /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-1/wal/wals/8bd3e1c06b89426da57e2b259e7983af
I20260130 02:58:08.851600 20208 ts_tablet_manager.cc:1950] T 8bd3e1c06b89426da57e2b259e7983af P 78c2c46c7520492db35d51c244fb4f74: Deleting consensus metadata
I20260130 02:58:08.853132 19807 ts_tablet_manager.cc:1434] T 04f4ac27800c4ac3a79ae12d35479757 P 371a36d219164974abdf848841528152: Time spent starting tablet: real 0.036s user 0.000s sys 0.006s
I20260130 02:58:08.853665 20208 tablet_replica.cc:333] T 3b4908bd1a634ca0907234dd3c39e1c3 P 78c2c46c7520492db35d51c244fb4f74: stopping tablet replica
I20260130 02:58:08.839085 20348 tablet.cc:1722] T 91f954f3ef4244c9a575a7421a7e75eb P 78c2c46c7520492db35d51c244fb4f74: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260130 02:58:08.858855 20208 raft_consensus.cc:2243] T 3b4908bd1a634ca0907234dd3c39e1c3 P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260130 02:58:08.859306 20208 raft_consensus.cc:2272] T 3b4908bd1a634ca0907234dd3c39e1c3 P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260130 02:58:08.861200 20208 ts_tablet_manager.cc:1916] T 3b4908bd1a634ca0907234dd3c39e1c3 P 78c2c46c7520492db35d51c244fb4f74: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260130 02:58:08.862886 18860 catalog_manager.cc:4985] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767): tablet 8bd3e1c06b89426da57e2b259e7983af (table default.Table_1d9d1f3b2b6048f8949fdba26f5fd3d7 [id=1b409bcef4ca4d248418597c1aa69fdb]) successfully deleted
I20260130 02:58:08.864327 19840 ts_tablet_manager.cc:1434] T 3ae5472cba1e4613abe4cda42ccb2496 P 78c2c46c7520492db35d51c244fb4f74: Time spent starting tablet: real 0.054s user 0.000s sys 0.006s
I20260130 02:58:08.867393 19807 tablet_bootstrap.cc:492] T 7d96e93d95f14af79cfb268626559b1e P 371a36d219164974abdf848841528152: Bootstrap starting.
I20260130 02:58:08.844491 20256 ts_tablet_manager.cc:1929] T 3b4908bd1a634ca0907234dd3c39e1c3 P 371a36d219164974abdf848841528152: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260130 02:58:08.870560 20256 log.cc:1199] T 3b4908bd1a634ca0907234dd3c39e1c3 P 371a36d219164974abdf848841528152: Deleting WAL directory at /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-0/wal/wals/3b4908bd1a634ca0907234dd3c39e1c3
I20260130 02:58:08.878398 20256 ts_tablet_manager.cc:1950] T 3b4908bd1a634ca0907234dd3c39e1c3 P 371a36d219164974abdf848841528152: Deleting consensus metadata
I20260130 02:58:08.881584 19807 tablet_bootstrap.cc:654] T 7d96e93d95f14af79cfb268626559b1e P 371a36d219164974abdf848841528152: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:08.894317 18861 catalog_manager.cc:4985] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021): tablet 3b4908bd1a634ca0907234dd3c39e1c3 (table default.Table_0ad5d205dfea4344a759522f45a55475 [id=5b7d0747e1bd438ea7e2f10abd339c2c]) successfully deleted
W20260130 02:58:08.895656 18861 catalog_manager.cc:4965] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021): delete failed for tablet e2aaaf00f53d418b87d5c7af286e4d84 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e2aaaf00f53d418b87d5c7af286e4d84 already in progress: creating tablet
I20260130 02:58:08.870393 19947 tablet_replica.cc:333] T 3b4908bd1a634ca0907234dd3c39e1c3 P d870164263bb44b6b780a557daea3656: stopping tablet replica
I20260130 02:58:08.912338 19807 tablet_bootstrap.cc:492] T 7d96e93d95f14af79cfb268626559b1e P 371a36d219164974abdf848841528152: No bootstrap required, opened a new log
I20260130 02:58:08.912775 19807 ts_tablet_manager.cc:1403] T 7d96e93d95f14af79cfb268626559b1e P 371a36d219164974abdf848841528152: Time spent bootstrapping tablet: real 0.047s user 0.003s sys 0.005s
I20260130 02:58:08.915277 19807 raft_consensus.cc:359] T 7d96e93d95f14af79cfb268626559b1e P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:08.915954 19807 raft_consensus.cc:385] T 7d96e93d95f14af79cfb268626559b1e P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:08.916249 19807 raft_consensus.cc:740] T 7d96e93d95f14af79cfb268626559b1e P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 371a36d219164974abdf848841528152, State: Initialized, Role: FOLLOWER
I20260130 02:58:08.916859 19807 consensus_queue.cc:260] T 7d96e93d95f14af79cfb268626559b1e P 371a36d219164974abdf848841528152 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:08.919214 19807 ts_tablet_manager.cc:1434] T 7d96e93d95f14af79cfb268626559b1e P 371a36d219164974abdf848841528152: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260130 02:58:08.921634 19625 tablet_bootstrap.cc:654] T 8bd3e1c06b89426da57e2b259e7983af P d870164263bb44b6b780a557daea3656: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:08.922175 19807 tablet_bootstrap.cc:492] T 3ae5472cba1e4613abe4cda42ccb2496 P 371a36d219164974abdf848841528152: Bootstrap starting.
I20260130 02:58:08.927205 19807 tablet_bootstrap.cc:654] T 3ae5472cba1e4613abe4cda42ccb2496 P 371a36d219164974abdf848841528152: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:08.935599 19840 tablet_bootstrap.cc:492] T 04f4ac27800c4ac3a79ae12d35479757 P 78c2c46c7520492db35d51c244fb4f74: Bootstrap starting.
I20260130 02:58:08.937613 20208 ts_tablet_manager.cc:1929] T 3b4908bd1a634ca0907234dd3c39e1c3 P 78c2c46c7520492db35d51c244fb4f74: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260130 02:58:08.940649 20341 raft_consensus.cc:493] T 3ae5472cba1e4613abe4cda42ccb2496 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:08.942889 20341 raft_consensus.cc:515] T 3ae5472cba1e4613abe4cda42ccb2496 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:08.944777 20341 leader_election.cc:290] T 3ae5472cba1e4613abe4cda42ccb2496 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), d870164263bb44b6b780a557daea3656 (127.18.100.67:39571)
I20260130 02:58:08.946445 19222 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3ae5472cba1e4613abe4cda42ccb2496" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152" is_pre_election: true
W20260130 02:58:08.947932 19314 leader_election.cc:343] T 3ae5472cba1e4613abe4cda42ccb2496 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:08.948578 20208 log.cc:1199] T 3b4908bd1a634ca0907234dd3c39e1c3 P 78c2c46c7520492db35d51c244fb4f74: Deleting WAL directory at /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-1/wal/wals/3b4908bd1a634ca0907234dd3c39e1c3
I20260130 02:58:08.949942 20208 ts_tablet_manager.cc:1950] T 3b4908bd1a634ca0907234dd3c39e1c3 P 78c2c46c7520492db35d51c244fb4f74: Deleting consensus metadata
I20260130 02:58:08.952797 18875 catalog_manager.cc:5941] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4: Sending DeleteTablet for 3 replicas of tablet 1bbc55bbf2254d8bb6b721d7e8f69d85
I20260130 02:58:08.954126 18875 catalog_manager.cc:6724] ReplaceTablet: tablet 1bbc55bbf2254d8bb6b721d7e8f69d85 deleted and replaced by tablet 657e9b0d8c05413f904129dce745710b
I20260130 02:58:08.961508 19201 tablet_service.cc:1552] Processing DeleteTablet for tablet 1bbc55bbf2254d8bb6b721d7e8f69d85 with delete_type TABLET_DATA_DELETED (replaced by tablet 657e9b0d8c05413f904129dce745710b) from {username='slave'} at 127.0.0.1:42870
I20260130 02:58:08.962181 20256 tablet_replica.cc:333] T 1bbc55bbf2254d8bb6b721d7e8f69d85 P 371a36d219164974abdf848841528152: stopping tablet replica
I20260130 02:58:08.962802 18873 catalog_manager.cc:5941] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4: Sending DeleteTablet for 3 replicas of tablet 78e9bc3e55fe4e4a98e8eb01da59d365
I20260130 02:58:08.963042 20256 raft_consensus.cc:2243] T 1bbc55bbf2254d8bb6b721d7e8f69d85 P 371a36d219164974abdf848841528152 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260130 02:58:08.963194 18860 catalog_manager.cc:4985] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767): tablet 3b4908bd1a634ca0907234dd3c39e1c3 (table default.Table_0ad5d205dfea4344a759522f45a55475 [id=5b7d0747e1bd438ea7e2f10abd339c2c]) successfully deleted
I20260130 02:58:08.963575 20256 raft_consensus.cc:2272] T 1bbc55bbf2254d8bb6b721d7e8f69d85 P 371a36d219164974abdf848841528152 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260130 02:58:08.965508 20256 ts_tablet_manager.cc:1916] T 1bbc55bbf2254d8bb6b721d7e8f69d85 P 371a36d219164974abdf848841528152: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260130 02:58:08.966547 19355 tablet_service.cc:1552] Processing DeleteTablet for tablet 1bbc55bbf2254d8bb6b721d7e8f69d85 with delete_type TABLET_DATA_DELETED (replaced by tablet 657e9b0d8c05413f904129dce745710b) from {username='slave'} at 127.0.0.1:60110
I20260130 02:58:08.966825 19201 tablet_service.cc:1552] Processing DeleteTablet for tablet 78e9bc3e55fe4e4a98e8eb01da59d365 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:58:08 UTC) from {username='slave'} at 127.0.0.1:42870
I20260130 02:58:08.967216 19358 tablet_service.cc:1552] Processing DeleteTablet for tablet 78e9bc3e55fe4e4a98e8eb01da59d365 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:58:08 UTC) from {username='slave'} at 127.0.0.1:60110
I20260130 02:58:08.968448 18875 master_service.cc:954] ReplaceTablet: received request to replace tablet d27ca7dd87f44102b7680e1f4a88954a from {username='slave'} at 127.0.0.1:50284
W20260130 02:58:08.969213 18860 catalog_manager.cc:4965] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767): delete failed for tablet e2aaaf00f53d418b87d5c7af286e4d84 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e2aaaf00f53d418b87d5c7af286e4d84 already in progress: creating tablet
I20260130 02:58:08.971369 20208 tablet_replica.cc:333] T 1bbc55bbf2254d8bb6b721d7e8f69d85 P 78c2c46c7520492db35d51c244fb4f74: stopping tablet replica
I20260130 02:58:08.972584 20208 raft_consensus.cc:2243] T 1bbc55bbf2254d8bb6b721d7e8f69d85 P 78c2c46c7520492db35d51c244fb4f74 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260130 02:58:08.975916 20208 raft_consensus.cc:2272] T 1bbc55bbf2254d8bb6b721d7e8f69d85 P 78c2c46c7520492db35d51c244fb4f74 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260130 02:58:08.978197 20208 ts_tablet_manager.cc:1916] T 1bbc55bbf2254d8bb6b721d7e8f69d85 P 78c2c46c7520492db35d51c244fb4f74: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260130 02:58:08.978803 20243 raft_consensus.cc:493] T 7d96e93d95f14af79cfb268626559b1e P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:08.979334 20243 raft_consensus.cc:515] T 7d96e93d95f14af79cfb268626559b1e P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:08.981177 20243 leader_election.cc:290] T 7d96e93d95f14af79cfb268626559b1e P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767), d870164263bb44b6b780a557daea3656 (127.18.100.67:39571)
I20260130 02:58:08.982364 19378 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7d96e93d95f14af79cfb268626559b1e" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74" is_pre_election: true
I20260130 02:58:08.983444 19378 raft_consensus.cc:2468] T 7d96e93d95f14af79cfb268626559b1e P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 371a36d219164974abdf848841528152 in term 0.
I20260130 02:58:08.987903 19525 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7d96e93d95f14af79cfb268626559b1e" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
I20260130 02:58:08.989461 19501 tablet_service.cc:1552] Processing DeleteTablet for tablet 1bbc55bbf2254d8bb6b721d7e8f69d85 with delete_type TABLET_DATA_DELETED (replaced by tablet 657e9b0d8c05413f904129dce745710b) from {username='slave'} at 127.0.0.1:59068
I20260130 02:58:08.989905 19496 tablet_service.cc:1552] Processing DeleteTablet for tablet 78e9bc3e55fe4e4a98e8eb01da59d365 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:58:08 UTC) from {username='slave'} at 127.0.0.1:59068
I20260130 02:58:08.990823 19807 tablet_bootstrap.cc:492] T 3ae5472cba1e4613abe4cda42ccb2496 P 371a36d219164974abdf848841528152: No bootstrap required, opened a new log
I20260130 02:58:08.991303 19807 ts_tablet_manager.cc:1403] T 3ae5472cba1e4613abe4cda42ccb2496 P 371a36d219164974abdf848841528152: Time spent bootstrapping tablet: real 0.070s user 0.011s sys 0.022s
I20260130 02:58:08.990975 20244 consensus_queue.cc:1048] T 3b148691090e4ee581d46f9845a6e49a P 371a36d219164974abdf848841528152 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260130 02:58:08.993410 19159 leader_election.cc:343] T 7d96e93d95f14af79cfb268626559b1e P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:08.995288 19625 tablet_bootstrap.cc:492] T 8bd3e1c06b89426da57e2b259e7983af P d870164263bb44b6b780a557daea3656: No bootstrap required, opened a new log
I20260130 02:58:08.995666 19625 ts_tablet_manager.cc:1403] T 8bd3e1c06b89426da57e2b259e7983af P d870164263bb44b6b780a557daea3656: Time spent bootstrapping tablet: real 0.148s user 0.014s sys 0.008s
I20260130 02:58:08.997692 19159 leader_election.cc:304] T 7d96e93d95f14af79cfb268626559b1e P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 371a36d219164974abdf848841528152, 78c2c46c7520492db35d51c244fb4f74; no voters: d870164263bb44b6b780a557daea3656
I20260130 02:58:08.998782 19521 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3ae5472cba1e4613abe4cda42ccb2496" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
I20260130 02:58:09.001870 19625 raft_consensus.cc:359] T 8bd3e1c06b89426da57e2b259e7983af P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
W20260130 02:58:09.004492 19313 leader_election.cc:343] T 3ae5472cba1e4613abe4cda42ccb2496 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:09.004891 19313 leader_election.cc:304] T 3ae5472cba1e4613abe4cda42ccb2496 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78c2c46c7520492db35d51c244fb4f74; no voters: 371a36d219164974abdf848841528152, d870164263bb44b6b780a557daea3656
I20260130 02:58:09.005872 20341 raft_consensus.cc:2749] T 3ae5472cba1e4613abe4cda42ccb2496 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260130 02:58:09.007871 18873 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50284:
table { table_name: "default.Table_6b60df181f10452ba3f2e20e847bf502" } new_table_name: "default.Table_c4f4904b6fac4b9880979cf0314e71c1" modify_external_catalogs: true
I20260130 02:58:09.009514 20256 ts_tablet_manager.cc:1929] T 1bbc55bbf2254d8bb6b721d7e8f69d85 P 371a36d219164974abdf848841528152: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 2.2
I20260130 02:58:09.009682 20244 raft_consensus.cc:2804] T 7d96e93d95f14af79cfb268626559b1e P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260130 02:58:09.009881 20256 log.cc:1199] T 1bbc55bbf2254d8bb6b721d7e8f69d85 P 371a36d219164974abdf848841528152: Deleting WAL directory at /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-0/wal/wals/1bbc55bbf2254d8bb6b721d7e8f69d85
I20260130 02:58:09.010080 20244 raft_consensus.cc:493] T 7d96e93d95f14af79cfb268626559b1e P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260130 02:58:09.010390 20244 raft_consensus.cc:3060] T 7d96e93d95f14af79cfb268626559b1e P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:09.010610 20256 ts_tablet_manager.cc:1950] T 1bbc55bbf2254d8bb6b721d7e8f69d85 P 371a36d219164974abdf848841528152: Deleting consensus metadata
I20260130 02:58:09.013216 18861 catalog_manager.cc:4985] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021): tablet 1bbc55bbf2254d8bb6b721d7e8f69d85 (table default.Table_526677c8ef4144bd9de7a54a5e78e2be [id=b16908b75587459597a978c71756451c]) successfully deleted
W20260130 02:58:09.014313 18861 catalog_manager.cc:4965] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021): delete failed for tablet 78e9bc3e55fe4e4a98e8eb01da59d365 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 78e9bc3e55fe4e4a98e8eb01da59d365 already in progress: creating tablet
I20260130 02:58:09.016789 20244 raft_consensus.cc:515] T 7d96e93d95f14af79cfb268626559b1e P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:09.017616 19625 raft_consensus.cc:385] T 8bd3e1c06b89426da57e2b259e7983af P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:09.018757 20244 leader_election.cc:290] T 7d96e93d95f14af79cfb268626559b1e P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 election: Requested vote from peers 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767), d870164263bb44b6b780a557daea3656 (127.18.100.67:39571)
I20260130 02:58:08.993453 19807 raft_consensus.cc:359] T 3ae5472cba1e4613abe4cda42ccb2496 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:09.019235 19625 raft_consensus.cc:740] T 8bd3e1c06b89426da57e2b259e7983af P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d870164263bb44b6b780a557daea3656, State: Initialized, Role: FOLLOWER
I20260130 02:58:09.019377 19807 raft_consensus.cc:385] T 3ae5472cba1e4613abe4cda42ccb2496 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:09.019629 19807 raft_consensus.cc:740] T 3ae5472cba1e4613abe4cda42ccb2496 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 371a36d219164974abdf848841528152, State: Initialized, Role: FOLLOWER
I20260130 02:58:09.020198 19807 consensus_queue.cc:260] T 3ae5472cba1e4613abe4cda42ccb2496 P 371a36d219164974abdf848841528152 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:09.021154 19625 consensus_queue.cc:260] T 8bd3e1c06b89426da57e2b259e7983af P d870164263bb44b6b780a557daea3656 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:09.022311 19807 ts_tablet_manager.cc:1434] T 3ae5472cba1e4613abe4cda42ccb2496 P 371a36d219164974abdf848841528152: Time spent starting tablet: real 0.031s user 0.004s sys 0.001s
I20260130 02:58:09.023177 19807 tablet_bootstrap.cc:492] T 127a84346b9f473d9e7f739b1e65feb8 P 371a36d219164974abdf848841528152: Bootstrap starting.
I20260130 02:58:09.040123 19375 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7d96e93d95f14af79cfb268626559b1e" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74"
I20260130 02:58:09.040679 19375 raft_consensus.cc:3060] T 7d96e93d95f14af79cfb268626559b1e P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:09.049520 19375 raft_consensus.cc:2468] T 7d96e93d95f14af79cfb268626559b1e P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 371a36d219164974abdf848841528152 in term 1.
I20260130 02:58:09.047411 19526 raft_consensus.cc:3060] T b8c579ff5fa84651ba9125457d927162 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:09.052307 19840 tablet_bootstrap.cc:654] T 04f4ac27800c4ac3a79ae12d35479757 P 78c2c46c7520492db35d51c244fb4f74: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:09.047969 19521 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7d96e93d95f14af79cfb268626559b1e" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656"
I20260130 02:58:09.061179 19159 leader_election.cc:304] T 7d96e93d95f14af79cfb268626559b1e P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 371a36d219164974abdf848841528152, 78c2c46c7520492db35d51c244fb4f74; no voters:
I20260130 02:58:09.063311 19947 raft_consensus.cc:2243] T 3b4908bd1a634ca0907234dd3c39e1c3 P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260130 02:58:09.064172 20243 raft_consensus.cc:2804] T 7d96e93d95f14af79cfb268626559b1e P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Leader election won for term 1
I20260130 02:58:09.064594 20243 raft_consensus.cc:697] T 7d96e93d95f14af79cfb268626559b1e P 371a36d219164974abdf848841528152 [term 1 LEADER]: Becoming Leader. State: Replica: 371a36d219164974abdf848841528152, State: Running, Role: LEADER
I20260130 02:58:09.065730 20243 consensus_queue.cc:237] T 7d96e93d95f14af79cfb268626559b1e P 371a36d219164974abdf848841528152 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:09.068106 19947 raft_consensus.cc:2272] T 3b4908bd1a634ca0907234dd3c39e1c3 P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260130 02:58:09.070900 19947 ts_tablet_manager.cc:1916] T 3b4908bd1a634ca0907234dd3c39e1c3 P d870164263bb44b6b780a557daea3656: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260130 02:58:09.074110 20208 ts_tablet_manager.cc:1929] T 1bbc55bbf2254d8bb6b721d7e8f69d85 P 78c2c46c7520492db35d51c244fb4f74: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 2.2
I20260130 02:58:09.081387 20208 log.cc:1199] T 1bbc55bbf2254d8bb6b721d7e8f69d85 P 78c2c46c7520492db35d51c244fb4f74: Deleting WAL directory at /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-1/wal/wals/1bbc55bbf2254d8bb6b721d7e8f69d85
I20260130 02:58:09.082238 20208 ts_tablet_manager.cc:1950] T 1bbc55bbf2254d8bb6b721d7e8f69d85 P 78c2c46c7520492db35d51c244fb4f74: Deleting consensus metadata
I20260130 02:58:09.084640 18860 catalog_manager.cc:4985] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767): tablet 1bbc55bbf2254d8bb6b721d7e8f69d85 (table default.Table_526677c8ef4144bd9de7a54a5e78e2be [id=b16908b75587459597a978c71756451c]) successfully deleted
I20260130 02:58:09.093158 19807 tablet_bootstrap.cc:654] T 127a84346b9f473d9e7f739b1e65feb8 P 371a36d219164974abdf848841528152: Neither blocks nor log segments found. Creating new log.
W20260130 02:58:09.095674 18860 catalog_manager.cc:4965] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767): delete failed for tablet 78e9bc3e55fe4e4a98e8eb01da59d365 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 78e9bc3e55fe4e4a98e8eb01da59d365 already in progress: creating tablet
I20260130 02:58:09.099469 20341 raft_consensus.cc:493] T 127a84346b9f473d9e7f739b1e65feb8 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:09.099910 20341 raft_consensus.cc:515] T 127a84346b9f473d9e7f739b1e65feb8 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:09.101814 20341 leader_election.cc:290] T 127a84346b9f473d9e7f739b1e65feb8 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), d870164263bb44b6b780a557daea3656 (127.18.100.67:39571)
I20260130 02:58:09.103739 19222 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "127a84346b9f473d9e7f739b1e65feb8" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152" is_pre_election: true
W20260130 02:58:09.105876 19314 leader_election.cc:343] T 127a84346b9f473d9e7f739b1e65feb8 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:09.108202 19840 tablet_bootstrap.cc:492] T 04f4ac27800c4ac3a79ae12d35479757 P 78c2c46c7520492db35d51c244fb4f74: No bootstrap required, opened a new log
I20260130 02:58:09.108781 19840 ts_tablet_manager.cc:1403] T 04f4ac27800c4ac3a79ae12d35479757 P 78c2c46c7520492db35d51c244fb4f74: Time spent bootstrapping tablet: real 0.244s user 0.001s sys 0.010s
W20260130 02:58:09.121762 19159 leader_election.cc:343] T 7d96e93d95f14af79cfb268626559b1e P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:09.122598 19525 raft_consensus.cc:3060] T 3b148691090e4ee581d46f9845a6e49a P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:09.132893 19807 tablet_bootstrap.cc:492] T 127a84346b9f473d9e7f739b1e65feb8 P 371a36d219164974abdf848841528152: No bootstrap required, opened a new log
I20260130 02:58:09.133394 19807 ts_tablet_manager.cc:1403] T 127a84346b9f473d9e7f739b1e65feb8 P 371a36d219164974abdf848841528152: Time spent bootstrapping tablet: real 0.110s user 0.008s sys 0.001s
I20260130 02:58:09.119398 19840 raft_consensus.cc:359] T 04f4ac27800c4ac3a79ae12d35479757 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:09.144017 19840 raft_consensus.cc:385] T 04f4ac27800c4ac3a79ae12d35479757 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:09.145063 19840 raft_consensus.cc:740] T 04f4ac27800c4ac3a79ae12d35479757 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78c2c46c7520492db35d51c244fb4f74, State: Initialized, Role: FOLLOWER
I20260130 02:58:09.146904 19840 consensus_queue.cc:260] T 04f4ac27800c4ac3a79ae12d35479757 P 78c2c46c7520492db35d51c244fb4f74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:09.150075 19807 raft_consensus.cc:359] T 127a84346b9f473d9e7f739b1e65feb8 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:09.164410 19807 raft_consensus.cc:385] T 127a84346b9f473d9e7f739b1e65feb8 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:09.167340 19807 raft_consensus.cc:740] T 127a84346b9f473d9e7f739b1e65feb8 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 371a36d219164974abdf848841528152, State: Initialized, Role: FOLLOWER
I20260130 02:58:09.166885 19840 ts_tablet_manager.cc:1434] T 04f4ac27800c4ac3a79ae12d35479757 P 78c2c46c7520492db35d51c244fb4f74: Time spent starting tablet: real 0.050s user 0.004s sys 0.002s
I20260130 02:58:09.176795 19807 consensus_queue.cc:260] T 127a84346b9f473d9e7f739b1e65feb8 P 371a36d219164974abdf848841528152 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:09.191403 19840 tablet_bootstrap.cc:492] T 43cec19c51ec4f05b63800c07f9534ef P 78c2c46c7520492db35d51c244fb4f74: Bootstrap starting.
I20260130 02:58:09.209120 20327 raft_consensus.cc:493] T 8bd3e1c06b89426da57e2b259e7983af P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:09.209589 20327 raft_consensus.cc:515] T 8bd3e1c06b89426da57e2b259e7983af P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:09.216003 19807 ts_tablet_manager.cc:1434] T 127a84346b9f473d9e7f739b1e65feb8 P 371a36d219164974abdf848841528152: Time spent starting tablet: real 0.076s user 0.004s sys 0.002s
I20260130 02:58:09.237876 19840 tablet_bootstrap.cc:654] T 43cec19c51ec4f05b63800c07f9534ef P 78c2c46c7520492db35d51c244fb4f74: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:09.106739 19521 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "127a84346b9f473d9e7f739b1e65feb8" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
W20260130 02:58:09.251927 19313 leader_election.cc:343] T 127a84346b9f473d9e7f739b1e65feb8 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:09.252373 19313 leader_election.cc:304] T 127a84346b9f473d9e7f739b1e65feb8 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78c2c46c7520492db35d51c244fb4f74; no voters: 371a36d219164974abdf848841528152, d870164263bb44b6b780a557daea3656
I20260130 02:58:09.253037 20341 raft_consensus.cc:2749] T 127a84346b9f473d9e7f739b1e65feb8 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260130 02:58:09.256251 20327 leader_election.cc:290] T 8bd3e1c06b89426da57e2b259e7983af P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767)
I20260130 02:58:09.258148 19625 ts_tablet_manager.cc:1434] T 8bd3e1c06b89426da57e2b259e7983af P d870164263bb44b6b780a557daea3656: Time spent starting tablet: real 0.262s user 0.007s sys 0.000s
W20260130 02:58:09.258455 19463 leader_election.cc:343] T 8bd3e1c06b89426da57e2b259e7983af P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Not found: Tablet not found: 8bd3e1c06b89426da57e2b259e7983af
I20260130 02:58:09.261296 19625 tablet_bootstrap.cc:492] T f54addb1a4304ce69c711ae0cc3b452c P d870164263bb44b6b780a557daea3656: Bootstrap starting.
I20260130 02:58:09.264578 19840 tablet_bootstrap.cc:492] T 43cec19c51ec4f05b63800c07f9534ef P 78c2c46c7520492db35d51c244fb4f74: No bootstrap required, opened a new log
I20260130 02:58:09.266099 19840 ts_tablet_manager.cc:1403] T 43cec19c51ec4f05b63800c07f9534ef P 78c2c46c7520492db35d51c244fb4f74: Time spent bootstrapping tablet: real 0.075s user 0.015s sys 0.002s
W20260130 02:58:09.268046 19462 leader_election.cc:343] T 8bd3e1c06b89426da57e2b259e7983af P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767): Not found: Tablet not found: 8bd3e1c06b89426da57e2b259e7983af
I20260130 02:58:09.268465 19462 leader_election.cc:304] T 8bd3e1c06b89426da57e2b259e7983af P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d870164263bb44b6b780a557daea3656; no voters: 371a36d219164974abdf848841528152, 78c2c46c7520492db35d51c244fb4f74
I20260130 02:58:09.270218 19840 raft_consensus.cc:359] T 43cec19c51ec4f05b63800c07f9534ef P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:09.272723 19840 raft_consensus.cc:385] T 43cec19c51ec4f05b63800c07f9534ef P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:09.273489 19840 raft_consensus.cc:740] T 43cec19c51ec4f05b63800c07f9534ef P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78c2c46c7520492db35d51c244fb4f74, State: Initialized, Role: FOLLOWER
I20260130 02:58:09.274822 19840 consensus_queue.cc:260] T 43cec19c51ec4f05b63800c07f9534ef P 78c2c46c7520492db35d51c244fb4f74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:09.275054 20244 consensus_queue.cc:1048] T 54be22352e404ff9b798a075693f7f21 P 371a36d219164974abdf848841528152 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260130 02:58:09.283231 19807 tablet_bootstrap.cc:492] T 8e57ae31ab3941f789ac69627384e3b0 P 371a36d219164974abdf848841528152: Bootstrap starting.
I20260130 02:58:09.292146 19807 tablet_bootstrap.cc:654] T 8e57ae31ab3941f789ac69627384e3b0 P 371a36d219164974abdf848841528152: Neither blocks nor log segments found. Creating new log.
W20260130 02:58:09.308454 18873 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 274530359b174ff79f046c2d4a6afa76 consensus state has no leader
I20260130 02:58:09.311491 19807 tablet_bootstrap.cc:492] T 8e57ae31ab3941f789ac69627384e3b0 P 371a36d219164974abdf848841528152: No bootstrap required, opened a new log
I20260130 02:58:09.311908 19807 ts_tablet_manager.cc:1403] T 8e57ae31ab3941f789ac69627384e3b0 P 371a36d219164974abdf848841528152: Time spent bootstrapping tablet: real 0.029s user 0.010s sys 0.002s
I20260130 02:58:09.315346 19840 ts_tablet_manager.cc:1434] T 43cec19c51ec4f05b63800c07f9534ef P 78c2c46c7520492db35d51c244fb4f74: Time spent starting tablet: real 0.048s user 0.005s sys 0.000s
I20260130 02:58:09.318408 19840 tablet_bootstrap.cc:492] T dc09a0f0cecf49738bab04596944c0f3 P 78c2c46c7520492db35d51c244fb4f74: Bootstrap starting.
I20260130 02:58:09.326721 20327 raft_consensus.cc:2749] T 8bd3e1c06b89426da57e2b259e7983af P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260130 02:58:09.343145 19521 raft_consensus.cc:3060] T 54be22352e404ff9b798a075693f7f21 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:09.314443 19807 raft_consensus.cc:359] T 8e57ae31ab3941f789ac69627384e3b0 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:09.356199 19807 raft_consensus.cc:385] T 8e57ae31ab3941f789ac69627384e3b0 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:09.356493 19807 raft_consensus.cc:740] T 8e57ae31ab3941f789ac69627384e3b0 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 371a36d219164974abdf848841528152, State: Initialized, Role: FOLLOWER
I20260130 02:58:09.357122 19807 consensus_queue.cc:260] T 8e57ae31ab3941f789ac69627384e3b0 P 371a36d219164974abdf848841528152 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:09.370050 18873 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50284:
name: "default.Table_e8fd7516280f4dc4a73eb1f88fe916a2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260130 02:58:09.371937 18873 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e8fd7516280f4dc4a73eb1f88fe916a2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260130 02:58:09.385746 20341 raft_consensus.cc:493] T 04f4ac27800c4ac3a79ae12d35479757 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:09.386237 20341 raft_consensus.cc:515] T 04f4ac27800c4ac3a79ae12d35479757 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:09.388049 20341 leader_election.cc:290] T 04f4ac27800c4ac3a79ae12d35479757 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), d870164263bb44b6b780a557daea3656 (127.18.100.67:39571)
W20260130 02:58:09.392617 18860 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.100.125:39711, user_credentials={real_user=slave}} blocked reactor thread for 72058.8us
I20260130 02:58:09.392768 20243 consensus_queue.cc:1048] T 54be22352e404ff9b798a075693f7f21 P 371a36d219164974abdf848841528152 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260130 02:58:09.395059 19222 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "04f4ac27800c4ac3a79ae12d35479757" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152" is_pre_election: true
I20260130 02:58:09.395646 19222 raft_consensus.cc:2468] T 04f4ac27800c4ac3a79ae12d35479757 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78c2c46c7520492db35d51c244fb4f74 in term 0.
I20260130 02:58:09.397583 19314 leader_election.cc:304] T 04f4ac27800c4ac3a79ae12d35479757 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 371a36d219164974abdf848841528152, 78c2c46c7520492db35d51c244fb4f74; no voters:
I20260130 02:58:09.398316 20341 raft_consensus.cc:2804] T 04f4ac27800c4ac3a79ae12d35479757 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260130 02:58:09.398656 20341 raft_consensus.cc:493] T 04f4ac27800c4ac3a79ae12d35479757 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260130 02:58:09.399073 20341 raft_consensus.cc:3060] T 04f4ac27800c4ac3a79ae12d35479757 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:09.407755 20330 consensus_queue.cc:1048] T b2ac91da01954c2390b6b7dcc06c394f P 371a36d219164974abdf848841528152 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260130 02:58:09.425315 20180 consensus_queue.cc:1048] T 3593f8b89da24ff9b6cf549d5ac58bc6 P 78c2c46c7520492db35d51c244fb4f74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260130 02:58:09.428072 19496 tablet_service.cc:1505] Processing CreateTablet for tablet e12d130bcdf34965888753f12c1ff848 (DEFAULT_TABLE table=default.Table_a675d7371dcd46b293f893ebe4939e7a [id=c7084428dde74c3ebc46943c44211d7b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:09.429169 19496 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e12d130bcdf34965888753f12c1ff848. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:09.439680 20341 raft_consensus.cc:515] T 04f4ac27800c4ac3a79ae12d35479757 P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:09.450261 20244 raft_consensus.cc:493] T 127a84346b9f473d9e7f739b1e65feb8 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:09.450834 20244 raft_consensus.cc:515] T 127a84346b9f473d9e7f739b1e65feb8 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
W20260130 02:58:09.452821 20356 log.cc:927] Time spent T b8c579ff5fa84651ba9125457d927162 P d870164263bb44b6b780a557daea3656: Append to log took a long time: real 0.062s user 0.000s sys 0.000s
I20260130 02:58:09.455415 19375 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "127a84346b9f473d9e7f739b1e65feb8" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74" is_pre_election: true
I20260130 02:58:09.456288 19375 raft_consensus.cc:2468] T 127a84346b9f473d9e7f739b1e65feb8 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 371a36d219164974abdf848841528152 in term 0.
I20260130 02:58:09.457625 19159 leader_election.cc:304] T 127a84346b9f473d9e7f739b1e65feb8 P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 371a36d219164974abdf848841528152, 78c2c46c7520492db35d51c244fb4f74; no voters:
I20260130 02:58:09.458361 20330 raft_consensus.cc:2804] T 127a84346b9f473d9e7f739b1e65feb8 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260130 02:58:09.458745 20330 raft_consensus.cc:493] T 127a84346b9f473d9e7f739b1e65feb8 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260130 02:58:09.459127 20330 raft_consensus.cc:3060] T 127a84346b9f473d9e7f739b1e65feb8 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:09.459726 19528 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "127a84346b9f473d9e7f739b1e65feb8" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
I20260130 02:58:09.465865 19201 tablet_service.cc:1505] Processing CreateTablet for tablet e12d130bcdf34965888753f12c1ff848 (DEFAULT_TABLE table=default.Table_a675d7371dcd46b293f893ebe4939e7a [id=c7084428dde74c3ebc46943c44211d7b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:09.466866 19201 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e12d130bcdf34965888753f12c1ff848. 1 dirs total, 0 dirs full, 0 dirs failed
W20260130 02:58:09.466893 19159 leader_election.cc:343] T 127a84346b9f473d9e7f739b1e65feb8 P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:09.471019 19358 tablet_service.cc:1505] Processing CreateTablet for tablet e12d130bcdf34965888753f12c1ff848 (DEFAULT_TABLE table=default.Table_a675d7371dcd46b293f893ebe4939e7a [id=c7084428dde74c3ebc46943c44211d7b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:09.471987 19358 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e12d130bcdf34965888753f12c1ff848. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:09.474973 20313 raft_consensus.cc:493] T 3b148691090e4ee581d46f9845a6e49a P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:09.475422 20313 raft_consensus.cc:515] T 3b148691090e4ee581d46f9845a6e49a P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:09.477751 20243 raft_consensus.cc:493] T b89ed298e3dc49d789d166c613ad7fb8 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:09.478197 20243 raft_consensus.cc:515] T b89ed298e3dc49d789d166c613ad7fb8 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:09.478837 20313 leader_election.cc:290] T 3b148691090e4ee581d46f9845a6e49a P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767)
I20260130 02:58:09.480032 20243 leader_election.cc:290] T b89ed298e3dc49d789d166c613ad7fb8 P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767), d870164263bb44b6b780a557daea3656 (127.18.100.67:39571)
I20260130 02:58:09.481360 19223 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3b148691090e4ee581d46f9845a6e49a" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152" is_pre_election: true
I20260130 02:58:09.484839 19625 tablet_bootstrap.cc:654] T f54addb1a4304ce69c711ae0cc3b452c P d870164263bb44b6b780a557daea3656: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:09.488960 19501 tablet_service.cc:1505] Processing CreateTablet for tablet 3aa76993beb94b12bc706a76db26a03b (DEFAULT_TABLE table=default.Table_477fb92965284222aa1ecaa0107d1ec2 [id=ad82881a19e748edb569e74d692bb2bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:09.490005 19501 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3aa76993beb94b12bc706a76db26a03b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:09.492491 19807 ts_tablet_manager.cc:1434] T 8e57ae31ab3941f789ac69627384e3b0 P 371a36d219164974abdf848841528152: Time spent starting tablet: real 0.180s user 0.003s sys 0.003s
I20260130 02:58:09.493414 19807 tablet_bootstrap.cc:492] T 43cec19c51ec4f05b63800c07f9534ef P 371a36d219164974abdf848841528152: Bootstrap starting.
I20260130 02:58:09.496264 19223 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "04f4ac27800c4ac3a79ae12d35479757" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152"
I20260130 02:58:09.496742 19223 raft_consensus.cc:3060] T 04f4ac27800c4ac3a79ae12d35479757 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:09.503257 20341 leader_election.cc:290] T 04f4ac27800c4ac3a79ae12d35479757 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 election: Requested vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), d870164263bb44b6b780a557daea3656 (127.18.100.67:39571)
I20260130 02:58:09.504770 19223 raft_consensus.cc:2468] T 04f4ac27800c4ac3a79ae12d35479757 P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78c2c46c7520492db35d51c244fb4f74 in term 1.
I20260130 02:58:09.514531 20243 consensus_queue.cc:1048] T 3b148691090e4ee581d46f9845a6e49a P 371a36d219164974abdf848841528152 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260130 02:58:09.518499 19522 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "04f4ac27800c4ac3a79ae12d35479757" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
I20260130 02:58:09.505893 19314 leader_election.cc:304] T 04f4ac27800c4ac3a79ae12d35479757 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 371a36d219164974abdf848841528152, 78c2c46c7520492db35d51c244fb4f74; no voters:
I20260130 02:58:09.510316 19380 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b89ed298e3dc49d789d166c613ad7fb8" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74" is_pre_election: true
I20260130 02:58:09.525105 19380 raft_consensus.cc:2468] T b89ed298e3dc49d789d166c613ad7fb8 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 371a36d219164974abdf848841528152 in term 0.
I20260130 02:58:09.525880 20180 raft_consensus.cc:2804] T 04f4ac27800c4ac3a79ae12d35479757 P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Leader election won for term 1
I20260130 02:58:09.536490 19380 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3b148691090e4ee581d46f9845a6e49a" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74" is_pre_election: true
I20260130 02:58:09.511489 19840 tablet_bootstrap.cc:700] T dc09a0f0cecf49738bab04596944c0f3 P 78c2c46c7520492db35d51c244fb4f74: Time spent opening tablet: real 0.128s user 0.001s sys 0.000s
I20260130 02:58:09.541574 19840 tablet_bootstrap.cc:654] T dc09a0f0cecf49738bab04596944c0f3 P 78c2c46c7520492db35d51c244fb4f74: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:09.545634 19201 tablet_service.cc:1505] Processing CreateTablet for tablet 3aa76993beb94b12bc706a76db26a03b (DEFAULT_TABLE table=default.Table_477fb92965284222aa1ecaa0107d1ec2 [id=ad82881a19e748edb569e74d692bb2bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:09.546698 19201 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3aa76993beb94b12bc706a76db26a03b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:09.547737 20244 leader_election.cc:290] T 127a84346b9f473d9e7f739b1e65feb8 P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767), d870164263bb44b6b780a557daea3656 (127.18.100.67:39571)
I20260130 02:58:09.548079 19205 tablet_service.cc:1505] Processing CreateTablet for tablet 13f0ace4d9de43abab08eca32cdb30c1 (DEFAULT_TABLE table=default.Table_9d0cd9756483406081c8bdb18565a42f [id=5f6488df8ba94ef293b181328cc3b680]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:09.548961 19205 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 13f0ace4d9de43abab08eca32cdb30c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:09.549916 19525 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b89ed298e3dc49d789d166c613ad7fb8" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
I20260130 02:58:09.551803 19355 tablet_service.cc:1505] Processing CreateTablet for tablet 3aa76993beb94b12bc706a76db26a03b (DEFAULT_TABLE table=default.Table_477fb92965284222aa1ecaa0107d1ec2 [id=ad82881a19e748edb569e74d692bb2bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:09.552845 19355 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3aa76993beb94b12bc706a76db26a03b. 1 dirs total, 0 dirs full, 0 dirs failed
W20260130 02:58:09.564342 19159 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.100.67:39571, user_credentials={real_user=slave}} blocked reactor thread for 49988.1us
I20260130 02:58:09.566357 19159 leader_election.cc:304] T b89ed298e3dc49d789d166c613ad7fb8 P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 371a36d219164974abdf848841528152, 78c2c46c7520492db35d51c244fb4f74; no voters:
W20260130 02:58:09.567250 19159 leader_election.cc:343] T b89ed298e3dc49d789d166c613ad7fb8 P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:09.567541 20323 raft_consensus.cc:2804] T b89ed298e3dc49d789d166c613ad7fb8 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260130 02:58:09.567905 20323 raft_consensus.cc:493] T b89ed298e3dc49d789d166c613ad7fb8 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260130 02:58:09.568218 20323 raft_consensus.cc:3060] T b89ed298e3dc49d789d166c613ad7fb8 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:09.426299 18874 catalog_manager.cc:5746] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767) does not have the latest schema for tablet c7c621b99deb457998c44ab1644d3939 (table default.Table_3413dae2efac41488448e6e5975cf710 [id=7a08fa3986d0410d81de0235d2fb0f92]). Expected version 1 got 0
I20260130 02:58:09.571038 18874 catalog_manager.cc:5746] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767) does not have the latest schema for tablet 3b148691090e4ee581d46f9845a6e49a (table default.Table_8edea1fae34d4d18b54b0f801a145225 [id=2009496525c94d7f8ee6ce59694b652d]). Expected version 2 got 0
I20260130 02:58:09.571555 18874 catalog_manager.cc:5746] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767) does not have the latest schema for tablet 91f954f3ef4244c9a575a7421a7e75eb (table default.Table_9f83f87711d94e8394abe1925ac84ad1 [id=db23c1be42c44b6f9eb774f14d2b8fd6]). Expected version 1 got 0
I20260130 02:58:09.572002 18874 catalog_manager.cc:5746] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767) does not have the latest schema for tablet 54be22352e404ff9b798a075693f7f21 (table default.Table_3a858a3cb886427cb8268f42c953ad4c [id=17150cb73e374ff9a1e92c43e0c3f2c1]). Expected version 1 got 0
I20260130 02:58:09.574854 20323 raft_consensus.cc:515] T b89ed298e3dc49d789d166c613ad7fb8 P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:09.576639 20323 leader_election.cc:290] T b89ed298e3dc49d789d166c613ad7fb8 P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 election: Requested vote from peers 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767), d870164263bb44b6b780a557daea3656 (127.18.100.67:39571)
W20260130 02:58:09.579957 19313 leader_election.cc:343] T 04f4ac27800c4ac3a79ae12d35479757 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:09.581180 19462 leader_election.cc:304] T 3b148691090e4ee581d46f9845a6e49a P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d870164263bb44b6b780a557daea3656; no voters: 371a36d219164974abdf848841528152, 78c2c46c7520492db35d51c244fb4f74
W20260130 02:58:09.582585 19462 consensus_peers.cc:597] T 1bbc55bbf2254d8bb6b721d7e8f69d85 P d870164263bb44b6b780a557daea3656 -> Peer 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767): Couldn't send request to peer 78c2c46c7520492db35d51c244fb4f74. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1bbc55bbf2254d8bb6b721d7e8f69d85. This is attempt 1: this message will repeat every 5th retry.
I20260130 02:58:09.591691 19356 tablet_service.cc:1505] Processing CreateTablet for tablet 13f0ace4d9de43abab08eca32cdb30c1 (DEFAULT_TABLE table=default.Table_9d0cd9756483406081c8bdb18565a42f [id=5f6488df8ba94ef293b181328cc3b680]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:09.592722 19356 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 13f0ace4d9de43abab08eca32cdb30c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:09.595388 19201 tablet_service.cc:1505] Processing CreateTablet for tablet af0f90071a334016aa17fae2e3a97fc8 (DEFAULT_TABLE table=default.Table_22d10332a83f4700ac21892bab67f46b [id=da3a9cb1f6d5419abc1afc93659b5324]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:09.595801 19351 tablet_service.cc:1505] Processing CreateTablet for tablet af0f90071a334016aa17fae2e3a97fc8 (DEFAULT_TABLE table=default.Table_22d10332a83f4700ac21892bab67f46b [id=da3a9cb1f6d5419abc1afc93659b5324]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:09.596792 19351 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet af0f90071a334016aa17fae2e3a97fc8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:09.601011 20180 raft_consensus.cc:697] T 04f4ac27800c4ac3a79ae12d35479757 P 78c2c46c7520492db35d51c244fb4f74 [term 1 LEADER]: Becoming Leader. State: Replica: 78c2c46c7520492db35d51c244fb4f74, State: Running, Role: LEADER
I20260130 02:58:09.601891 20180 consensus_queue.cc:237] T 04f4ac27800c4ac3a79ae12d35479757 P 78c2c46c7520492db35d51c244fb4f74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:09.578202 20330 raft_consensus.cc:515] T 127a84346b9f473d9e7f739b1e65feb8 P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:09.609077 20330 leader_election.cc:290] T 127a84346b9f473d9e7f739b1e65feb8 P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 election: Requested vote from peers 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767), d870164263bb44b6b780a557daea3656 (127.18.100.67:39571)
I20260130 02:58:09.621371 18875 catalog_manager.cc:5941] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4: Sending DeleteTablet for 3 replicas of tablet d27ca7dd87f44102b7680e1f4a88954a
I20260130 02:58:09.622613 18875 catalog_manager.cc:6724] ReplaceTablet: tablet d27ca7dd87f44102b7680e1f4a88954a deleted and replaced by tablet d480bf18140245e3870a62582d5a8318
I20260130 02:58:09.627105 19199 tablet_service.cc:1552] Processing DeleteTablet for tablet d27ca7dd87f44102b7680e1f4a88954a with delete_type TABLET_DATA_DELETED (replaced by tablet d480bf18140245e3870a62582d5a8318) from {username='slave'} at 127.0.0.1:42870
I20260130 02:58:09.635226 19350 tablet_service.cc:1552] Processing DeleteTablet for tablet d27ca7dd87f44102b7680e1f4a88954a with delete_type TABLET_DATA_DELETED (replaced by tablet d480bf18140245e3870a62582d5a8318) from {username='slave'} at 127.0.0.1:60110
I20260130 02:58:09.635900 19380 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b89ed298e3dc49d789d166c613ad7fb8" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74"
I20260130 02:58:09.636394 19380 raft_consensus.cc:3060] T b89ed298e3dc49d789d166c613ad7fb8 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:09.638258 19947 ts_tablet_manager.cc:1929] T 3b4908bd1a634ca0907234dd3c39e1c3 P d870164263bb44b6b780a557daea3656: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260130 02:58:09.638643 19947 log.cc:1199] T 3b4908bd1a634ca0907234dd3c39e1c3 P d870164263bb44b6b780a557daea3656: Deleting WAL directory at /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-2/wal/wals/3b4908bd1a634ca0907234dd3c39e1c3
I20260130 02:58:09.639271 19947 ts_tablet_manager.cc:1950] T 3b4908bd1a634ca0907234dd3c39e1c3 P d870164263bb44b6b780a557daea3656: Deleting consensus metadata
I20260130 02:58:09.573474 18874 catalog_manager.cc:5654] T b8c579ff5fa84651ba9125457d927162 P 78c2c46c7520492db35d51c244fb4f74 reported cstate change: term changed from 0 to 1, leader changed from <none> to 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66). New cstate: current_term: 1 leader_uuid: "78c2c46c7520492db35d51c244fb4f74" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } health_report { overall_health: HEALTHY } } }
I20260130 02:58:09.641247 19575 heartbeater.cc:344] Connected to a master server at 127.18.100.126:38191
I20260130 02:58:09.642088 19508 tablet_service.cc:1552] Processing DeleteTablet for tablet d27ca7dd87f44102b7680e1f4a88954a with delete_type TABLET_DATA_DELETED (replaced by tablet d480bf18140245e3870a62582d5a8318) from {username='slave'} at 127.0.0.1:59068
I20260130 02:58:09.644121 19523 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b89ed298e3dc49d789d166c613ad7fb8" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656"
I20260130 02:58:09.644724 19524 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "127a84346b9f473d9e7f739b1e65feb8" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656"
I20260130 02:58:09.650545 19523 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "04f4ac27800c4ac3a79ae12d35479757" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656"
I20260130 02:58:09.638060 20371 raft_consensus.cc:2749] T 3b148691090e4ee581d46f9845a6e49a P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260130 02:58:09.656751 19947 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59068 (request call id 212) took 1341 ms. Trace:
I20260130 02:58:09.662801 18873 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50284:
name: "default.Table_f3e214a955f6447482099175b92ecb5c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260130 02:58:09.662945 18875 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50284:
name: "default.Table_12c2fe20cc3e45d29b7be37f900f68a9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260130 02:58:09.664326 18873 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f3e214a955f6447482099175b92ecb5c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260130 02:58:09.664341 18875 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_12c2fe20cc3e45d29b7be37f900f68a9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260130 02:58:09.658257 19947 rpcz_store.cc:276] 0130 02:58:08.316404 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:58:08.316597 (+ 193us) service_pool.cc:225] Handling call
0130 02:58:08.870245 (+553648us) ts_tablet_manager.cc:1151] Deleting tablet 3b4908bd1a634ca0907234dd3c39e1c3
0130 02:58:08.870287 (+ 42us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0130 02:58:09.384210 (+513923us) tablet_metadata.cc:641] Metadata flushed
0130 02:58:09.638024 (+253814us) tablet_metadata.cc:641] Metadata flushed
0130 02:58:09.656694 (+ 18670us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":157652,"tablet-delete.queue_time_us":553073}
I20260130 02:58:09.669809 18860 catalog_manager.cc:4985] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): tablet 3b4908bd1a634ca0907234dd3c39e1c3 (table default.Table_0ad5d205dfea4344a759522f45a55475 [id=5b7d0747e1bd438ea7e2f10abd339c2c]) successfully deleted
I20260130 02:58:09.670179 19501 tablet_service.cc:1505] Processing CreateTablet for tablet 13f0ace4d9de43abab08eca32cdb30c1 (DEFAULT_TABLE table=default.Table_9d0cd9756483406081c8bdb18565a42f [id=5f6488df8ba94ef293b181328cc3b680]), partition=RANGE (key) PARTITION UNBOUNDED
W20260130 02:58:09.672490 18860 catalog_manager.cc:4965] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): delete failed for tablet e2aaaf00f53d418b87d5c7af286e4d84 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e2aaaf00f53d418b87d5c7af286e4d84 already in progress: creating tablet
I20260130 02:58:09.681783 19947 tablet_replica.cc:333] T 1bbc55bbf2254d8bb6b721d7e8f69d85 P d870164263bb44b6b780a557daea3656: stopping tablet replica
I20260130 02:58:09.688550 19501 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 13f0ace4d9de43abab08eca32cdb30c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:09.682821 19947 raft_consensus.cc:2243] T 1bbc55bbf2254d8bb6b721d7e8f69d85 P d870164263bb44b6b780a557daea3656 [term 2 LEADER]: Raft consensus shutting down.
I20260130 02:58:09.690742 19947 raft_consensus.cc:2272] T 1bbc55bbf2254d8bb6b721d7e8f69d85 P d870164263bb44b6b780a557daea3656 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260130 02:58:09.702518 19201 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet af0f90071a334016aa17fae2e3a97fc8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:09.707628 19807 tablet_bootstrap.cc:654] T 43cec19c51ec4f05b63800c07f9534ef P 371a36d219164974abdf848841528152: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:09.715801 19508 tablet_service.cc:1505] Processing CreateTablet for tablet af0f90071a334016aa17fae2e3a97fc8 (DEFAULT_TABLE table=default.Table_22d10332a83f4700ac21892bab67f46b [id=da3a9cb1f6d5419abc1afc93659b5324]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:09.716800 19508 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet af0f90071a334016aa17fae2e3a97fc8. 1 dirs total, 0 dirs full, 0 dirs failed
W20260130 02:58:09.718159 18861 catalog_manager.cc:4965] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021): delete failed for tablet d27ca7dd87f44102b7680e1f4a88954a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d27ca7dd87f44102b7680e1f4a88954a already in progress: creating tablet
W20260130 02:58:09.676105 19313 leader_election.cc:343] T 04f4ac27800c4ac3a79ae12d35479757 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:09.761490 18867 catalog_manager.cc:5654] T 341041cb0be2456e88403367c0814704 P d870164263bb44b6b780a557daea3656 reported cstate change: term changed from 0 to 1, leader changed from <none> to d870164263bb44b6b780a557daea3656 (127.18.100.67). New cstate: current_term: 1 leader_uuid: "d870164263bb44b6b780a557daea3656" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } health_report { overall_health: HEALTHY } } }
I20260130 02:58:09.762837 19947 ts_tablet_manager.cc:1916] T 1bbc55bbf2254d8bb6b721d7e8f69d85 P d870164263bb44b6b780a557daea3656: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260130 02:58:09.764087 19313 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.100.67:39571, user_credentials={real_user=slave}} blocked reactor thread for 88003.9us
W20260130 02:58:09.764993 19313 consensus_peers.cc:597] T 3593f8b89da24ff9b6cf549d5ac58bc6 P 78c2c46c7520492db35d51c244fb4f74 -> Peer d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): Couldn't send request to peer d870164263bb44b6b780a557daea3656. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20260130 02:58:09.765513 19313 consensus_peers.cc:597] T f54addb1a4304ce69c711ae0cc3b452c P 78c2c46c7520492db35d51c244fb4f74 -> Peer d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): Couldn't send request to peer d870164263bb44b6b780a557daea3656. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20260130 02:58:09.792037 19201 tablet_service.cc:1505] Processing CreateTablet for tablet 320dbe98a32a43b9899bbeed80b20f55 (DEFAULT_TABLE table=default.Table_cda216a3bfc14ec0862b998bedd5b300 [id=35978c9d901c47eb81e565e37493b1ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:09.793010 19201 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 320dbe98a32a43b9899bbeed80b20f55. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:09.796659 19502 tablet_service.cc:1505] Processing CreateTablet for tablet 320dbe98a32a43b9899bbeed80b20f55 (DEFAULT_TABLE table=default.Table_cda216a3bfc14ec0862b998bedd5b300 [id=35978c9d901c47eb81e565e37493b1ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:09.792311 19625 tablet_bootstrap.cc:492] T f54addb1a4304ce69c711ae0cc3b452c P d870164263bb44b6b780a557daea3656: No bootstrap required, opened a new log
I20260130 02:58:09.797631 19625 ts_tablet_manager.cc:1403] T f54addb1a4304ce69c711ae0cc3b452c P d870164263bb44b6b780a557daea3656: Time spent bootstrapping tablet: real 0.537s user 0.009s sys 0.003s
I20260130 02:58:09.797694 19502 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 320dbe98a32a43b9899bbeed80b20f55. 1 dirs total, 0 dirs full, 0 dirs failed
W20260130 02:58:09.807616 19159 consensus_peers.cc:597] T b2ac91da01954c2390b6b7dcc06c394f P 371a36d219164974abdf848841528152 -> Peer d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): Couldn't send request to peer d870164263bb44b6b780a557daea3656. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260130 02:58:09.813629 19505 tablet_service.cc:1505] Processing CreateTablet for tablet 12c3f3ec523040e7b7cd356e3da57e0d (DEFAULT_TABLE table=default.Table_81553b46d7804fd99ef36f431793b001 [id=cdfc1cfcc48f41f8a7a13f865174c3d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:09.814621 19505 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 12c3f3ec523040e7b7cd356e3da57e0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:09.823333 20374 raft_consensus.cc:493] T 43cec19c51ec4f05b63800c07f9534ef P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:09.823770 20374 raft_consensus.cc:515] T 43cec19c51ec4f05b63800c07f9534ef P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:09.842715 20244 raft_consensus.cc:493] T 3ae5472cba1e4613abe4cda42ccb2496 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:09.843235 18867 catalog_manager.cc:5746] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571) does not have the latest schema for tablet 91f954f3ef4244c9a575a7421a7e75eb (table default.Table_9f83f87711d94e8394abe1925ac84ad1 [id=db23c1be42c44b6f9eb774f14d2b8fd6]). Expected version 1 got 0
I20260130 02:58:09.843267 20244 raft_consensus.cc:515] T 3ae5472cba1e4613abe4cda42ccb2496 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:09.816995 19378 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "127a84346b9f473d9e7f739b1e65feb8" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74"
I20260130 02:58:09.844946 20244 leader_election.cc:290] T 3ae5472cba1e4613abe4cda42ccb2496 P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767), d870164263bb44b6b780a557daea3656 (127.18.100.67:39571)
I20260130 02:58:09.848989 19380 raft_consensus.cc:2468] T b89ed298e3dc49d789d166c613ad7fb8 P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 371a36d219164974abdf848841528152 in term 1.
I20260130 02:58:09.825536 20374 leader_election.cc:290] T 43cec19c51ec4f05b63800c07f9534ef P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), d870164263bb44b6b780a557daea3656 (127.18.100.67:39571)
W20260130 02:58:09.866606 18860 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.18.100.124:34629: connect: Connection refused (error 111) [suppressed 8 similar messages]
I20260130 02:58:09.872846 19525 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "43cec19c51ec4f05b63800c07f9534ef" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
W20260130 02:58:09.873768 19313 consensus_peers.cc:597] T ee7c8ff92ca9465fa18be6eae63d78ca P 78c2c46c7520492db35d51c244fb4f74 -> Peer d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): Couldn't send request to peer d870164263bb44b6b780a557daea3656. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260130 02:58:09.875979 19840 tablet_bootstrap.cc:492] T dc09a0f0cecf49738bab04596944c0f3 P 78c2c46c7520492db35d51c244fb4f74: No bootstrap required, opened a new log
I20260130 02:58:09.876451 19840 ts_tablet_manager.cc:1403] T dc09a0f0cecf49738bab04596944c0f3 P 78c2c46c7520492db35d51c244fb4f74: Time spent bootstrapping tablet: real 0.558s user 0.011s sys 0.000s
W20260130 02:58:09.876385 19159 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.100.67:39571, user_credentials={real_user=slave}} blocked reactor thread for 68226.4us
W20260130 02:58:09.876983 19159 leader_election.cc:343] T b89ed298e3dc49d789d166c613ad7fb8 P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): Illegal state: must be running to vote when last-logged opid is not known
W20260130 02:58:09.877501 19159 leader_election.cc:343] T 127a84346b9f473d9e7f739b1e65feb8 P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:09.880457 19807 tablet_bootstrap.cc:492] T 43cec19c51ec4f05b63800c07f9534ef P 371a36d219164974abdf848841528152: No bootstrap required, opened a new log
I20260130 02:58:09.880888 19807 ts_tablet_manager.cc:1403] T 43cec19c51ec4f05b63800c07f9534ef P 371a36d219164974abdf848841528152: Time spent bootstrapping tablet: real 0.388s user 0.010s sys 0.000s
W20260130 02:58:09.884586 20377 log.cc:927] Time spent T 04f4ac27800c4ac3a79ae12d35479757 P 78c2c46c7520492db35d51c244fb4f74: Append to log took a long time: real 0.143s user 0.001s sys 0.000s
I20260130 02:58:09.856081 19378 raft_consensus.cc:3060] T 127a84346b9f473d9e7f739b1e65feb8 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:09.905364 19378 raft_consensus.cc:2468] T 127a84346b9f473d9e7f739b1e65feb8 P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 371a36d219164974abdf848841528152 in term 1.
I20260130 02:58:09.906600 19840 raft_consensus.cc:359] T dc09a0f0cecf49738bab04596944c0f3 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:09.907361 19840 raft_consensus.cc:385] T dc09a0f0cecf49738bab04596944c0f3 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:09.907645 19840 raft_consensus.cc:740] T dc09a0f0cecf49738bab04596944c0f3 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78c2c46c7520492db35d51c244fb4f74, State: Initialized, Role: FOLLOWER
I20260130 02:58:09.908257 19840 consensus_queue.cc:260] T dc09a0f0cecf49738bab04596944c0f3 P 78c2c46c7520492db35d51c244fb4f74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:09.910689 19840 ts_tablet_manager.cc:1434] T dc09a0f0cecf49738bab04596944c0f3 P 78c2c46c7520492db35d51c244fb4f74: Time spent starting tablet: real 0.032s user 0.006s sys 0.000s
I20260130 02:58:09.933053 19201 tablet_service.cc:1505] Processing CreateTablet for tablet 12c3f3ec523040e7b7cd356e3da57e0d (DEFAULT_TABLE table=default.Table_81553b46d7804fd99ef36f431793b001 [id=cdfc1cfcc48f41f8a7a13f865174c3d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:09.941212 19201 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 12c3f3ec523040e7b7cd356e3da57e0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:09.879848 19625 raft_consensus.cc:359] T f54addb1a4304ce69c711ae0cc3b452c P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:09.943109 19625 raft_consensus.cc:385] T f54addb1a4304ce69c711ae0cc3b452c P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:09.943601 19625 raft_consensus.cc:740] T f54addb1a4304ce69c711ae0cc3b452c P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d870164263bb44b6b780a557daea3656, State: Initialized, Role: FOLLOWER
I20260130 02:58:09.944356 19625 consensus_queue.cc:260] T f54addb1a4304ce69c711ae0cc3b452c P d870164263bb44b6b780a557daea3656 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
W20260130 02:58:09.952908 19159 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.100.67:39571, user_credentials={real_user=slave}} blocked reactor thread for 74945.1us
I20260130 02:58:09.954010 19807 raft_consensus.cc:359] T 43cec19c51ec4f05b63800c07f9534ef P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:09.954739 19807 raft_consensus.cc:385] T 43cec19c51ec4f05b63800c07f9534ef P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:09.954996 19807 raft_consensus.cc:740] T 43cec19c51ec4f05b63800c07f9534ef P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 371a36d219164974abdf848841528152, State: Initialized, Role: FOLLOWER
I20260130 02:58:09.955083 19625 ts_tablet_manager.cc:1434] T f54addb1a4304ce69c711ae0cc3b452c P d870164263bb44b6b780a557daea3656: Time spent starting tablet: real 0.157s user 0.004s sys 0.002s
I20260130 02:58:09.955574 19807 consensus_queue.cc:260] T 43cec19c51ec4f05b63800c07f9534ef P 371a36d219164974abdf848841528152 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:09.957722 19807 ts_tablet_manager.cc:1434] T 43cec19c51ec4f05b63800c07f9534ef P 371a36d219164974abdf848841528152: Time spent starting tablet: real 0.077s user 0.006s sys 0.000s
I20260130 02:58:09.958844 19807 tablet_bootstrap.cc:492] T 3b02480d9c6f4328af83f7940716d698 P 371a36d219164974abdf848841528152: Bootstrap starting.
I20260130 02:58:09.964097 19807 tablet_bootstrap.cc:654] T 3b02480d9c6f4328af83f7940716d698 P 371a36d219164974abdf848841528152: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:09.969173 19521 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3ae5472cba1e4613abe4cda42ccb2496" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
W20260130 02:58:09.970402 19314 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.100.65:42021, user_credentials={real_user=slave}} blocked reactor thread for 286702us
W20260130 02:58:09.972394 19313 leader_election.cc:343] T 43cec19c51ec4f05b63800c07f9534ef P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:09.976153 19350 tablet_service.cc:1505] Processing CreateTablet for tablet 320dbe98a32a43b9899bbeed80b20f55 (DEFAULT_TABLE table=default.Table_cda216a3bfc14ec0862b998bedd5b300 [id=35978c9d901c47eb81e565e37493b1ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:09.977640 19350 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 320dbe98a32a43b9899bbeed80b20f55. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:09.979293 19625 tablet_bootstrap.cc:492] T b2ac91da01954c2390b6b7dcc06c394f P d870164263bb44b6b780a557daea3656: Bootstrap starting.
I20260130 02:58:09.975636 19222 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "43cec19c51ec4f05b63800c07f9534ef" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152" is_pre_election: true
I20260130 02:58:09.981169 19222 raft_consensus.cc:2468] T 43cec19c51ec4f05b63800c07f9534ef P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78c2c46c7520492db35d51c244fb4f74 in term 0.
I20260130 02:58:09.982481 19314 leader_election.cc:304] T 43cec19c51ec4f05b63800c07f9534ef P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 371a36d219164974abdf848841528152, 78c2c46c7520492db35d51c244fb4f74; no voters: d870164263bb44b6b780a557daea3656
I20260130 02:58:09.986335 19947 ts_tablet_manager.cc:1929] T 1bbc55bbf2254d8bb6b721d7e8f69d85 P d870164263bb44b6b780a557daea3656: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 2.2
I20260130 02:58:09.989493 19159 leader_election.cc:304] T b89ed298e3dc49d789d166c613ad7fb8 P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 371a36d219164974abdf848841528152, 78c2c46c7520492db35d51c244fb4f74; no voters: d870164263bb44b6b780a557daea3656
I20260130 02:58:09.990064 19947 log.cc:1199] T 1bbc55bbf2254d8bb6b721d7e8f69d85 P d870164263bb44b6b780a557daea3656: Deleting WAL directory at /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-2/wal/wals/1bbc55bbf2254d8bb6b721d7e8f69d85
I20260130 02:58:09.994704 20331 raft_consensus.cc:2804] T b89ed298e3dc49d789d166c613ad7fb8 P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Leader election won for term 1
I20260130 02:58:09.995247 19947 ts_tablet_manager.cc:1950] T 1bbc55bbf2254d8bb6b721d7e8f69d85 P d870164263bb44b6b780a557daea3656: Deleting consensus metadata
I20260130 02:58:09.995153 20331 raft_consensus.cc:697] T b89ed298e3dc49d789d166c613ad7fb8 P 371a36d219164974abdf848841528152 [term 1 LEADER]: Becoming Leader. State: Replica: 371a36d219164974abdf848841528152, State: Running, Role: LEADER
I20260130 02:58:09.995911 20331 consensus_queue.cc:237] T b89ed298e3dc49d789d166c613ad7fb8 P 371a36d219164974abdf848841528152 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:09.996491 19358 tablet_service.cc:1505] Processing CreateTablet for tablet 12c3f3ec523040e7b7cd356e3da57e0d (DEFAULT_TABLE table=default.Table_81553b46d7804fd99ef36f431793b001 [id=cdfc1cfcc48f41f8a7a13f865174c3d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:09.996376 19375 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3ae5472cba1e4613abe4cda42ccb2496" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74" is_pre_election: true
I20260130 02:58:09.997658 19358 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 12c3f3ec523040e7b7cd356e3da57e0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:09.997406 19375 raft_consensus.cc:2468] T 3ae5472cba1e4613abe4cda42ccb2496 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 371a36d219164974abdf848841528152 in term 0.
I20260130 02:58:10.008323 19947 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59068 (request call id 215) took 1034 ms. Trace:
I20260130 02:58:10.017839 19947 rpcz_store.cc:276] 0130 02:58:08.974417 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:58:08.974537 (+ 120us) service_pool.cc:225] Handling call
0130 02:58:09.675520 (+700983us) ts_tablet_manager.cc:1151] Deleting tablet 1bbc55bbf2254d8bb6b721d7e8f69d85
0130 02:58:09.681662 (+ 6142us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0130 02:58:09.873326 (+191664us) tablet_metadata.cc:641] Metadata flushed
0130 02:58:09.986111 (+112785us) tablet_metadata.cc:641] Metadata flushed
0130 02:58:10.008250 (+ 22139us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":685637}
I20260130 02:58:10.016482 19625 tablet_bootstrap.cc:654] T b2ac91da01954c2390b6b7dcc06c394f P d870164263bb44b6b780a557daea3656: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:10.020660 19159 leader_election.cc:304] T 127a84346b9f473d9e7f739b1e65feb8 P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 371a36d219164974abdf848841528152, 78c2c46c7520492db35d51c244fb4f74; no voters: d870164263bb44b6b780a557daea3656
I20260130 02:58:10.022931 20244 raft_consensus.cc:2804] T 127a84346b9f473d9e7f739b1e65feb8 P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Leader election won for term 1
I20260130 02:58:10.023357 20244 raft_consensus.cc:697] T 127a84346b9f473d9e7f739b1e65feb8 P 371a36d219164974abdf848841528152 [term 1 LEADER]: Becoming Leader. State: Replica: 371a36d219164974abdf848841528152, State: Running, Role: LEADER
W20260130 02:58:10.024156 18862 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet 274530359b174ff79f046c2d4a6afa76 (table default.Table_c4f4904b6fac4b9880979cf0314e71c1 [id=c456c6418c93499a9d08f652720854ca], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 274530359b174ff79f046c2d4a6afa76 consensus state has no leader
I20260130 02:58:10.024096 20244 consensus_queue.cc:237] T 127a84346b9f473d9e7f739b1e65feb8 P 371a36d219164974abdf848841528152 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:10.023792 18860 catalog_manager.cc:4985] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): tablet 1bbc55bbf2254d8bb6b721d7e8f69d85 (table default.Table_526677c8ef4144bd9de7a54a5e78e2be [id=b16908b75587459597a978c71756451c]) successfully deleted
W20260130 02:58:10.026450 18860 catalog_manager.cc:4965] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767): delete failed for tablet d27ca7dd87f44102b7680e1f4a88954a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d27ca7dd87f44102b7680e1f4a88954a already in progress: creating tablet
I20260130 02:58:10.028573 19840 tablet_bootstrap.cc:492] T 8e57ae31ab3941f789ac69627384e3b0 P 78c2c46c7520492db35d51c244fb4f74: Bootstrap starting.
I20260130 02:58:10.029784 19947 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59068 (request call id 216) took 1040 ms. Trace:
I20260130 02:58:10.031491 18871 catalog_manager.cc:5654] T 3b148691090e4ee581d46f9845a6e49a P 371a36d219164974abdf848841528152 reported cstate change: term changed from 0 to 1, leader changed from <none> to 371a36d219164974abdf848841528152 (127.18.100.65). New cstate: current_term: 1 leader_uuid: "371a36d219164974abdf848841528152" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } health_report { overall_health: UNKNOWN } } }
I20260130 02:58:10.033332 18871 catalog_manager.cc:5746] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021) does not have the latest schema for tablet 3b148691090e4ee581d46f9845a6e49a (table default.Table_8edea1fae34d4d18b54b0f801a145225 [id=2009496525c94d7f8ee6ce59694b652d]). Expected version 2 got 0
I20260130 02:58:10.034773 19840 tablet_bootstrap.cc:654] T 8e57ae31ab3941f789ac69627384e3b0 P 78c2c46c7520492db35d51c244fb4f74: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:10.035434 18871 catalog_manager.cc:5746] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021) does not have the latest schema for tablet ee7c8ff92ca9465fa18be6eae63d78ca (table default.Table_dd22e9dc63f84d8688c0071f427f0b25 [id=77979d498aa040828de415129cb15a06]). Expected version 1 got 0
I20260130 02:58:10.036950 19159 leader_election.cc:304] T 3ae5472cba1e4613abe4cda42ccb2496 P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 371a36d219164974abdf848841528152, 78c2c46c7520492db35d51c244fb4f74; no voters:
I20260130 02:58:10.042392 20243 raft_consensus.cc:2804] T 3ae5472cba1e4613abe4cda42ccb2496 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260130 02:58:10.042754 20243 raft_consensus.cc:493] T 3ae5472cba1e4613abe4cda42ccb2496 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260130 02:58:10.043028 20243 raft_consensus.cc:3060] T 3ae5472cba1e4613abe4cda42ccb2496 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:10.038347 18871 catalog_manager.cc:5746] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021) does not have the latest schema for tablet 91f954f3ef4244c9a575a7421a7e75eb (table default.Table_9f83f87711d94e8394abe1925ac84ad1 [id=db23c1be42c44b6f9eb774f14d2b8fd6]). Expected version 1 got 0
I20260130 02:58:10.048519 20180 raft_consensus.cc:2804] T 43cec19c51ec4f05b63800c07f9534ef P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260130 02:58:10.048980 20180 raft_consensus.cc:493] T 43cec19c51ec4f05b63800c07f9534ef P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260130 02:58:10.049374 20180 raft_consensus.cc:3060] T 43cec19c51ec4f05b63800c07f9534ef P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:10.030153 19947 rpcz_store.cc:276] 0130 02:58:08.989627 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:58:08.989780 (+ 153us) service_pool.cc:225] Handling call
0130 02:58:10.020939 (+1031159us) ts_tablet_manager.cc:1151] Deleting tablet 78e9bc3e55fe4e4a98e8eb01da59d365
0130 02:58:10.020977 (+ 38us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0130 02:58:10.029716 (+ 8739us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1030658}
W20260130 02:58:10.051097 18860 catalog_manager.cc:4965] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): delete failed for tablet 78e9bc3e55fe4e4a98e8eb01da59d365 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 78e9bc3e55fe4e4a98e8eb01da59d365 already in progress: creating tablet
W20260130 02:58:10.056293 18860 catalog_manager.cc:4965] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): delete failed for tablet d27ca7dd87f44102b7680e1f4a88954a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d27ca7dd87f44102b7680e1f4a88954a already in progress: creating tablet
W20260130 02:58:10.065757 19159 leader_election.cc:343] T 3ae5472cba1e4613abe4cda42ccb2496 P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:10.069594 18871 catalog_manager.cc:5654] T 54be22352e404ff9b798a075693f7f21 P 371a36d219164974abdf848841528152 reported cstate change: term changed from 0 to 1, leader changed from <none> to 371a36d219164974abdf848841528152 (127.18.100.65). New cstate: current_term: 1 leader_uuid: "371a36d219164974abdf848841528152" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } health_report { overall_health: UNKNOWN } } }
I20260130 02:58:10.078879 18871 catalog_manager.cc:5746] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021) does not have the latest schema for tablet 54be22352e404ff9b798a075693f7f21 (table default.Table_3a858a3cb886427cb8268f42c953ad4c [id=17150cb73e374ff9a1e92c43e0c3f2c1]). Expected version 1 got 0
I20260130 02:58:10.083091 20323 consensus_queue.cc:1048] T 7d96e93d95f14af79cfb268626559b1e P 371a36d219164974abdf848841528152 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260130 02:58:10.086331 20243 raft_consensus.cc:515] T 3ae5472cba1e4613abe4cda42ccb2496 P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:10.089485 20243 leader_election.cc:290] T 3ae5472cba1e4613abe4cda42ccb2496 P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 election: Requested vote from peers 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767), d870164263bb44b6b780a557daea3656 (127.18.100.67:39571)
I20260130 02:58:10.091336 19376 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3ae5472cba1e4613abe4cda42ccb2496" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74"
I20260130 02:58:10.091895 19376 raft_consensus.cc:3060] T 3ae5472cba1e4613abe4cda42ccb2496 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:10.104027 19521 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3ae5472cba1e4613abe4cda42ccb2496" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656"
W20260130 02:58:10.105494 19159 leader_election.cc:343] T 3ae5472cba1e4613abe4cda42ccb2496 P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:10.107347 19625 tablet_bootstrap.cc:492] T b2ac91da01954c2390b6b7dcc06c394f P d870164263bb44b6b780a557daea3656: No bootstrap required, opened a new log
I20260130 02:58:10.107748 19625 ts_tablet_manager.cc:1403] T b2ac91da01954c2390b6b7dcc06c394f P d870164263bb44b6b780a557daea3656: Time spent bootstrapping tablet: real 0.129s user 0.009s sys 0.003s
I20260130 02:58:10.110008 19625 raft_consensus.cc:359] T b2ac91da01954c2390b6b7dcc06c394f P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:10.112110 20180 raft_consensus.cc:515] T 43cec19c51ec4f05b63800c07f9534ef P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:10.119006 19625 raft_consensus.cc:385] T b2ac91da01954c2390b6b7dcc06c394f P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:10.119270 20331 raft_consensus.cc:493] T 8e57ae31ab3941f789ac69627384e3b0 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:10.119371 19625 raft_consensus.cc:740] T b2ac91da01954c2390b6b7dcc06c394f P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d870164263bb44b6b780a557daea3656, State: Initialized, Role: FOLLOWER
I20260130 02:58:10.119791 20331 raft_consensus.cc:515] T 8e57ae31ab3941f789ac69627384e3b0 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:10.120074 19625 consensus_queue.cc:260] T b2ac91da01954c2390b6b7dcc06c394f P d870164263bb44b6b780a557daea3656 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:10.124213 19840 tablet_bootstrap.cc:492] T 8e57ae31ab3941f789ac69627384e3b0 P 78c2c46c7520492db35d51c244fb4f74: No bootstrap required, opened a new log
I20260130 02:58:10.124118 19375 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8e57ae31ab3941f789ac69627384e3b0" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74" is_pre_election: true
I20260130 02:58:10.124643 19840 ts_tablet_manager.cc:1403] T 8e57ae31ab3941f789ac69627384e3b0 P 78c2c46c7520492db35d51c244fb4f74: Time spent bootstrapping tablet: real 0.096s user 0.009s sys 0.003s
W20260130 02:58:10.125833 19159 leader_election.cc:343] T 8e57ae31ab3941f789ac69627384e3b0 P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:10.121731 20331 leader_election.cc:290] T 8e57ae31ab3941f789ac69627384e3b0 P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767), d870164263bb44b6b780a557daea3656 (127.18.100.67:39571)
I20260130 02:58:10.126240 19521 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8e57ae31ab3941f789ac69627384e3b0" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
I20260130 02:58:10.126993 19625 ts_tablet_manager.cc:1434] T b2ac91da01954c2390b6b7dcc06c394f P d870164263bb44b6b780a557daea3656: Time spent starting tablet: real 0.019s user 0.007s sys 0.000s
W20260130 02:58:10.127357 19159 leader_election.cc:343] T 8e57ae31ab3941f789ac69627384e3b0 P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:10.127812 19625 tablet_bootstrap.cc:492] T 3593f8b89da24ff9b6cf549d5ac58bc6 P d870164263bb44b6b780a557daea3656: Bootstrap starting.
I20260130 02:58:10.127704 19159 leader_election.cc:304] T 8e57ae31ab3941f789ac69627384e3b0 P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 371a36d219164974abdf848841528152; no voters: 78c2c46c7520492db35d51c244fb4f74, d870164263bb44b6b780a557daea3656
I20260130 02:58:10.129912 20331 raft_consensus.cc:2749] T 8e57ae31ab3941f789ac69627384e3b0 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260130 02:58:10.136886 19376 raft_consensus.cc:2468] T 3ae5472cba1e4613abe4cda42ccb2496 P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 371a36d219164974abdf848841528152 in term 1.
I20260130 02:58:10.142343 19840 raft_consensus.cc:359] T 8e57ae31ab3941f789ac69627384e3b0 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:10.143038 19840 raft_consensus.cc:385] T 8e57ae31ab3941f789ac69627384e3b0 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:10.143358 19840 raft_consensus.cc:740] T 8e57ae31ab3941f789ac69627384e3b0 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78c2c46c7520492db35d51c244fb4f74, State: Initialized, Role: FOLLOWER
I20260130 02:58:10.143980 19840 consensus_queue.cc:260] T 8e57ae31ab3941f789ac69627384e3b0 P 78c2c46c7520492db35d51c244fb4f74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:10.145819 19840 ts_tablet_manager.cc:1434] T 8e57ae31ab3941f789ac69627384e3b0 P 78c2c46c7520492db35d51c244fb4f74: Time spent starting tablet: real 0.021s user 0.004s sys 0.000s
I20260130 02:58:10.162186 19159 leader_election.cc:304] T 3ae5472cba1e4613abe4cda42ccb2496 P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 371a36d219164974abdf848841528152, 78c2c46c7520492db35d51c244fb4f74; no voters: d870164263bb44b6b780a557daea3656
I20260130 02:58:10.163527 20243 raft_consensus.cc:2804] T 3ae5472cba1e4613abe4cda42ccb2496 P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Leader election won for term 1
I20260130 02:58:10.165246 20243 raft_consensus.cc:697] T 3ae5472cba1e4613abe4cda42ccb2496 P 371a36d219164974abdf848841528152 [term 1 LEADER]: Becoming Leader. State: Replica: 371a36d219164974abdf848841528152, State: Running, Role: LEADER
W20260130 02:58:10.166150 19314 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.100.65:42021, user_credentials={real_user=slave}} blocked reactor thread for 183819us
I20260130 02:58:10.166052 20243 consensus_queue.cc:237] T 3ae5472cba1e4613abe4cda42ccb2496 P 371a36d219164974abdf848841528152 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:10.167563 19521 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "43cec19c51ec4f05b63800c07f9534ef" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656"
W20260130 02:58:10.168872 19313 leader_election.cc:343] T 43cec19c51ec4f05b63800c07f9534ef P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:10.175598 20180 leader_election.cc:290] T 43cec19c51ec4f05b63800c07f9534ef P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 election: Requested vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), d870164263bb44b6b780a557daea3656 (127.18.100.67:39571)
I20260130 02:58:10.182476 19222 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "43cec19c51ec4f05b63800c07f9534ef" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152"
I20260130 02:58:10.183133 19222 raft_consensus.cc:3060] T 43cec19c51ec4f05b63800c07f9534ef P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:10.187713 18870 catalog_manager.cc:5654] T ee7c8ff92ca9465fa18be6eae63d78ca P 78c2c46c7520492db35d51c244fb4f74 reported cstate change: term changed from 0 to 1, leader changed from <none> to 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66). New cstate: current_term: 1 leader_uuid: "78c2c46c7520492db35d51c244fb4f74" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } health_report { overall_health: UNKNOWN } } }
I20260130 02:58:10.197057 18870 catalog_manager.cc:5746] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767) does not have the latest schema for tablet ee7c8ff92ca9465fa18be6eae63d78ca (table default.Table_dd22e9dc63f84d8688c0071f427f0b25 [id=77979d498aa040828de415129cb15a06]). Expected version 1 got 0
I20260130 02:58:10.198611 19840 tablet_bootstrap.cc:492] T a8a5038e9def44eb8b3b1e4aeae25652 P 78c2c46c7520492db35d51c244fb4f74: Bootstrap starting.
I20260130 02:58:10.200565 18870 catalog_manager.cc:5746] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767) does not have the latest schema for tablet 54be22352e404ff9b798a075693f7f21 (table default.Table_3a858a3cb886427cb8268f42c953ad4c [id=17150cb73e374ff9a1e92c43e0c3f2c1]). Expected version 1 got 0
W20260130 02:58:10.207005 19159 consensus_peers.cc:597] T 7d96e93d95f14af79cfb268626559b1e P 371a36d219164974abdf848841528152 -> Peer d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): Couldn't send request to peer d870164263bb44b6b780a557daea3656. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260130 02:58:10.202481 18870 catalog_manager.cc:5654] T 3593f8b89da24ff9b6cf549d5ac58bc6 P 78c2c46c7520492db35d51c244fb4f74 reported cstate change: term changed from 0 to 1, leader changed from <none> to 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66). New cstate: current_term: 1 leader_uuid: "78c2c46c7520492db35d51c244fb4f74" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } health_report { overall_health: UNKNOWN } } }
I20260130 02:58:10.209205 19807 tablet_bootstrap.cc:492] T 3b02480d9c6f4328af83f7940716d698 P 371a36d219164974abdf848841528152: No bootstrap required, opened a new log
I20260130 02:58:10.209614 19807 ts_tablet_manager.cc:1403] T 3b02480d9c6f4328af83f7940716d698 P 371a36d219164974abdf848841528152: Time spent bootstrapping tablet: real 0.251s user 0.029s sys 0.012s
I20260130 02:58:10.211735 19807 raft_consensus.cc:359] T 3b02480d9c6f4328af83f7940716d698 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:10.212317 19807 raft_consensus.cc:385] T 3b02480d9c6f4328af83f7940716d698 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:10.212563 19807 raft_consensus.cc:740] T 3b02480d9c6f4328af83f7940716d698 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 371a36d219164974abdf848841528152, State: Initialized, Role: FOLLOWER
I20260130 02:58:10.214778 20374 consensus_queue.cc:1048] T f54addb1a4304ce69c711ae0cc3b452c P 78c2c46c7520492db35d51c244fb4f74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260130 02:58:10.211962 18870 catalog_manager.cc:5746] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767) does not have the latest schema for tablet 127a84346b9f473d9e7f739b1e65feb8 (table default.Table_4f2d6e4d9c8e465087d65efc9c2de442 [id=1515db389f284e2f9ed1a3cd0061d0f1]). Expected version 5 got 0
I20260130 02:58:10.225675 20374 consensus_queue.cc:1048] T ee7c8ff92ca9465fa18be6eae63d78ca P 78c2c46c7520492db35d51c244fb4f74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260130 02:58:10.219228 18870 catalog_manager.cc:5746] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767) does not have the latest schema for tablet 3b148691090e4ee581d46f9845a6e49a (table default.Table_8edea1fae34d4d18b54b0f801a145225 [id=2009496525c94d7f8ee6ce59694b652d]). Expected version 2 got 0
W20260130 02:58:10.232846 18860 consensus_peers.cc:597] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4 -> Peer 3749baa287ed4f249329a74071781edd (127.18.100.124:34629): Couldn't send request to peer 3749baa287ed4f249329a74071781edd. Status: Network error: Client connection negotiation failed: client connection to 127.18.100.124:34629: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20260130 02:58:10.247651 19807 consensus_queue.cc:260] T 3b02480d9c6f4328af83f7940716d698 P 371a36d219164974abdf848841528152 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:10.249747 19807 ts_tablet_manager.cc:1434] T 3b02480d9c6f4328af83f7940716d698 P 371a36d219164974abdf848841528152: Time spent starting tablet: real 0.040s user 0.005s sys 0.000s
I20260130 02:58:10.252182 19222 raft_consensus.cc:2468] T 43cec19c51ec4f05b63800c07f9534ef P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78c2c46c7520492db35d51c244fb4f74 in term 1.
I20260130 02:58:10.253275 20374 raft_consensus.cc:493] T 8e57ae31ab3941f789ac69627384e3b0 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:10.253690 20374 raft_consensus.cc:515] T 8e57ae31ab3941f789ac69627384e3b0 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:10.256444 19521 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8e57ae31ab3941f789ac69627384e3b0" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
I20260130 02:58:10.256597 20374 leader_election.cc:290] T 8e57ae31ab3941f789ac69627384e3b0 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), d870164263bb44b6b780a557daea3656 (127.18.100.67:39571)
W20260130 02:58:10.257681 19313 leader_election.cc:343] T 8e57ae31ab3941f789ac69627384e3b0 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:10.257206 19314 leader_election.cc:304] T 43cec19c51ec4f05b63800c07f9534ef P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 371a36d219164974abdf848841528152, 78c2c46c7520492db35d51c244fb4f74; no voters: d870164263bb44b6b780a557daea3656
I20260130 02:58:10.258425 20374 raft_consensus.cc:2804] T 43cec19c51ec4f05b63800c07f9534ef P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Leader election won for term 1
I20260130 02:58:10.259395 20374 raft_consensus.cc:697] T 43cec19c51ec4f05b63800c07f9534ef P 78c2c46c7520492db35d51c244fb4f74 [term 1 LEADER]: Becoming Leader. State: Replica: 78c2c46c7520492db35d51c244fb4f74, State: Running, Role: LEADER
I20260130 02:58:10.260221 20374 consensus_queue.cc:237] T 43cec19c51ec4f05b63800c07f9534ef P 78c2c46c7520492db35d51c244fb4f74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:10.267127 19223 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8e57ae31ab3941f789ac69627384e3b0" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152" is_pre_election: true
I20260130 02:58:10.268119 19223 raft_consensus.cc:2468] T 8e57ae31ab3941f789ac69627384e3b0 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78c2c46c7520492db35d51c244fb4f74 in term 0.
I20260130 02:58:10.270242 19840 tablet_bootstrap.cc:654] T a8a5038e9def44eb8b3b1e4aeae25652 P 78c2c46c7520492db35d51c244fb4f74: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:10.258764 19625 tablet_bootstrap.cc:654] T 3593f8b89da24ff9b6cf549d5ac58bc6 P d870164263bb44b6b780a557daea3656: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:10.282306 19314 leader_election.cc:304] T 8e57ae31ab3941f789ac69627384e3b0 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 371a36d219164974abdf848841528152, 78c2c46c7520492db35d51c244fb4f74; no voters: d870164263bb44b6b780a557daea3656
I20260130 02:58:10.283151 20390 raft_consensus.cc:2804] T 8e57ae31ab3941f789ac69627384e3b0 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260130 02:58:10.283758 20390 raft_consensus.cc:493] T 8e57ae31ab3941f789ac69627384e3b0 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260130 02:58:10.284096 20390 raft_consensus.cc:3060] T 8e57ae31ab3941f789ac69627384e3b0 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:10.291692 18873 catalog_manager.cc:5746] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021) does not have the latest schema for tablet ee7c8ff92ca9465fa18be6eae63d78ca (table default.Table_dd22e9dc63f84d8688c0071f427f0b25 [id=77979d498aa040828de415129cb15a06]). Expected version 1 got 0
I20260130 02:58:10.290872 19807 tablet_bootstrap.cc:492] T d55291d7cea64a0d9a345ce712fbfa80 P 371a36d219164974abdf848841528152: Bootstrap starting.
I20260130 02:58:10.302876 20390 raft_consensus.cc:515] T 8e57ae31ab3941f789ac69627384e3b0 P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:10.305275 18873 catalog_manager.cc:5746] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021) does not have the latest schema for tablet 54be22352e404ff9b798a075693f7f21 (table default.Table_3a858a3cb886427cb8268f42c953ad4c [id=17150cb73e374ff9a1e92c43e0c3f2c1]). Expected version 1 got 0
I20260130 02:58:10.305761 19521 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8e57ae31ab3941f789ac69627384e3b0" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656"
W20260130 02:58:10.307018 19313 leader_election.cc:343] T 8e57ae31ab3941f789ac69627384e3b0 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:10.314659 19625 tablet_bootstrap.cc:492] T 3593f8b89da24ff9b6cf549d5ac58bc6 P d870164263bb44b6b780a557daea3656: No bootstrap required, opened a new log
I20260130 02:58:10.315121 19625 ts_tablet_manager.cc:1403] T 3593f8b89da24ff9b6cf549d5ac58bc6 P d870164263bb44b6b780a557daea3656: Time spent bootstrapping tablet: real 0.187s user 0.003s sys 0.007s
I20260130 02:58:10.318557 19223 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8e57ae31ab3941f789ac69627384e3b0" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152"
I20260130 02:58:10.320127 19625 raft_consensus.cc:359] T 3593f8b89da24ff9b6cf549d5ac58bc6 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:10.320991 19625 raft_consensus.cc:385] T 3593f8b89da24ff9b6cf549d5ac58bc6 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:10.321249 19625 raft_consensus.cc:740] T 3593f8b89da24ff9b6cf549d5ac58bc6 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d870164263bb44b6b780a557daea3656, State: Initialized, Role: FOLLOWER
I20260130 02:58:10.321424 19223 raft_consensus.cc:3060] T 8e57ae31ab3941f789ac69627384e3b0 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:10.324203 20390 leader_election.cc:290] T 8e57ae31ab3941f789ac69627384e3b0 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 election: Requested vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), d870164263bb44b6b780a557daea3656 (127.18.100.67:39571)
I20260130 02:58:10.324182 18873 catalog_manager.cc:5746] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021) does not have the latest schema for tablet 127a84346b9f473d9e7f739b1e65feb8 (table default.Table_4f2d6e4d9c8e465087d65efc9c2de442 [id=1515db389f284e2f9ed1a3cd0061d0f1]). Expected version 5 got 0
I20260130 02:58:10.323550 19625 consensus_queue.cc:260] T 3593f8b89da24ff9b6cf549d5ac58bc6 P d870164263bb44b6b780a557daea3656 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:10.325105 18873 catalog_manager.cc:5746] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021) does not have the latest schema for tablet 3b148691090e4ee581d46f9845a6e49a (table default.Table_8edea1fae34d4d18b54b0f801a145225 [id=2009496525c94d7f8ee6ce59694b652d]). Expected version 2 got 0
I20260130 02:58:10.325345 19807 tablet_bootstrap.cc:654] T d55291d7cea64a0d9a345ce712fbfa80 P 371a36d219164974abdf848841528152: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:10.329237 19223 raft_consensus.cc:2468] T 8e57ae31ab3941f789ac69627384e3b0 P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78c2c46c7520492db35d51c244fb4f74 in term 1.
I20260130 02:58:10.331804 19840 tablet_bootstrap.cc:492] T a8a5038e9def44eb8b3b1e4aeae25652 P 78c2c46c7520492db35d51c244fb4f74: No bootstrap required, opened a new log
I20260130 02:58:10.332219 19840 ts_tablet_manager.cc:1403] T a8a5038e9def44eb8b3b1e4aeae25652 P 78c2c46c7520492db35d51c244fb4f74: Time spent bootstrapping tablet: real 0.152s user 0.009s sys 0.000s
I20260130 02:58:10.334851 19314 leader_election.cc:304] T 8e57ae31ab3941f789ac69627384e3b0 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 371a36d219164974abdf848841528152, 78c2c46c7520492db35d51c244fb4f74; no voters: d870164263bb44b6b780a557daea3656
I20260130 02:58:10.334471 19840 raft_consensus.cc:359] T a8a5038e9def44eb8b3b1e4aeae25652 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:10.336751 19840 raft_consensus.cc:385] T a8a5038e9def44eb8b3b1e4aeae25652 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:10.337015 19840 raft_consensus.cc:740] T a8a5038e9def44eb8b3b1e4aeae25652 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78c2c46c7520492db35d51c244fb4f74, State: Initialized, Role: FOLLOWER
I20260130 02:58:10.348181 19840 consensus_queue.cc:260] T a8a5038e9def44eb8b3b1e4aeae25652 P 78c2c46c7520492db35d51c244fb4f74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:10.349006 20390 raft_consensus.cc:2804] T 8e57ae31ab3941f789ac69627384e3b0 P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Leader election won for term 1
I20260130 02:58:10.350659 19625 ts_tablet_manager.cc:1434] T 3593f8b89da24ff9b6cf549d5ac58bc6 P d870164263bb44b6b780a557daea3656: Time spent starting tablet: real 0.035s user 0.006s sys 0.001s
I20260130 02:58:10.354799 20390 raft_consensus.cc:697] T 8e57ae31ab3941f789ac69627384e3b0 P 78c2c46c7520492db35d51c244fb4f74 [term 1 LEADER]: Becoming Leader. State: Replica: 78c2c46c7520492db35d51c244fb4f74, State: Running, Role: LEADER
I20260130 02:58:10.355587 20390 consensus_queue.cc:237] T 8e57ae31ab3941f789ac69627384e3b0 P 78c2c46c7520492db35d51c244fb4f74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:10.356468 19840 ts_tablet_manager.cc:1434] T a8a5038e9def44eb8b3b1e4aeae25652 P 78c2c46c7520492db35d51c244fb4f74: Time spent starting tablet: real 0.024s user 0.007s sys 0.001s
I20260130 02:58:10.360234 18875 catalog_manager.cc:5746] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571) does not have the latest schema for tablet 3b148691090e4ee581d46f9845a6e49a (table default.Table_8edea1fae34d4d18b54b0f801a145225 [id=2009496525c94d7f8ee6ce59694b652d]). Expected version 2 got 0
I20260130 02:58:10.363018 19840 tablet_bootstrap.cc:492] T d55291d7cea64a0d9a345ce712fbfa80 P 78c2c46c7520492db35d51c244fb4f74: Bootstrap starting.
I20260130 02:58:10.369319 19625 tablet_bootstrap.cc:492] T 7d96e93d95f14af79cfb268626559b1e P d870164263bb44b6b780a557daea3656: Bootstrap starting.
I20260130 02:58:10.373632 19807 tablet_bootstrap.cc:492] T d55291d7cea64a0d9a345ce712fbfa80 P 371a36d219164974abdf848841528152: No bootstrap required, opened a new log
I20260130 02:58:10.374028 19807 ts_tablet_manager.cc:1403] T d55291d7cea64a0d9a345ce712fbfa80 P 371a36d219164974abdf848841528152: Time spent bootstrapping tablet: real 0.083s user 0.006s sys 0.003s
I20260130 02:58:10.377633 18875 catalog_manager.cc:5746] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571) does not have the latest schema for tablet 54be22352e404ff9b798a075693f7f21 (table default.Table_3a858a3cb886427cb8268f42c953ad4c [id=17150cb73e374ff9a1e92c43e0c3f2c1]). Expected version 1 got 0
I20260130 02:58:10.377624 19807 raft_consensus.cc:359] T d55291d7cea64a0d9a345ce712fbfa80 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:10.378245 19807 raft_consensus.cc:385] T d55291d7cea64a0d9a345ce712fbfa80 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:10.378479 19807 raft_consensus.cc:740] T d55291d7cea64a0d9a345ce712fbfa80 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 371a36d219164974abdf848841528152, State: Initialized, Role: FOLLOWER
I20260130 02:58:10.379065 19807 consensus_queue.cc:260] T d55291d7cea64a0d9a345ce712fbfa80 P 371a36d219164974abdf848841528152 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:10.379881 18867 catalog_manager.cc:6199] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4: default.Table_3413dae2efac41488448e6e5975cf710 [id=7a08fa3986d0410d81de0235d2fb0f92] alter complete (version 1)
I20260130 02:58:10.380138 20375 consensus_queue.cc:1048] T 04f4ac27800c4ac3a79ae12d35479757 P 78c2c46c7520492db35d51c244fb4f74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260130 02:58:10.393137 18866 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:50284 (request call id 209) took 1863 ms. Trace:
I20260130 02:58:10.397991 19625 tablet_bootstrap.cc:654] T 7d96e93d95f14af79cfb268626559b1e P d870164263bb44b6b780a557daea3656: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:10.399048 19807 ts_tablet_manager.cc:1434] T d55291d7cea64a0d9a345ce712fbfa80 P 371a36d219164974abdf848841528152: Time spent starting tablet: real 0.025s user 0.004s sys 0.004s
I20260130 02:58:10.393457 18866 rpcz_store.cc:276] 0130 02:58:08.529931 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:58:08.707482 (+177551us) service_pool.cc:225] Handling call
0130 02:58:08.708415 (+ 933us) catalog_manager.cc:2393] Looking up, locking, and authorizing table
0130 02:58:08.709077 (+ 662us) catalog_manager.cc:3708] Apply alter schema
0130 02:58:08.709634 (+ 557us) catalog_manager.cc:3740] Acquired catalog manager lock
0130 02:58:08.709830 (+ 196us) catalog_manager.cc:3898] Updating metadata on disk
0130 02:58:08.954683 (+244853us) catalog_manager.cc:3929] Committing alterations to in-memory state
0130 02:58:10.392976 (+1438293us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0130 02:58:08.794661 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0130 02:58:08.795186 (+ 525us) write_op.cc:432] Acquiring schema lock in shared mode
0130 02:58:08.795214 (+ 28us) write_op.cc:435] Acquired schema lock
0130 02:58:08.795231 (+ 17us) tablet.cc:660] Decoding operations
0130 02:58:08.795535 (+ 304us) write_op.cc:620] Acquiring the partition lock for write op
0130 02:58:08.795576 (+ 41us) write_op.cc:641] Partition lock acquired for write op
0130 02:58:08.795599 (+ 23us) tablet.cc:683] Acquiring locks for 1 operations
0130 02:58:08.795750 (+ 151us) tablet.cc:699] Row locks acquired
0130 02:58:08.795765 (+ 15us) write_op.cc:260] PREPARE: finished
0130 02:58:08.795897 (+ 132us) write_op.cc:270] Start()
0130 02:58:08.795985 (+ 88us) write_op.cc:276] Timestamp: P: 1769741888795841 usec, L: 0
0130 02:58:08.796000 (+ 15us) op_driver.cc:348] REPLICATION: starting
0130 02:58:08.796504 (+ 504us) log.cc:844] Serialized 405 byte log entry
0130 02:58:08.944410 (+147906us) op_driver.cc:464] REPLICATION: finished
0130 02:58:08.948700 (+ 4290us) write_op.cc:301] APPLY: starting
0130 02:58:08.948753 (+ 53us) tablet.cc:1365] starting BulkCheckPresence
0130 02:58:08.948839 (+ 86us) tablet.cc:1368] finished BulkCheckPresence
0130 02:58:08.948950 (+ 111us) tablet.cc:1370] starting ApplyRowOperation cycle
0130 02:58:08.949095 (+ 145us) tablet.cc:1381] finished ApplyRowOperation cycle
0130 02:58:08.949228 (+ 133us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0130 02:58:08.949273 (+ 45us) write_op.cc:312] APPLY: finished
0130 02:58:08.949437 (+ 164us) log.cc:844] Serialized 23 byte log entry
0130 02:58:08.949572 (+ 135us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:58:08.949625 (+ 53us) write_op.cc:454] Released schema lock
0130 02:58:08.949654 (+ 29us) write_op.cc:341] FINISH: Updating metrics
0130 02:58:08.949863 (+ 209us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:58:08.949892 (+ 29us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":4153,"apply.run_cpu_time_us":1131,"apply.run_wall_time_us":1132,"num_ops":1,"prepare.queue_time_us":43016,"prepare.run_cpu_time_us":2162,"prepare.run_wall_time_us":3280,"raft.queue_time_us":74,"raft.run_cpu_time_us":10,"raft.run_wall_time_us":11,"replication_time_us":148316}]]}
I20260130 02:58:10.399946 19807 tablet_bootstrap.cc:492] T d27ca7dd87f44102b7680e1f4a88954a P 371a36d219164974abdf848841528152: Bootstrap starting.
I20260130 02:58:10.401388 19840 tablet_bootstrap.cc:654] T d55291d7cea64a0d9a345ce712fbfa80 P 78c2c46c7520492db35d51c244fb4f74: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:10.408102 18868 catalog_manager.cc:6199] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4: default.Table_89d4117da99842719125411e30d10169 [id=d128767e303145a885eec3c5f714a27f] alter complete (version 3)
I20260130 02:58:10.420089 19625 tablet_bootstrap.cc:492] T 7d96e93d95f14af79cfb268626559b1e P d870164263bb44b6b780a557daea3656: No bootstrap required, opened a new log
I20260130 02:58:10.420521 19625 ts_tablet_manager.cc:1403] T 7d96e93d95f14af79cfb268626559b1e P d870164263bb44b6b780a557daea3656: Time spent bootstrapping tablet: real 0.051s user 0.007s sys 0.003s
I20260130 02:58:10.422824 19625 raft_consensus.cc:359] T 7d96e93d95f14af79cfb268626559b1e P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:10.423027 19350 tablet_service.cc:1552] Processing DeleteTablet for tablet 3b4908bd1a634ca0907234dd3c39e1c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:58:08 UTC) from {username='slave'} at 127.0.0.1:60110
I20260130 02:58:10.423441 19625 raft_consensus.cc:385] T 7d96e93d95f14af79cfb268626559b1e P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:10.423578 19807 tablet_bootstrap.cc:654] T d27ca7dd87f44102b7680e1f4a88954a P 371a36d219164974abdf848841528152: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:10.423689 19625 raft_consensus.cc:740] T 7d96e93d95f14af79cfb268626559b1e P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d870164263bb44b6b780a557daea3656, State: Initialized, Role: FOLLOWER
I20260130 02:58:10.428215 19625 consensus_queue.cc:260] T 7d96e93d95f14af79cfb268626559b1e P d870164263bb44b6b780a557daea3656 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:10.431242 19358 tablet_service.cc:1552] Processing DeleteTablet for tablet 8bd3e1c06b89426da57e2b259e7983af with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:58:08 UTC) from {username='slave'} at 127.0.0.1:60110
I20260130 02:58:10.436250 19358 tablet_service.cc:1552] Processing DeleteTablet for tablet 7d76046ee6c8420fa4964decbe8209c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:58:06 UTC) from {username='slave'} at 127.0.0.1:60110
I20260130 02:58:10.438450 19358 tablet_service.cc:1552] Processing DeleteTablet for tablet 481bd4cca6ee4c178873bf12392e3441 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:58:02 UTC) from {username='slave'} at 127.0.0.1:60110
I20260130 02:58:10.440646 19521 raft_consensus.cc:3060] T f54addb1a4304ce69c711ae0cc3b452c P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:10.451095 18870 catalog_manager.cc:6199] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4: default.Table_9f83f87711d94e8394abe1925ac84ad1 [id=db23c1be42c44b6f9eb774f14d2b8fd6] alter complete (version 1)
I20260130 02:58:10.456851 19840 tablet_bootstrap.cc:492] T d55291d7cea64a0d9a345ce712fbfa80 P 78c2c46c7520492db35d51c244fb4f74: No bootstrap required, opened a new log
I20260130 02:58:10.457291 19840 ts_tablet_manager.cc:1403] T d55291d7cea64a0d9a345ce712fbfa80 P 78c2c46c7520492db35d51c244fb4f74: Time spent bootstrapping tablet: real 0.094s user 0.005s sys 0.003s
I20260130 02:58:10.458827 19625 ts_tablet_manager.cc:1434] T 7d96e93d95f14af79cfb268626559b1e P d870164263bb44b6b780a557daea3656: Time spent starting tablet: real 0.038s user 0.006s sys 0.000s
I20260130 02:58:10.463281 19625 tablet_bootstrap.cc:492] T ee7c8ff92ca9465fa18be6eae63d78ca P d870164263bb44b6b780a557daea3656: Bootstrap starting.
I20260130 02:58:10.469944 19840 raft_consensus.cc:359] T d55291d7cea64a0d9a345ce712fbfa80 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:10.470362 19508 tablet_service.cc:1552] Processing DeleteTablet for tablet 5ca9919f80954e74a4d91c94b74de0e4 with delete_type TABLET_DATA_DELETED (replaced by tablet 13f0ace4d9de43abab08eca32cdb30c1) from {username='slave'} at 127.0.0.1:59068
I20260130 02:58:10.470604 19840 raft_consensus.cc:385] T d55291d7cea64a0d9a345ce712fbfa80 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:10.470670 19502 tablet_service.cc:1552] Processing DeleteTablet for tablet 7d76046ee6c8420fa4964decbe8209c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:58:06 UTC) from {username='slave'} at 127.0.0.1:59068
I20260130 02:58:10.470837 19840 raft_consensus.cc:740] T d55291d7cea64a0d9a345ce712fbfa80 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78c2c46c7520492db35d51c244fb4f74, State: Initialized, Role: FOLLOWER
I20260130 02:58:10.471272 19505 tablet_service.cc:1552] Processing DeleteTablet for tablet 730bd1d7d15b4cbe945b3ef03b2455f7 with delete_type TABLET_DATA_DELETED (replaced by tablet 3b02480d9c6f4328af83f7940716d698) from {username='slave'} at 127.0.0.1:59068
I20260130 02:58:10.471390 19840 consensus_queue.cc:260] T d55291d7cea64a0d9a345ce712fbfa80 P 78c2c46c7520492db35d51c244fb4f74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:10.472769 19947 tablet_replica.cc:333] T 730bd1d7d15b4cbe945b3ef03b2455f7 P d870164263bb44b6b780a557daea3656: stopping tablet replica
I20260130 02:58:10.473309 19351 tablet_service.cc:1552] Processing DeleteTablet for tablet 3cf5154d8cc44f2bb12978202699c193 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:58:06 UTC) from {username='slave'} at 127.0.0.1:60110
I20260130 02:58:10.473488 19947 raft_consensus.cc:2243] T 730bd1d7d15b4cbe945b3ef03b2455f7 P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260130 02:58:10.473940 19947 raft_consensus.cc:2272] T 730bd1d7d15b4cbe945b3ef03b2455f7 P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260130 02:58:10.476647 19358 tablet_service.cc:1552] Processing DeleteTablet for tablet 927c90b84d7449efa36bc5e2035bf0da with delete_type TABLET_DATA_DELETED (replaced by tablet 25fd48e693424537b91ba7c3fcc96878) from {username='slave'} at 127.0.0.1:60110
I20260130 02:58:10.477197 19947 ts_tablet_manager.cc:1916] T 730bd1d7d15b4cbe945b3ef03b2455f7 P d870164263bb44b6b780a557daea3656: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260130 02:58:10.477949 18860 catalog_manager.cc:4952] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): delete failed for tablet 5ca9919f80954e74a4d91c94b74de0e4 because the tablet was not found. No further retry: Not found: Tablet not found: 5ca9919f80954e74a4d91c94b74de0e4
W20260130 02:58:10.483006 18860 catalog_manager.cc:4952] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): delete failed for tablet 7d76046ee6c8420fa4964decbe8209c1 because the tablet was not found. No further retry: Not found: Tablet not found: 7d76046ee6c8420fa4964decbe8209c1
I20260130 02:58:10.486824 19205 tablet_service.cc:1552] Processing DeleteTablet for tablet 481bd4cca6ee4c178873bf12392e3441 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:58:02 UTC) from {username='slave'} at 127.0.0.1:42870
I20260130 02:58:10.486824 19201 tablet_service.cc:1552] Processing DeleteTablet for tablet 5ca9919f80954e74a4d91c94b74de0e4 with delete_type TABLET_DATA_DELETED (replaced by tablet 13f0ace4d9de43abab08eca32cdb30c1) from {username='slave'} at 127.0.0.1:42870
W20260130 02:58:10.462450 18872 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.18.100.67:47213 (request call id 23) took 3305 ms (client timeout 1000 ms). Trace:
W20260130 02:58:10.492081 18861 catalog_manager.cc:4952] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021): delete failed for tablet 481bd4cca6ee4c178873bf12392e3441 because the tablet was not found. No further retry: Not found: Tablet not found: 481bd4cca6ee4c178873bf12392e3441
I20260130 02:58:10.494089 19351 tablet_service.cc:1552] Processing DeleteTablet for tablet 2112b0402de5464cb1caf2c2abb67d78 with delete_type TABLET_DATA_DELETED (replaced by tablet 838e49e5e8874db393bdc15a1c92f87e) from {username='slave'} at 127.0.0.1:60110
I20260130 02:58:10.494817 19358 tablet_service.cc:1552] Processing DeleteTablet for tablet 730bd1d7d15b4cbe945b3ef03b2455f7 with delete_type TABLET_DATA_DELETED (replaced by tablet 3b02480d9c6f4328af83f7940716d698) from {username='slave'} at 127.0.0.1:60110
W20260130 02:58:10.498157 18861 catalog_manager.cc:4952] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021): delete failed for tablet 5ca9919f80954e74a4d91c94b74de0e4 because the tablet was not found. No further retry: Not found: Tablet not found: 5ca9919f80954e74a4d91c94b74de0e4
I20260130 02:58:10.500602 19205 tablet_service.cc:1552] Processing DeleteTablet for tablet bd43c5d148514836869a8ac648c8ad11 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:58:00 UTC) from {username='slave'} at 127.0.0.1:42870
I20260130 02:58:10.502774 19351 tablet_service.cc:1552] Processing DeleteTablet for tablet 1bbc55bbf2254d8bb6b721d7e8f69d85 with delete_type TABLET_DATA_DELETED (replaced by tablet 657e9b0d8c05413f904129dce745710b) from {username='slave'} at 127.0.0.1:60110
W20260130 02:58:10.503712 18861 catalog_manager.cc:4952] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021): delete failed for tablet bd43c5d148514836869a8ac648c8ad11 because the tablet was not found. No further retry: Not found: Tablet not found: bd43c5d148514836869a8ac648c8ad11
I20260130 02:58:10.503952 19351 tablet_service.cc:1552] Processing DeleteTablet for tablet 6573198830ad45628e71952a7d9c00a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:58:03 UTC) from {username='slave'} at 127.0.0.1:60110
I20260130 02:58:10.505798 19358 tablet_service.cc:1552] Processing DeleteTablet for tablet f54addb1a4304ce69c711ae0cc3b452c with delete_type TABLET_DATA_DELETED (replaced by tablet 7783db3568f4434e91e9e456799cc312) from {username='slave'} at 127.0.0.1:60110
W20260130 02:58:10.507781 18868 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.18.100.66:59691 (request call id 25) took 3988 ms (client timeout 1000 ms). Trace:
I20260130 02:58:10.510126 19205 tablet_service.cc:1552] Processing DeleteTablet for tablet 7d76046ee6c8420fa4964decbe8209c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:58:06 UTC) from {username='slave'} at 127.0.0.1:42870
I20260130 02:58:10.513903 19947 ts_tablet_manager.cc:1929] T 730bd1d7d15b4cbe945b3ef03b2455f7 P d870164263bb44b6b780a557daea3656: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260130 02:58:10.514248 19947 log.cc:1199] T 730bd1d7d15b4cbe945b3ef03b2455f7 P d870164263bb44b6b780a557daea3656: Deleting WAL directory at /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-2/wal/wals/730bd1d7d15b4cbe945b3ef03b2455f7
I20260130 02:58:10.523620 19947 ts_tablet_manager.cc:1950] T 730bd1d7d15b4cbe945b3ef03b2455f7 P d870164263bb44b6b780a557daea3656: Deleting consensus metadata
I20260130 02:58:10.525782 19625 tablet_bootstrap.cc:654] T ee7c8ff92ca9465fa18be6eae63d78ca P d870164263bb44b6b780a557daea3656: Neither blocks nor log segments found. Creating new log.
W20260130 02:58:10.511381 18861 catalog_manager.cc:4952] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021): delete failed for tablet 7d76046ee6c8420fa4964decbe8209c1 because the tablet was not found. No further retry: Not found: Tablet not found: 7d76046ee6c8420fa4964decbe8209c1
I20260130 02:58:10.531381 19205 tablet_service.cc:1552] Processing DeleteTablet for tablet 5ca9919f80954e74a4d91c94b74de0e4 with delete_type TABLET_DATA_DELETED (replaced by tablet 13f0ace4d9de43abab08eca32cdb30c1) from {username='slave'} at 127.0.0.1:42870
I20260130 02:58:10.532786 19201 tablet_service.cc:1552] Processing DeleteTablet for tablet 3b4908bd1a634ca0907234dd3c39e1c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:58:08 UTC) from {username='slave'} at 127.0.0.1:42870
I20260130 02:58:10.533418 19199 tablet_service.cc:1552] Processing DeleteTablet for tablet 8bd3e1c06b89426da57e2b259e7983af with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:58:08 UTC) from {username='slave'} at 127.0.0.1:42870
I20260130 02:58:10.533973 19198 tablet_service.cc:1552] Processing DeleteTablet for tablet 730bd1d7d15b4cbe945b3ef03b2455f7 with delete_type TABLET_DATA_DELETED (replaced by tablet 3b02480d9c6f4328af83f7940716d698) from {username='slave'} at 127.0.0.1:42870
I20260130 02:58:10.534237 18860 catalog_manager.cc:4985] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): tablet 730bd1d7d15b4cbe945b3ef03b2455f7 (table default.Table_1e9e20a1bfec4bcf81f9eb6219c33d59 [id=f675b7a3ec744533903d8834eeed51d7]) successfully deleted
I20260130 02:58:10.534565 19197 tablet_service.cc:1552] Processing DeleteTablet for tablet 3cf5154d8cc44f2bb12978202699c193 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:58:06 UTC) from {username='slave'} at 127.0.0.1:42870
I20260130 02:58:10.534972 19201 tablet_service.cc:1552] Processing DeleteTablet for tablet 6573198830ad45628e71952a7d9c00a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:58:03 UTC) from {username='slave'} at 127.0.0.1:42870
W20260130 02:58:10.535897 18861 catalog_manager.cc:4952] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021): delete failed for tablet 5ca9919f80954e74a4d91c94b74de0e4 because the tablet was not found. No further retry: Not found: Tablet not found: 5ca9919f80954e74a4d91c94b74de0e4
I20260130 02:58:10.536185 20422 tablet_replica.cc:333] T 730bd1d7d15b4cbe945b3ef03b2455f7 P 371a36d219164974abdf848841528152: stopping tablet replica
W20260130 02:58:10.536932 18861 catalog_manager.cc:4952] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021): delete failed for tablet 3b4908bd1a634ca0907234dd3c39e1c3 because the tablet was not found. No further retry: Not found: Tablet not found: 3b4908bd1a634ca0907234dd3c39e1c3
I20260130 02:58:10.537081 20422 raft_consensus.cc:2243] T 730bd1d7d15b4cbe945b3ef03b2455f7 P 371a36d219164974abdf848841528152 [term 1 LEADER]: Raft consensus shutting down.
W20260130 02:58:10.537424 18861 catalog_manager.cc:4952] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021): delete failed for tablet 8bd3e1c06b89426da57e2b259e7983af because the tablet was not found. No further retry: Not found: Tablet not found: 8bd3e1c06b89426da57e2b259e7983af
I20260130 02:58:10.534181 19205 tablet_service.cc:1552] Processing DeleteTablet for tablet f54addb1a4304ce69c711ae0cc3b452c with delete_type TABLET_DATA_DELETED (replaced by tablet 7783db3568f4434e91e9e456799cc312) from {username='slave'} at 127.0.0.1:42870
I20260130 02:58:10.537942 20422 raft_consensus.cc:2272] T 730bd1d7d15b4cbe945b3ef03b2455f7 P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260130 02:58:10.542084 19840 ts_tablet_manager.cc:1434] T d55291d7cea64a0d9a345ce712fbfa80 P 78c2c46c7520492db35d51c244fb4f74: Time spent starting tablet: real 0.085s user 0.008s sys 0.000s
I20260130 02:58:10.543797 19358 tablet_service.cc:1552] Processing DeleteTablet for tablet b2ac91da01954c2390b6b7dcc06c394f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:58:04 UTC) from {username='slave'} at 127.0.0.1:60110
I20260130 02:58:10.531199 20371 raft_consensus.cc:493] T 7d96e93d95f14af79cfb268626559b1e P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260130 02:58:10.490988 18872 rpcz_store.cc:269] 0130 02:58:07.157074 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:58:07.404825 (+247751us) service_pool.cc:225] Handling call
0130 02:58:10.462242 (+3057417us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0130 02:58:08.330633 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0130 02:58:08.331086 (+ 453us) write_op.cc:432] Acquiring schema lock in shared mode
0130 02:58:08.331112 (+ 26us) write_op.cc:435] Acquired schema lock
0130 02:58:08.331127 (+ 15us) tablet.cc:660] Decoding operations
0130 02:58:08.331403 (+ 276us) write_op.cc:620] Acquiring the partition lock for write op
0130 02:58:08.331439 (+ 36us) write_op.cc:641] Partition lock acquired for write op
0130 02:58:08.331465 (+ 26us) tablet.cc:683] Acquiring locks for 1 operations
0130 02:58:08.331620 (+ 155us) tablet.cc:699] Row locks acquired
0130 02:58:08.331637 (+ 17us) write_op.cc:260] PREPARE: finished
0130 02:58:08.331749 (+ 112us) write_op.cc:270] Start()
0130 02:58:08.331834 (+ 85us) write_op.cc:276] Timestamp: P: 1769741888331686 usec, L: 0
0130 02:58:08.331852 (+ 18us) op_driver.cc:348] REPLICATION: starting
0130 02:58:08.333474 (+ 1622us) log.cc:844] Serialized 533 byte log entry
0130 02:58:08.372835 (+ 39361us) op_driver.cc:464] REPLICATION: finished
0130 02:58:08.376608 (+ 3773us) write_op.cc:301] APPLY: starting
0130 02:58:08.376659 (+ 51us) tablet.cc:1365] starting BulkCheckPresence
0130 02:58:08.376737 (+ 78us) tablet.cc:1368] finished BulkCheckPresence
0130 02:58:08.376821 (+ 84us) tablet.cc:1370] starting ApplyRowOperation cycle
0130 02:58:08.376957 (+ 136us) tablet.cc:1381] finished ApplyRowOperation cycle
0130 02:58:08.377195 (+ 238us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0130 02:58:08.377234 (+ 39us) write_op.cc:312] APPLY: finished
0130 02:58:08.377405 (+ 171us) log.cc:844] Serialized 23 byte log entry
0130 02:58:08.377541 (+ 136us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:58:08.377603 (+ 62us) write_op.cc:454] Released schema lock
0130 02:58:08.377630 (+ 27us) write_op.cc:341] FINISH: Updating metrics
0130 02:58:08.377826 (+ 196us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:58:08.377849 (+ 23us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":23,"child_traces":[["op",{"apply.queue_time_us":3658,"apply.run_cpu_time_us":1188,"apply.run_wall_time_us":1188,"num_ops":1,"prepare.queue_time_us":2040,"prepare.run_cpu_time_us":2172,"prepare.run_wall_time_us":4357,"raft.queue_time_us":79,"raft.run_cpu_time_us":13,"raft.run_wall_time_us":14,"replication_time_us":40902}]]}
I20260130 02:58:10.544703 20371 raft_consensus.cc:515] T 7d96e93d95f14af79cfb268626559b1e P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:10.546634 20371 leader_election.cc:290] T 7d96e93d95f14af79cfb268626559b1e P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767)
W20260130 02:58:10.527860 18869 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.18.100.65:51021 (request call id 30) took 3050 ms (client timeout 1000 ms). Trace:
I20260130 02:58:10.548661 19225 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7d96e93d95f14af79cfb268626559b1e" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152" is_pre_election: true
I20260130 02:58:10.549932 19378 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7d96e93d95f14af79cfb268626559b1e" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74" is_pre_election: true
I20260130 02:58:10.553107 19358 tablet_service.cc:1552] Processing DeleteTablet for tablet 5ca9919f80954e74a4d91c94b74de0e4 with delete_type TABLET_DATA_DELETED (replaced by tablet 13f0ace4d9de43abab08eca32cdb30c1) from {username='slave'} at 127.0.0.1:60110
I20260130 02:58:10.556946 19496 tablet_service.cc:1552] Processing DeleteTablet for tablet 3b4908bd1a634ca0907234dd3c39e1c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:58:08 UTC) from {username='slave'} at 127.0.0.1:59068
I20260130 02:58:10.558524 19496 tablet_service.cc:1552] Processing DeleteTablet for tablet 3cf5154d8cc44f2bb12978202699c193 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:58:06 UTC) from {username='slave'} at 127.0.0.1:59068
I20260130 02:58:10.559183 19947 tablet_replica.cc:333] T 3cf5154d8cc44f2bb12978202699c193 P d870164263bb44b6b780a557daea3656: stopping tablet replica
I20260130 02:58:10.559865 19496 tablet_service.cc:1552] Processing DeleteTablet for tablet 730bd1d7d15b4cbe945b3ef03b2455f7 with delete_type TABLET_DATA_DELETED (replaced by tablet 3b02480d9c6f4328af83f7940716d698) from {username='slave'} at 127.0.0.1:59068
I20260130 02:58:10.559918 19947 raft_consensus.cc:2243] T 3cf5154d8cc44f2bb12978202699c193 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260130 02:58:10.560618 19947 raft_consensus.cc:2272] T 3cf5154d8cc44f2bb12978202699c193 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260130 02:58:10.562314 19947 ts_tablet_manager.cc:1916] T 3cf5154d8cc44f2bb12978202699c193 P d870164263bb44b6b780a557daea3656: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260130 02:58:10.545702 18872 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50284:
table { table_name: "default.Table_bc11cb1bce0348ed9ea0912b3ee521b9" } modify_external_catalogs: true
I20260130 02:58:10.563215 19462 leader_election.cc:304] T 7d96e93d95f14af79cfb268626559b1e P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d870164263bb44b6b780a557daea3656; no voters: 371a36d219164974abdf848841528152, 78c2c46c7520492db35d51c244fb4f74
I20260130 02:58:10.564680 20371 raft_consensus.cc:2749] T 7d96e93d95f14af79cfb268626559b1e P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260130 02:58:10.568593 20422 ts_tablet_manager.cc:1916] T 730bd1d7d15b4cbe945b3ef03b2455f7 P 371a36d219164974abdf848841528152: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260130 02:58:10.580566 19840 tablet_bootstrap.cc:492] T d27ca7dd87f44102b7680e1f4a88954a P 78c2c46c7520492db35d51c244fb4f74: Bootstrap starting.
I20260130 02:58:10.583151 18872 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50284:
table { table_name: "default.Table_bc11cb1bce0348ed9ea0912b3ee521b9" } modify_external_catalogs: true
W20260130 02:58:10.554524 18870 catalog_manager.cc:5846] Failed to send AlterTable RPC for tablet 127a84346b9f473d9e7f739b1e65feb8 (table default.Table_4f2d6e4d9c8e465087d65efc9c2de442 [id=1515db389f284e2f9ed1a3cd0061d0f1], current schema version=5): Not found: failed to reset TS proxy: No leader found: Tablet 127a84346b9f473d9e7f739b1e65feb8 consensus state has no leader
W20260130 02:58:10.560714 18860 catalog_manager.cc:4952] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): delete failed for tablet 3b4908bd1a634ca0907234dd3c39e1c3 because the tablet was not found. No further retry: Not found: Tablet not found: 3b4908bd1a634ca0907234dd3c39e1c3
I20260130 02:58:10.587651 19358 tablet_service.cc:1552] Processing DeleteTablet for tablet 7d96e93d95f14af79cfb268626559b1e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:58:04 UTC) from {username='slave'} at 127.0.0.1:60110
I20260130 02:58:10.593539 20371 raft_consensus.cc:493] T b2ac91da01954c2390b6b7dcc06c394f P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:10.593668 19496 tablet_service.cc:1552] Processing DeleteTablet for tablet 5ca9919f80954e74a4d91c94b74de0e4 with delete_type TABLET_DATA_DELETED (replaced by tablet 13f0ace4d9de43abab08eca32cdb30c1) from {username='slave'} at 127.0.0.1:59068
I20260130 02:58:10.594005 20371 raft_consensus.cc:515] T b2ac91da01954c2390b6b7dcc06c394f P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
W20260130 02:58:10.595207 18870 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.18.100.66:59691 (request call id 28) took 1684 ms (client timeout 1000 ms). Trace:
I20260130 02:58:10.598901 20371 leader_election.cc:290] T b2ac91da01954c2390b6b7dcc06c394f P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767)
I20260130 02:58:10.599336 19378 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b2ac91da01954c2390b6b7dcc06c394f" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74" is_pre_election: true
I20260130 02:58:10.600064 19496 tablet_service.cc:1552] Processing DeleteTablet for tablet 1bbc55bbf2254d8bb6b721d7e8f69d85 with delete_type TABLET_DATA_DELETED (replaced by tablet 657e9b0d8c05413f904129dce745710b) from {username='slave'} at 127.0.0.1:59068
I20260130 02:58:10.601338 19496 tablet_service.cc:1552] Processing DeleteTablet for tablet 6573198830ad45628e71952a7d9c00a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:58:03 UTC) from {username='slave'} at 127.0.0.1:59068
I20260130 02:58:10.601101 19225 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b2ac91da01954c2390b6b7dcc06c394f" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152" is_pre_election: true
I20260130 02:58:10.603438 19625 tablet_bootstrap.cc:492] T ee7c8ff92ca9465fa18be6eae63d78ca P d870164263bb44b6b780a557daea3656: No bootstrap required, opened a new log
I20260130 02:58:10.604059 19463 leader_election.cc:304] T b2ac91da01954c2390b6b7dcc06c394f P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d870164263bb44b6b780a557daea3656; no voters: 371a36d219164974abdf848841528152, 78c2c46c7520492db35d51c244fb4f74
I20260130 02:58:10.604240 19625 ts_tablet_manager.cc:1403] T ee7c8ff92ca9465fa18be6eae63d78ca P d870164263bb44b6b780a557daea3656: Time spent bootstrapping tablet: real 0.141s user 0.008s sys 0.002s
I20260130 02:58:10.605201 20371 raft_consensus.cc:2749] T b2ac91da01954c2390b6b7dcc06c394f P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260130 02:58:10.607875 19625 raft_consensus.cc:359] T ee7c8ff92ca9465fa18be6eae63d78ca P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:10.612459 19205 tablet_service.cc:1552] Processing DeleteTablet for tablet b2ac91da01954c2390b6b7dcc06c394f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:58:04 UTC) from {username='slave'} at 127.0.0.1:42870
I20260130 02:58:10.623270 19840 tablet_bootstrap.cc:654] T d27ca7dd87f44102b7680e1f4a88954a P 78c2c46c7520492db35d51c244fb4f74: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:10.631461 19625 raft_consensus.cc:385] T ee7c8ff92ca9465fa18be6eae63d78ca P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:10.634537 19625 raft_consensus.cc:740] T ee7c8ff92ca9465fa18be6eae63d78ca P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d870164263bb44b6b780a557daea3656, State: Initialized, Role: FOLLOWER
I20260130 02:58:10.635028 19205 tablet_service.cc:1552] Processing DeleteTablet for tablet 1bbc55bbf2254d8bb6b721d7e8f69d85 with delete_type TABLET_DATA_DELETED (replaced by tablet 657e9b0d8c05413f904129dce745710b) from {username='slave'} at 127.0.0.1:42870
I20260130 02:58:10.639801 20374 raft_consensus.cc:493] T d55291d7cea64a0d9a345ce712fbfa80 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:10.640300 20374 raft_consensus.cc:515] T d55291d7cea64a0d9a345ce712fbfa80 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:10.640089 19625 consensus_queue.cc:260] T ee7c8ff92ca9465fa18be6eae63d78ca P d870164263bb44b6b780a557daea3656 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:10.642205 20374 leader_election.cc:290] T d55291d7cea64a0d9a345ce712fbfa80 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), d870164263bb44b6b780a557daea3656 (127.18.100.67:39571)
I20260130 02:58:10.644601 19525 raft_consensus.cc:3060] T b2ac91da01954c2390b6b7dcc06c394f P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:10.645287 19523 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d55291d7cea64a0d9a345ce712fbfa80" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
I20260130 02:58:10.648447 19222 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d55291d7cea64a0d9a345ce712fbfa80" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152" is_pre_election: true
I20260130 02:58:10.649029 19222 raft_consensus.cc:2468] T d55291d7cea64a0d9a345ce712fbfa80 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78c2c46c7520492db35d51c244fb4f74 in term 0.
I20260130 02:58:10.650717 19314 leader_election.cc:304] T d55291d7cea64a0d9a345ce712fbfa80 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 371a36d219164974abdf848841528152, 78c2c46c7520492db35d51c244fb4f74; no voters:
I20260130 02:58:10.651510 19625 ts_tablet_manager.cc:1434] T ee7c8ff92ca9465fa18be6eae63d78ca P d870164263bb44b6b780a557daea3656: Time spent starting tablet: real 0.046s user 0.002s sys 0.002s
I20260130 02:58:10.651523 20383 raft_consensus.cc:2804] T d55291d7cea64a0d9a345ce712fbfa80 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260130 02:58:10.651844 20383 raft_consensus.cc:493] T d55291d7cea64a0d9a345ce712fbfa80 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260130 02:58:10.652138 20383 raft_consensus.cc:3060] T d55291d7cea64a0d9a345ce712fbfa80 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Advancing to term 1
W20260130 02:58:10.547456 18869 rpcz_store.cc:269] 0130 02:58:07.477698 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:58:07.504853 (+ 27155us) service_pool.cc:225] Handling call
0130 02:58:10.527671 (+3022818us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0130 02:58:08.470926 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0130 02:58:08.471395 (+ 469us) write_op.cc:432] Acquiring schema lock in shared mode
0130 02:58:08.471425 (+ 30us) write_op.cc:435] Acquired schema lock
0130 02:58:08.471441 (+ 16us) tablet.cc:660] Decoding operations
0130 02:58:08.471722 (+ 281us) write_op.cc:620] Acquiring the partition lock for write op
0130 02:58:08.471760 (+ 38us) write_op.cc:641] Partition lock acquired for write op
0130 02:58:08.471785 (+ 25us) tablet.cc:683] Acquiring locks for 1 operations
0130 02:58:08.472512 (+ 727us) tablet.cc:699] Row locks acquired
0130 02:58:08.472531 (+ 19us) write_op.cc:260] PREPARE: finished
0130 02:58:08.472650 (+ 119us) write_op.cc:270] Start()
0130 02:58:08.472762 (+ 112us) write_op.cc:276] Timestamp: P: 1769741888472571 usec, L: 0
0130 02:58:08.472780 (+ 18us) op_driver.cc:348] REPLICATION: starting
0130 02:58:08.473293 (+ 513us) log.cc:844] Serialized 533 byte log entry
0130 02:58:08.673192 (+199899us) op_driver.cc:464] REPLICATION: finished
0130 02:58:08.676615 (+ 3423us) write_op.cc:301] APPLY: starting
0130 02:58:08.676665 (+ 50us) tablet.cc:1365] starting BulkCheckPresence
0130 02:58:08.676745 (+ 80us) tablet.cc:1368] finished BulkCheckPresence
0130 02:58:08.676824 (+ 79us) tablet.cc:1370] starting ApplyRowOperation cycle
0130 02:58:08.676976 (+ 152us) tablet.cc:1381] finished ApplyRowOperation cycle
0130 02:58:08.677089 (+ 113us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0130 02:58:08.677128 (+ 39us) write_op.cc:312] APPLY: finished
0130 02:58:08.677284 (+ 156us) log.cc:844] Serialized 23 byte log entry
0130 02:58:08.677398 (+ 114us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:58:08.677447 (+ 49us) write_op.cc:454] Released schema lock
0130 02:58:08.677474 (+ 27us) write_op.cc:341] FINISH: Updating metrics
0130 02:58:08.677682 (+ 208us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:58:08.677703 (+ 21us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":10846,"reported_tablets":30,"child_traces":[["op",{"apply.queue_time_us":3302,"apply.run_cpu_time_us":1036,"apply.run_wall_time_us":1037,"num_ops":1,"prepare.queue_time_us":85,"prepare.run_cpu_time_us":2649,"prepare.run_wall_time_us":2649,"raft.queue_time_us":1197,"raft.run_cpu_time_us":14,"raft.run_wall_time_us":14,"replication_time_us":200340}]]}
W20260130 02:58:10.655362 19313 leader_election.cc:343] T d55291d7cea64a0d9a345ce712fbfa80 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:10.656586 18869 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50284:
table { table_name: "default.Table_8edea1fae34d4d18b54b0f801a145225" } modify_external_catalogs: true
I20260130 02:58:10.657086 18869 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50284:
table { table_name: "default.Table_8edea1fae34d4d18b54b0f801a145225" } modify_external_catalogs: true
W20260130 02:58:10.658443 19313 consensus_peers.cc:597] T 04f4ac27800c4ac3a79ae12d35479757 P 78c2c46c7520492db35d51c244fb4f74 -> Peer d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): Couldn't send request to peer d870164263bb44b6b780a557daea3656. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20260130 02:58:10.663733 18871 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.18.100.65:51021 (request call id 31) took 2059 ms (client timeout 1000 ms). Trace:
I20260130 02:58:10.663578 20383 raft_consensus.cc:515] T d55291d7cea64a0d9a345ce712fbfa80 P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:10.666739 19225 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d55291d7cea64a0d9a345ce712fbfa80" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152"
I20260130 02:58:10.666934 20383 leader_election.cc:290] T d55291d7cea64a0d9a345ce712fbfa80 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 election: Requested vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), d870164263bb44b6b780a557daea3656 (127.18.100.67:39571)
I20260130 02:58:10.667268 19225 raft_consensus.cc:3060] T d55291d7cea64a0d9a345ce712fbfa80 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:10.668357 19526 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d55291d7cea64a0d9a345ce712fbfa80" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656"
W20260130 02:58:10.538280 18868 rpcz_store.cc:269] 0130 02:58:06.519883 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:58:06.964013 (+444130us) service_pool.cc:225] Handling call
0130 02:58:10.507706 (+3543693us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0130 02:58:07.570355 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0130 02:58:07.570824 (+ 469us) write_op.cc:432] Acquiring schema lock in shared mode
0130 02:58:07.570852 (+ 28us) write_op.cc:435] Acquired schema lock
0130 02:58:07.570867 (+ 15us) tablet.cc:660] Decoding operations
0130 02:58:07.571137 (+ 270us) write_op.cc:620] Acquiring the partition lock for write op
0130 02:58:07.571178 (+ 41us) write_op.cc:641] Partition lock acquired for write op
0130 02:58:07.571203 (+ 25us) tablet.cc:683] Acquiring locks for 1 operations
0130 02:58:07.571347 (+ 144us) tablet.cc:699] Row locks acquired
0130 02:58:07.571363 (+ 16us) write_op.cc:260] PREPARE: finished
0130 02:58:07.571477 (+ 114us) write_op.cc:270] Start()
0130 02:58:07.571561 (+ 84us) write_op.cc:276] Timestamp: P: 1769741887571382 usec, L: 0
0130 02:58:07.571579 (+ 18us) op_driver.cc:348] REPLICATION: starting
0130 02:58:07.572076 (+ 497us) log.cc:844] Serialized 533 byte log entry
0130 02:58:07.877391 (+305315us) op_driver.cc:464] REPLICATION: finished
0130 02:58:07.891046 (+ 13655us) write_op.cc:301] APPLY: starting
0130 02:58:07.891110 (+ 64us) tablet.cc:1365] starting BulkCheckPresence
0130 02:58:07.891189 (+ 79us) tablet.cc:1368] finished BulkCheckPresence
0130 02:58:07.891291 (+ 102us) tablet.cc:1370] starting ApplyRowOperation cycle
0130 02:58:07.891448 (+ 157us) tablet.cc:1381] finished ApplyRowOperation cycle
0130 02:58:07.891579 (+ 131us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0130 02:58:07.891617 (+ 38us) write_op.cc:312] APPLY: finished
0130 02:58:07.891777 (+ 160us) log.cc:844] Serialized 23 byte log entry
0130 02:58:07.891931 (+ 154us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:58:07.891987 (+ 56us) write_op.cc:454] Released schema lock
0130 02:58:07.892016 (+ 29us) write_op.cc:341] FINISH: Updating metrics
Related trace 'op':
0130 02:58:07.919089 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0130 02:58:07.919521 (+ 432us) write_op.cc:432] Acquiring schema lock in shared mode
0130 02:58:07.919548 (+ 27us) write_op.cc:435] Acquired schema lock
0130 02:58:07.919563 (+ 15us) tablet.cc:660] Decoding operations
0130 02:58:07.919828 (+ 265us) write_op.cc:620] Acquiring the partition lock for write op
0130 02:58:07.919877 (+ 49us) write_op.cc:641] Partition lock acquired for write op
0130 02:58:07.919902 (+ 25us) tablet.cc:683] Acquiring locks for 1 operations
0130 02:58:07.920046 (+ 144us) tablet.cc:699] Row locks acquired
0130 02:58:07.920063 (+ 17us) write_op.cc:260] PREPARE: finished
0130 02:58:07.920171 (+ 108us) write_op.cc:270] Start()
0130 02:58:07.920255 (+ 84us) write_op.cc:276] Timestamp: P: 1769741887920098 usec, L: 0
0130 02:58:07.920273 (+ 18us) op_driver.cc:348] REPLICATION: starting
0130 02:58:07.920746 (+ 473us) log.cc:844] Serialized 356 byte log entry
0130 02:58:08.168029 (+247283us) op_driver.cc:464] REPLICATION: finished
0130 02:58:08.182854 (+ 14825us) write_op.cc:301] APPLY: starting
0130 02:58:08.182906 (+ 52us) tablet.cc:1365] starting BulkCheckPresence
0130 02:58:08.182987 (+ 81us) tablet.cc:1368] finished BulkCheckPresence
0130 02:58:08.183070 (+ 83us) tablet.cc:1370] starting ApplyRowOperation cycle
0130 02:58:08.183208 (+ 138us) tablet.cc:1381] finished ApplyRowOperation cycle
0130 02:58:08.183331 (+ 123us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0130 02:58:08.183374 (+ 43us) write_op.cc:312] APPLY: finished
0130 02:58:08.183533 (+ 159us) log.cc:844] Serialized 23 byte log entry
0130 02:58:08.184117 (+ 584us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:58:08.184176 (+ 59us) write_op.cc:454] Released schema lock
0130 02:58:08.184207 (+ 31us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"reported_tablets":32,"child_traces":[["op",{"apply.queue_time_us":13497,"apply.run_cpu_time_us":1179,"apply.run_wall_time_us":1180,"num_ops":1,"prepare.queue_time_us":149,"prepare.run_cpu_time_us":2008,"prepare.run_wall_time_us":2009,"raft.queue_time_us":1139,"raft.run_cpu_time_us":536,"raft.run_wall_time_us":534,"replication_time_us":305712,"thread_start_us":14763,"threads_started":1}],["op",{"apply.queue_time_us":14704,"apply.run_cpu_time_us":1100,"apply.run_wall_time_us":1643,"num_ops":1,"prepare.queue_time_us":245,"prepare.run_cpu_time_us":1923,"prepare.run_wall_time_us":20134,"replication_time_us":247691,"thread_start_us":28501,"threads_started":1}]]}
W20260130 02:58:10.669533 19313 leader_election.cc:343] T d55291d7cea64a0d9a345ce712fbfa80 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:10.673178 18868 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50284:
name: "default.Table_68be882f76b74def9608a4147dd883d0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260130 02:58:10.673527 19225 raft_consensus.cc:2468] T d55291d7cea64a0d9a345ce712fbfa80 P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78c2c46c7520492db35d51c244fb4f74 in term 1.
W20260130 02:58:10.674834 18868 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_68be882f76b74def9608a4147dd883d0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260130 02:58:10.615002 18860 catalog_manager.cc:4952] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767): delete failed for tablet 3b4908bd1a634ca0907234dd3c39e1c3 because the tablet was not found. No further retry: Not found: Tablet not found: 3b4908bd1a634ca0907234dd3c39e1c3
W20260130 02:58:10.675427 18860 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.18.100.66:40767, user_credentials={real_user=slave}} blocked reactor thread for 60453.6us
W20260130 02:58:10.623971 18874 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.18.100.66:59691 (request call id 27) took 2763 ms (client timeout 1000 ms). Trace:
I20260130 02:58:10.677402 19314 leader_election.cc:304] T d55291d7cea64a0d9a345ce712fbfa80 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 371a36d219164974abdf848841528152, 78c2c46c7520492db35d51c244fb4f74; no voters: d870164263bb44b6b780a557daea3656
I20260130 02:58:10.678648 20374 raft_consensus.cc:2804] T d55291d7cea64a0d9a345ce712fbfa80 P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Leader election won for term 1
I20260130 02:58:10.679064 20374 raft_consensus.cc:697] T d55291d7cea64a0d9a345ce712fbfa80 P 78c2c46c7520492db35d51c244fb4f74 [term 1 LEADER]: Becoming Leader. State: Replica: 78c2c46c7520492db35d51c244fb4f74, State: Running, Role: LEADER
W20260130 02:58:10.679952 18867 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.18.100.67:47213 (request call id 24) took 2243 ms (client timeout 1000 ms). Trace:
I20260130 02:58:10.679826 20374 consensus_queue.cc:237] T d55291d7cea64a0d9a345ce712fbfa80 P 78c2c46c7520492db35d51c244fb4f74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:10.681600 20422 ts_tablet_manager.cc:1929] T 730bd1d7d15b4cbe945b3ef03b2455f7 P 371a36d219164974abdf848841528152: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260130 02:58:10.687815 20422 log.cc:1199] T 730bd1d7d15b4cbe945b3ef03b2455f7 P 371a36d219164974abdf848841528152: Deleting WAL directory at /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-0/wal/wals/730bd1d7d15b4cbe945b3ef03b2455f7
I20260130 02:58:10.692462 20422 ts_tablet_manager.cc:1950] T 730bd1d7d15b4cbe945b3ef03b2455f7 P 371a36d219164974abdf848841528152: Deleting consensus metadata
W20260130 02:58:10.694423 18860 catalog_manager.cc:4952] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767): delete failed for tablet 8bd3e1c06b89426da57e2b259e7983af because the tablet was not found. No further retry: Not found: Tablet not found: 8bd3e1c06b89426da57e2b259e7983af
I20260130 02:58:10.694837 18861 catalog_manager.cc:4985] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021): tablet 730bd1d7d15b4cbe945b3ef03b2455f7 (table default.Table_1e9e20a1bfec4bcf81f9eb6219c33d59 [id=f675b7a3ec744533903d8834eeed51d7]) successfully deleted
W20260130 02:58:10.695099 18860 catalog_manager.cc:4952] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767): delete failed for tablet 7d76046ee6c8420fa4964decbe8209c1 because the tablet was not found. No further retry: Not found: Tablet not found: 7d76046ee6c8420fa4964decbe8209c1
W20260130 02:58:10.695699 18860 catalog_manager.cc:4952] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767): delete failed for tablet 481bd4cca6ee4c178873bf12392e3441 because the tablet was not found. No further retry: Not found: Tablet not found: 481bd4cca6ee4c178873bf12392e3441
I20260130 02:58:10.699862 20422 tablet_replica.cc:333] T 6573198830ad45628e71952a7d9c00a0 P 371a36d219164974abdf848841528152: stopping tablet replica
W20260130 02:58:10.700109 18861 catalog_manager.cc:4952] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021): delete failed for tablet 3cf5154d8cc44f2bb12978202699c193 because the tablet was not found. No further retry: Not found: Tablet not found: 3cf5154d8cc44f2bb12978202699c193
I20260130 02:58:10.700829 19625 tablet_bootstrap.cc:492] T 127a84346b9f473d9e7f739b1e65feb8 P d870164263bb44b6b780a557daea3656: Bootstrap starting.
I20260130 02:58:10.702770 20426 tablet.cc:1722] T 3b148691090e4ee581d46f9845a6e49a P 371a36d219164974abdf848841528152: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20260130 02:58:10.705907 19625 tablet_bootstrap.cc:654] T 127a84346b9f473d9e7f739b1e65feb8 P d870164263bb44b6b780a557daea3656: Neither blocks nor log segments found. Creating new log.
W20260130 02:58:10.676478 18874 rpcz_store.cc:269] 0130 02:58:07.860332 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:58:08.062408 (+202076us) service_pool.cc:225] Handling call
0130 02:58:08.198605 (+136197us) spinlock_profiling.cc:243] Waited 129 ms on lock 0x7b5400030554. stack: 00007fb0bd3f7373 00007fb0bd3f7270 00007fb0bcf6c574 0000000000508744 00000000005086d9 00007fb0c44490c0 00007fb0c44e7bba 00007fb0c61e6eac 00007fb0c62e110e 00007fb0c3efe4a4 00007fb0c3efe432 00007fb0c3efe361 00007fb0c3efe2dc 00007fb0c3efd592 00007fb0c21dd34c 00007fb0c21dc786
0130 02:58:10.623772 (+2425167us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0130 02:58:09.657316 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0130 02:58:09.657759 (+ 443us) write_op.cc:432] Acquiring schema lock in shared mode
0130 02:58:09.657784 (+ 25us) write_op.cc:435] Acquired schema lock
0130 02:58:09.657797 (+ 13us) tablet.cc:660] Decoding operations
0130 02:58:09.658077 (+ 280us) write_op.cc:620] Acquiring the partition lock for write op
0130 02:58:09.668321 (+ 10244us) write_op.cc:641] Partition lock acquired for write op
0130 02:58:09.668350 (+ 29us) tablet.cc:683] Acquiring locks for 1 operations
0130 02:58:09.673280 (+ 4930us) tablet.cc:699] Row locks acquired
0130 02:58:09.673299 (+ 19us) write_op.cc:260] PREPARE: finished
0130 02:58:09.673422 (+ 123us) write_op.cc:270] Start()
0130 02:58:09.673517 (+ 95us) write_op.cc:276] Timestamp: P: 1769741889673391 usec, L: 0
0130 02:58:09.673536 (+ 19us) op_driver.cc:348] REPLICATION: starting
0130 02:58:09.674113 (+ 577us) log.cc:844] Serialized 533 byte log entry
0130 02:58:09.710070 (+ 35957us) op_driver.cc:464] REPLICATION: finished
0130 02:58:09.711930 (+ 1860us) write_op.cc:301] APPLY: starting
0130 02:58:09.711998 (+ 68us) tablet.cc:1365] starting BulkCheckPresence
0130 02:58:09.712093 (+ 95us) tablet.cc:1368] finished BulkCheckPresence
0130 02:58:09.712258 (+ 165us) tablet.cc:1370] starting ApplyRowOperation cycle
0130 02:58:09.712403 (+ 145us) tablet.cc:1381] finished ApplyRowOperation cycle
0130 02:58:09.712519 (+ 116us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0130 02:58:09.712555 (+ 36us) write_op.cc:312] APPLY: finished
0130 02:58:09.712740 (+ 185us) log.cc:844] Serialized 23 byte log entry
0130 02:58:09.712892 (+ 152us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:58:09.712946 (+ 54us) write_op.cc:454] Released schema lock
0130 02:58:09.712973 (+ 27us) write_op.cc:341] FINISH: Updating metrics
0130 02:58:09.713182 (+ 209us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:58:09.713205 (+ 23us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":5295,"reported_tablets":33,"spinlock_wait_cycles":297756928,"child_traces":[["op",{"apply.queue_time_us":1709,"apply.run_cpu_time_us":1218,"apply.run_wall_time_us":1218,"num_ops":1,"prepare.queue_time_us":709,"prepare.run_cpu_time_us":2827,"prepare.run_wall_time_us":26447,"raft.queue_time_us":2988,"raft.run_cpu_time_us":599,"raft.run_wall_time_us":1418,"replication_time_us":36431}]]}
I20260130 02:58:10.706153 20427 tablet.cc:1722] T 3b148691090e4ee581d46f9845a6e49a P 78c2c46c7520492db35d51c244fb4f74: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20260130 02:58:10.707608 18874 master_service.cc:954] ReplaceTablet: received request to replace tablet 3b148691090e4ee581d46f9845a6e49a from {username='slave'} at 127.0.0.1:50284
I20260130 02:58:10.684672 20428 tablet.cc:1722] T 3b148691090e4ee581d46f9845a6e49a P d870164263bb44b6b780a557daea3656: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20260130 02:58:10.714624 20422 raft_consensus.cc:2243] T 6573198830ad45628e71952a7d9c00a0 P 371a36d219164974abdf848841528152 [term 1 LEADER]: Raft consensus shutting down.
I20260130 02:58:10.717579 20422 raft_consensus.cc:2272] T 6573198830ad45628e71952a7d9c00a0 P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260130 02:58:10.719149 19947 ts_tablet_manager.cc:1929] T 3cf5154d8cc44f2bb12978202699c193 P d870164263bb44b6b780a557daea3656: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260130 02:58:10.719504 19947 log.cc:1199] T 3cf5154d8cc44f2bb12978202699c193 P d870164263bb44b6b780a557daea3656: Deleting WAL directory at /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-2/wal/wals/3cf5154d8cc44f2bb12978202699c193
I20260130 02:58:10.720121 19947 ts_tablet_manager.cc:1950] T 3cf5154d8cc44f2bb12978202699c193 P d870164263bb44b6b780a557daea3656: Deleting consensus metadata
W20260130 02:58:10.721220 18860 catalog_manager.cc:4952] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767): delete failed for tablet 3cf5154d8cc44f2bb12978202699c193 because the tablet was not found. No further retry: Not found: Tablet not found: 3cf5154d8cc44f2bb12978202699c193
W20260130 02:58:10.721830 18860 catalog_manager.cc:4952] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767): delete failed for tablet 927c90b84d7449efa36bc5e2035bf0da because the tablet was not found. No further retry: Not found: Tablet not found: 927c90b84d7449efa36bc5e2035bf0da
W20260130 02:58:10.722361 18860 catalog_manager.cc:4952] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767): delete failed for tablet 2112b0402de5464cb1caf2c2abb67d78 because the tablet was not found. No further retry: Not found: Tablet not found: 2112b0402de5464cb1caf2c2abb67d78
W20260130 02:58:10.722935 18860 catalog_manager.cc:4952] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767): delete failed for tablet 730bd1d7d15b4cbe945b3ef03b2455f7 because the tablet was not found. No further retry: Not found: Tablet not found: 730bd1d7d15b4cbe945b3ef03b2455f7
I20260130 02:58:10.733738 20422 ts_tablet_manager.cc:1916] T 6573198830ad45628e71952a7d9c00a0 P 371a36d219164974abdf848841528152: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260130 02:58:10.647696 18870 rpcz_store.cc:269] 0130 02:58:08.910443 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:58:08.955615 (+ 45172us) service_pool.cc:225] Handling call
0130 02:58:09.001154 (+ 45539us) spinlock_profiling.cc:243] Waited 24 ms on lock 0x7b5400030554. stack: 00007fb0bd3f7373 00007fb0bd3f7270 00007fb0bcf6c574 0000000000508744 00000000005086d9 00007fb0c44490c0 00007fb0c44e7bba 00007fb0c61e6eac 00007fb0c62e110e 00007fb0c3efe4a4 00007fb0c3efe432 00007fb0c3efe361 00007fb0c3efe2dc 00007fb0c3efd592 00007fb0c21dd34c 00007fb0c21dc786
0130 02:58:10.595115 (+1593961us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0130 02:58:10.234989 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0130 02:58:10.235470 (+ 481us) write_op.cc:432] Acquiring schema lock in shared mode
0130 02:58:10.235501 (+ 31us) write_op.cc:435] Acquired schema lock
0130 02:58:10.235514 (+ 13us) tablet.cc:660] Decoding operations
0130 02:58:10.235911 (+ 397us) write_op.cc:620] Acquiring the partition lock for write op
0130 02:58:10.235953 (+ 42us) write_op.cc:641] Partition lock acquired for write op
0130 02:58:10.235980 (+ 27us) tablet.cc:683] Acquiring locks for 2 operations
0130 02:58:10.236290 (+ 310us) tablet.cc:699] Row locks acquired
0130 02:58:10.236311 (+ 21us) write_op.cc:260] PREPARE: finished
0130 02:58:10.236436 (+ 125us) write_op.cc:270] Start()
0130 02:58:10.236531 (+ 95us) write_op.cc:276] Timestamp: P: 1769741890236343 usec, L: 0
0130 02:58:10.236552 (+ 21us) op_driver.cc:348] REPLICATION: starting
0130 02:58:10.237114 (+ 562us) log.cc:844] Serialized 913 byte log entry
0130 02:58:10.243374 (+ 6260us) op_driver.cc:464] REPLICATION: finished
0130 02:58:10.243759 (+ 385us) write_op.cc:301] APPLY: starting
0130 02:58:10.243814 (+ 55us) tablet.cc:1365] starting BulkCheckPresence
0130 02:58:10.243916 (+ 102us) tablet.cc:1368] finished BulkCheckPresence
0130 02:58:10.244006 (+ 90us) tablet.cc:1370] starting ApplyRowOperation cycle
0130 02:58:10.244290 (+ 284us) tablet.cc:1381] finished ApplyRowOperation cycle
0130 02:58:10.244428 (+ 138us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0130 02:58:10.244470 (+ 42us) write_op.cc:312] APPLY: finished
0130 02:58:10.244648 (+ 178us) log.cc:844] Serialized 29 byte log entry
0130 02:58:10.244781 (+ 133us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:58:10.244847 (+ 66us) write_op.cc:454] Released schema lock
0130 02:58:10.244881 (+ 34us) write_op.cc:341] FINISH: Updating metrics
0130 02:58:10.245078 (+ 197us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:58:10.245100 (+ 22us) write_op.cc:454] Released schema lock
Related trace 'op':
0130 02:58:10.305825 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0130 02:58:10.306312 (+ 487us) write_op.cc:432] Acquiring schema lock in shared mode
0130 02:58:10.306344 (+ 32us) write_op.cc:435] Acquired schema lock
0130 02:58:10.306361 (+ 17us) tablet.cc:660] Decoding operations
0130 02:58:10.306689 (+ 328us) write_op.cc:620] Acquiring the partition lock for write op
0130 02:58:10.306745 (+ 56us) write_op.cc:641] Partition lock acquired for write op
0130 02:58:10.306772 (+ 27us) tablet.cc:683] Acquiring locks for 1 operations
0130 02:58:10.306933 (+ 161us) tablet.cc:699] Row locks acquired
0130 02:58:10.306950 (+ 17us) write_op.cc:260] PREPARE: finished
0130 02:58:10.307080 (+ 130us) write_op.cc:270] Start()
0130 02:58:10.307181 (+ 101us) write_op.cc:276] Timestamp: P: 1769741890306977 usec, L: 0
0130 02:58:10.307195 (+ 14us) op_driver.cc:348] REPLICATION: starting
0130 02:58:10.307740 (+ 545us) log.cc:844] Serialized 356 byte log entry
0130 02:58:10.315206 (+ 7466us) op_driver.cc:464] REPLICATION: finished
0130 02:58:10.315602 (+ 396us) write_op.cc:301] APPLY: starting
0130 02:58:10.315662 (+ 60us) tablet.cc:1365] starting BulkCheckPresence
0130 02:58:10.315747 (+ 85us) tablet.cc:1368] finished BulkCheckPresence
0130 02:58:10.315838 (+ 91us) tablet.cc:1370] starting ApplyRowOperation cycle
0130 02:58:10.316033 (+ 195us) tablet.cc:1381] finished ApplyRowOperation cycle
0130 02:58:10.316166 (+ 133us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0130 02:58:10.316206 (+ 40us) write_op.cc:312] APPLY: finished
0130 02:58:10.316365 (+ 159us) log.cc:844] Serialized 23 byte log entry
0130 02:58:10.316506 (+ 141us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:58:10.316565 (+ 59us) write_op.cc:454] Released schema lock
0130 02:58:10.316594 (+ 29us) write_op.cc:341] FINISH: Updating metrics
0130 02:58:10.316795 (+ 201us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:58:10.316819 (+ 24us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":16972,"reported_tablets":35,"spinlock_wait_cycles":55260032,"child_traces":[["op",{"apply.queue_time_us":240,"apply.run_cpu_time_us":1288,"apply.run_wall_time_us":1287,"num_ops":2,"prepare.queue_time_us":81,"prepare.run_cpu_time_us":2448,"prepare.run_wall_time_us":2448,"raft.queue_time_us":449,"raft.run_cpu_time_us":671,"raft.run_wall_time_us":670,"replication_time_us":6728}],["op",{"apply.queue_time_us":264,"apply.run_cpu_time_us":1157,"apply.run_wall_time_us":1157,"num_ops":1,"prepare.queue_time_us":11490,"prepare.run_cpu_time_us":2289,"prepare.run_wall_time_us":2289,"raft.queue_time_us":2297,"raft.run_cpu_time_us":643,"raft.run_wall_time_us":643,"replication_time_us":7915}]]}
I20260130 02:58:10.746592 19840 tablet_bootstrap.cc:492] T d27ca7dd87f44102b7680e1f4a88954a P 78c2c46c7520492db35d51c244fb4f74: No bootstrap required, opened a new log
I20260130 02:58:10.747078 19840 ts_tablet_manager.cc:1403] T d27ca7dd87f44102b7680e1f4a88954a P 78c2c46c7520492db35d51c244fb4f74: Time spent bootstrapping tablet: real 0.204s user 0.006s sys 0.019s
W20260130 02:58:10.680313 18867 rpcz_store.cc:269] 0130 02:58:08.436118 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:58:08.436253 (+ 135us) service_pool.cc:225] Handling call
0130 02:58:08.441645 (+ 5392us) spinlock_profiling.cc:243] Waited 1.45 ms on lock 0x7b5400030554. stack: 00007fb0bd3f7373 00007fb0bd3f7270 00007fb0bcf6c574 0000000000508744 00000000005086d9 00007fb0c44490c0 00007fb0c44e7bba 00007fb0c61e6eac 00007fb0c62e110e 00007fb0c3efe4a4 00007fb0c3efe432 00007fb0c3efe361 00007fb0c3efe2dc 00007fb0c3efd592 00007fb0c21dd34c 00007fb0c21dc786
0130 02:58:10.679867 (+2238222us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0130 02:58:09.854349 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0130 02:58:09.854822 (+ 473us) write_op.cc:432] Acquiring schema lock in shared mode
0130 02:58:09.854848 (+ 26us) write_op.cc:435] Acquired schema lock
0130 02:58:09.854868 (+ 20us) tablet.cc:660] Decoding operations
0130 02:58:09.855152 (+ 284us) write_op.cc:620] Acquiring the partition lock for write op
0130 02:58:09.855198 (+ 46us) write_op.cc:641] Partition lock acquired for write op
0130 02:58:09.855222 (+ 24us) tablet.cc:683] Acquiring locks for 1 operations
0130 02:58:09.855375 (+ 153us) tablet.cc:699] Row locks acquired
0130 02:58:09.855388 (+ 13us) write_op.cc:260] PREPARE: finished
0130 02:58:09.855508 (+ 120us) write_op.cc:270] Start()
0130 02:58:09.855599 (+ 91us) write_op.cc:276] Timestamp: P: 1769741889855455 usec, L: 0
0130 02:58:09.855613 (+ 14us) op_driver.cc:348] REPLICATION: starting
0130 02:58:09.856134 (+ 521us) log.cc:844] Serialized 533 byte log entry
0130 02:58:09.877828 (+ 21694us) op_driver.cc:464] REPLICATION: finished
0130 02:58:09.878172 (+ 344us) write_op.cc:301] APPLY: starting
0130 02:58:09.878226 (+ 54us) tablet.cc:1365] starting BulkCheckPresence
0130 02:58:09.878310 (+ 84us) tablet.cc:1368] finished BulkCheckPresence
0130 02:58:09.878408 (+ 98us) tablet.cc:1370] starting ApplyRowOperation cycle
0130 02:58:09.878560 (+ 152us) tablet.cc:1381] finished ApplyRowOperation cycle
0130 02:58:09.878701 (+ 141us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0130 02:58:09.878745 (+ 44us) write_op.cc:312] APPLY: finished
0130 02:58:09.878937 (+ 192us) log.cc:844] Serialized 23 byte log entry
0130 02:58:09.879081 (+ 144us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:58:09.879144 (+ 63us) write_op.cc:454] Released schema lock
0130 02:58:09.879173 (+ 29us) write_op.cc:341] FINISH: Updating metrics
0130 02:58:09.879379 (+ 206us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:58:09.879403 (+ 24us) write_op.cc:454] Released schema lock
Related trace 'op':
0130 02:58:09.888059 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0130 02:58:09.888501 (+ 442us) write_op.cc:432] Acquiring schema lock in shared mode
0130 02:58:09.888527 (+ 26us) write_op.cc:435] Acquired schema lock
0130 02:58:09.888543 (+ 16us) tablet.cc:660] Decoding operations
0130 02:58:09.888808 (+ 265us) write_op.cc:620] Acquiring the partition lock for write op
0130 02:58:09.888852 (+ 44us) write_op.cc:641] Partition lock acquired for write op
0130 02:58:09.888876 (+ 24us) tablet.cc:683] Acquiring locks for 1 operations
0130 02:58:09.889023 (+ 147us) tablet.cc:699] Row locks acquired
0130 02:58:09.889039 (+ 16us) write_op.cc:260] PREPARE: finished
0130 02:58:09.889162 (+ 123us) write_op.cc:270] Start()
0130 02:58:09.889247 (+ 85us) write_op.cc:276] Timestamp: P: 1769741889889109 usec, L: 0
0130 02:58:09.889261 (+ 14us) op_driver.cc:348] REPLICATION: starting
0130 02:58:09.889773 (+ 512us) log.cc:844] Serialized 356 byte log entry
0130 02:58:09.895450 (+ 5677us) op_driver.cc:464] REPLICATION: finished
0130 02:58:09.895881 (+ 431us) write_op.cc:301] APPLY: starting
0130 02:58:09.895929 (+ 48us) tablet.cc:1365] starting BulkCheckPresence
0130 02:58:09.896016 (+ 87us) tablet.cc:1368] finished BulkCheckPresence
0130 02:58:09.896114 (+ 98us) tablet.cc:1370] starting ApplyRowOperation cycle
0130 02:58:09.896268 (+ 154us) tablet.cc:1381] finished ApplyRowOperation cycle
0130 02:58:09.896389 (+ 121us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0130 02:58:09.896430 (+ 41us) write_op.cc:312] APPLY: finished
0130 02:58:09.896590 (+ 160us) log.cc:844] Serialized 23 byte log entry
0130 02:58:09.896718 (+ 128us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:58:09.896781 (+ 63us) write_op.cc:454] Released schema lock
0130 02:58:09.896812 (+ 31us) write_op.cc:341] FINISH: Updating metrics
0130 02:58:09.897030 (+ 218us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:58:09.897054 (+ 24us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":28,"spinlock_wait_cycles":3324160,"child_traces":[["op",{"apply.queue_time_us":194,"apply.run_cpu_time_us":1173,"apply.run_wall_time_us":1173,"num_ops":1,"prepare.queue_time_us":2789,"prepare.run_cpu_time_us":2084,"prepare.run_wall_time_us":3028,"raft.queue_time_us":1562,"raft.run_cpu_time_us":568,"raft.run_wall_time_us":567,"replication_time_us":22126}],["op",{"apply.queue_time_us":289,"apply.run_cpu_time_us":1124,"apply.run_wall_time_us":1123,"mutex_wait_us":140,"num_ops":1,"prepare.queue_time_us":99,"prepare.run_cpu_time_us":2173,"prepare.run_wall_time_us":2301,"raft.queue_time_us":408,"raft.run_cpu_time_us":600,"raft.run_wall_time_us":601,"replication_time_us":6101}]]}
I20260130 02:58:10.766600 20414 tablet_replica.cc:333] T 6573198830ad45628e71952a7d9c00a0 P 78c2c46c7520492db35d51c244fb4f74: stopping tablet replica
I20260130 02:58:10.768625 20424 raft_consensus.cc:493] T 3593f8b89da24ff9b6cf549d5ac58bc6 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:10.769091 20424 raft_consensus.cc:515] T 3593f8b89da24ff9b6cf549d5ac58bc6 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:10.775848 19223 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3593f8b89da24ff9b6cf549d5ac58bc6" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152" is_pre_election: true
I20260130 02:58:10.777631 20424 leader_election.cc:290] T 3593f8b89da24ff9b6cf549d5ac58bc6 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767)
I20260130 02:58:10.779539 19625 tablet_bootstrap.cc:492] T 127a84346b9f473d9e7f739b1e65feb8 P d870164263bb44b6b780a557daea3656: No bootstrap required, opened a new log
I20260130 02:58:10.779945 19625 ts_tablet_manager.cc:1403] T 127a84346b9f473d9e7f739b1e65feb8 P d870164263bb44b6b780a557daea3656: Time spent bootstrapping tablet: real 0.079s user 0.010s sys 0.005s
I20260130 02:58:10.783020 19625 raft_consensus.cc:359] T 127a84346b9f473d9e7f739b1e65feb8 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:10.783613 19625 raft_consensus.cc:385] T 127a84346b9f473d9e7f739b1e65feb8 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:10.783852 19625 raft_consensus.cc:740] T 127a84346b9f473d9e7f739b1e65feb8 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d870164263bb44b6b780a557daea3656, State: Initialized, Role: FOLLOWER
I20260130 02:58:10.784384 19625 consensus_queue.cc:260] T 127a84346b9f473d9e7f739b1e65feb8 P d870164263bb44b6b780a557daea3656 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:10.787524 20434 tablet.cc:1722] T 341041cb0be2456e88403367c0814704 P d870164263bb44b6b780a557daea3656: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260130 02:58:10.788169 19840 raft_consensus.cc:359] T d27ca7dd87f44102b7680e1f4a88954a P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:10.790498 20243 consensus_queue.cc:1048] T b89ed298e3dc49d789d166c613ad7fb8 P 371a36d219164974abdf848841528152 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260130 02:58:10.792055 19375 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3593f8b89da24ff9b6cf549d5ac58bc6" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74" is_pre_election: true
I20260130 02:58:10.796403 20244 consensus_queue.cc:1048] T 127a84346b9f473d9e7f739b1e65feb8 P 371a36d219164974abdf848841528152 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260130 02:58:10.799288 19625 ts_tablet_manager.cc:1434] T 127a84346b9f473d9e7f739b1e65feb8 P d870164263bb44b6b780a557daea3656: Time spent starting tablet: real 0.019s user 0.007s sys 0.000s
I20260130 02:58:10.800191 19625 tablet_bootstrap.cc:492] T 04f4ac27800c4ac3a79ae12d35479757 P d870164263bb44b6b780a557daea3656: Bootstrap starting.
I20260130 02:58:10.801339 19525 raft_consensus.cc:3060] T 3593f8b89da24ff9b6cf549d5ac58bc6 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:10.810878 20427 tablet.cc:1722] T 341041cb0be2456e88403367c0814704 P 78c2c46c7520492db35d51c244fb4f74: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260130 02:58:10.814900 19462 leader_election.cc:304] T 3593f8b89da24ff9b6cf549d5ac58bc6 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d870164263bb44b6b780a557daea3656; no voters: 371a36d219164974abdf848841528152, 78c2c46c7520492db35d51c244fb4f74
I20260130 02:58:10.818300 19807 tablet_bootstrap.cc:492] T d27ca7dd87f44102b7680e1f4a88954a P 371a36d219164974abdf848841528152: No bootstrap required, opened a new log
I20260130 02:58:10.818769 19807 ts_tablet_manager.cc:1403] T d27ca7dd87f44102b7680e1f4a88954a P 371a36d219164974abdf848841528152: Time spent bootstrapping tablet: real 0.419s user 0.030s sys 0.018s
I20260130 02:58:10.823975 20414 raft_consensus.cc:2243] T 6573198830ad45628e71952a7d9c00a0 P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260130 02:58:10.664068 18871 rpcz_store.cc:269] 0130 02:58:08.604454 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:58:08.611088 (+ 6634us) service_pool.cc:225] Handling call
0130 02:58:10.663664 (+2052576us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0130 02:58:10.090922 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0130 02:58:10.091452 (+ 530us) write_op.cc:432] Acquiring schema lock in shared mode
0130 02:58:10.091488 (+ 36us) write_op.cc:435] Acquired schema lock
0130 02:58:10.091514 (+ 26us) tablet.cc:660] Decoding operations
0130 02:58:10.091919 (+ 405us) write_op.cc:620] Acquiring the partition lock for write op
0130 02:58:10.091982 (+ 63us) write_op.cc:641] Partition lock acquired for write op
0130 02:58:10.092008 (+ 26us) tablet.cc:683] Acquiring locks for 2 operations
0130 02:58:10.092365 (+ 357us) tablet.cc:699] Row locks acquired
0130 02:58:10.092391 (+ 26us) write_op.cc:260] PREPARE: finished
0130 02:58:10.092514 (+ 123us) write_op.cc:270] Start()
0130 02:58:10.092633 (+ 119us) write_op.cc:276] Timestamp: P: 1769741890092430 usec, L: 0
0130 02:58:10.092657 (+ 24us) op_driver.cc:348] REPLICATION: starting
0130 02:58:10.093374 (+ 717us) log.cc:844] Serialized 913 byte log entry
0130 02:58:10.100403 (+ 7029us) op_driver.cc:464] REPLICATION: finished
0130 02:58:10.100962 (+ 559us) write_op.cc:301] APPLY: starting
0130 02:58:10.101023 (+ 61us) tablet.cc:1365] starting BulkCheckPresence
0130 02:58:10.101135 (+ 112us) tablet.cc:1368] finished BulkCheckPresence
0130 02:58:10.101248 (+ 113us) tablet.cc:1370] starting ApplyRowOperation cycle
0130 02:58:10.101580 (+ 332us) tablet.cc:1381] finished ApplyRowOperation cycle
0130 02:58:10.101776 (+ 196us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0130 02:58:10.101821 (+ 45us) write_op.cc:312] APPLY: finished
0130 02:58:10.102034 (+ 213us) log.cc:844] Serialized 29 byte log entry
0130 02:58:10.102196 (+ 162us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:58:10.102261 (+ 65us) write_op.cc:454] Released schema lock
0130 02:58:10.102302 (+ 41us) write_op.cc:341] FINISH: Updating metrics
0130 02:58:10.102520 (+ 218us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:58:10.102543 (+ 23us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":188700,"reported_tablets":33,"child_traces":[["op",{"apply.queue_time_us":338,"apply.run_cpu_time_us":1561,"apply.run_wall_time_us":1561,"num_ops":2,"prepare.queue_time_us":544,"prepare.run_cpu_time_us":2876,"prepare.run_wall_time_us":2877,"raft.queue_time_us":224,"raft.run_cpu_time_us":790,"raft.run_wall_time_us":790,"replication_time_us":7607}]]}
I20260130 02:58:10.827134 18871 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50284:
table { table_name: "default.Table_9e52337cb913484caafe98b1e08c642e" } new_table_name: "default.Table_2e81f0a0cbc449a397141b717db17025" modify_external_catalogs: true
W20260130 02:58:10.776028 18860 catalog_manager.cc:4952] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767): delete failed for tablet 1bbc55bbf2254d8bb6b721d7e8f69d85 because the tablet was not found. No further retry: Not found: Tablet not found: 1bbc55bbf2254d8bb6b721d7e8f69d85
W20260130 02:58:10.829844 18860 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.18.100.66:40767, user_credentials={real_user=slave}} blocked reactor thread for 53844.2us
I20260130 02:58:10.830557 19840 raft_consensus.cc:385] T d27ca7dd87f44102b7680e1f4a88954a P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:10.838711 19840 raft_consensus.cc:740] T d27ca7dd87f44102b7680e1f4a88954a P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78c2c46c7520492db35d51c244fb4f74, State: Initialized, Role: FOLLOWER
I20260130 02:58:10.839469 19840 consensus_queue.cc:260] T d27ca7dd87f44102b7680e1f4a88954a P 78c2c46c7520492db35d51c244fb4f74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:10.842741 19840 ts_tablet_manager.cc:1434] T d27ca7dd87f44102b7680e1f4a88954a P 78c2c46c7520492db35d51c244fb4f74: Time spent starting tablet: real 0.095s user 0.006s sys 0.000s
I20260130 02:58:10.844221 19840 tablet_bootstrap.cc:492] T 16691c9315e94253bb9b3fb5058e5636 P 78c2c46c7520492db35d51c244fb4f74: Bootstrap starting.
I20260130 02:58:10.849962 18860 catalog_manager.cc:4985] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): tablet 3cf5154d8cc44f2bb12978202699c193 (table default.Table_bb334df0beee420e883c3225d93892c0 [id=b7b1d250632741a2ad0ed194aae9b197]) successfully deleted
I20260130 02:58:10.857445 19807 raft_consensus.cc:359] T d27ca7dd87f44102b7680e1f4a88954a P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:10.849542 20414 raft_consensus.cc:2272] T 6573198830ad45628e71952a7d9c00a0 P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260130 02:58:10.859616 20371 raft_consensus.cc:2749] T 3593f8b89da24ff9b6cf549d5ac58bc6 P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260130 02:58:10.861321 19625 tablet_bootstrap.cc:654] T 04f4ac27800c4ac3a79ae12d35479757 P d870164263bb44b6b780a557daea3656: Neither blocks nor log segments found. Creating new log.
W20260130 02:58:10.879662 18860 catalog_manager.cc:4952] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): delete failed for tablet 730bd1d7d15b4cbe945b3ef03b2455f7 because the tablet was not found. No further retry: Not found: Tablet not found: 730bd1d7d15b4cbe945b3ef03b2455f7
I20260130 02:58:10.882364 19521 raft_consensus.cc:3060] T 127a84346b9f473d9e7f739b1e65feb8 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:10.886443 20374 raft_consensus.cc:493] T a8a5038e9def44eb8b3b1e4aeae25652 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:10.886883 20374 raft_consensus.cc:515] T a8a5038e9def44eb8b3b1e4aeae25652 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:10.889680 20374 leader_election.cc:290] T a8a5038e9def44eb8b3b1e4aeae25652 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), d870164263bb44b6b780a557daea3656 (127.18.100.67:39571)
I20260130 02:58:10.860658 20422 ts_tablet_manager.cc:1929] T 6573198830ad45628e71952a7d9c00a0 P 371a36d219164974abdf848841528152: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260130 02:58:10.891937 19625 tablet_bootstrap.cc:492] T 04f4ac27800c4ac3a79ae12d35479757 P d870164263bb44b6b780a557daea3656: No bootstrap required, opened a new log
I20260130 02:58:10.892406 19625 ts_tablet_manager.cc:1403] T 04f4ac27800c4ac3a79ae12d35479757 P d870164263bb44b6b780a557daea3656: Time spent bootstrapping tablet: real 0.092s user 0.006s sys 0.003s
W20260130 02:58:10.892450 18860 catalog_manager.cc:4952] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): delete failed for tablet 5ca9919f80954e74a4d91c94b74de0e4 because the tablet was not found. No further retry: Not found: Tablet not found: 5ca9919f80954e74a4d91c94b74de0e4
W20260130 02:58:10.893018 18860 catalog_manager.cc:4952] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): delete failed for tablet 1bbc55bbf2254d8bb6b721d7e8f69d85 because the tablet was not found. No further retry: Not found: Tablet not found: 1bbc55bbf2254d8bb6b721d7e8f69d85
I20260130 02:58:10.893612 20422 log.cc:1199] T 6573198830ad45628e71952a7d9c00a0 P 371a36d219164974abdf848841528152: Deleting WAL directory at /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-0/wal/wals/6573198830ad45628e71952a7d9c00a0
I20260130 02:58:10.894387 20422 ts_tablet_manager.cc:1950] T 6573198830ad45628e71952a7d9c00a0 P 371a36d219164974abdf848841528152: Deleting consensus metadata
I20260130 02:58:10.894933 19625 raft_consensus.cc:359] T 04f4ac27800c4ac3a79ae12d35479757 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:10.896656 19625 raft_consensus.cc:385] T 04f4ac27800c4ac3a79ae12d35479757 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:10.898186 20414 ts_tablet_manager.cc:1916] T 6573198830ad45628e71952a7d9c00a0 P 78c2c46c7520492db35d51c244fb4f74: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260130 02:58:10.898586 19625 raft_consensus.cc:740] T 04f4ac27800c4ac3a79ae12d35479757 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d870164263bb44b6b780a557daea3656, State: Initialized, Role: FOLLOWER
I20260130 02:58:10.898939 19947 tablet_replica.cc:333] T 6573198830ad45628e71952a7d9c00a0 P d870164263bb44b6b780a557daea3656: stopping tablet replica
I20260130 02:58:10.899686 19947 raft_consensus.cc:2243] T 6573198830ad45628e71952a7d9c00a0 P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260130 02:58:10.900125 19947 raft_consensus.cc:2272] T 6573198830ad45628e71952a7d9c00a0 P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260130 02:58:10.900592 19625 consensus_queue.cc:260] T 04f4ac27800c4ac3a79ae12d35479757 P d870164263bb44b6b780a557daea3656 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:10.902537 20422 tablet_replica.cc:333] T f54addb1a4304ce69c711ae0cc3b452c P 371a36d219164974abdf848841528152: stopping tablet replica
I20260130 02:58:10.903432 20422 raft_consensus.cc:2243] T f54addb1a4304ce69c711ae0cc3b452c P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260130 02:58:10.903964 20422 raft_consensus.cc:2272] T f54addb1a4304ce69c711ae0cc3b452c P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260130 02:58:10.904865 18861 catalog_manager.cc:4985] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021): tablet 6573198830ad45628e71952a7d9c00a0 (table default.Table_1c2b251034c0418cba750ae4d1308b9f [id=9a5fe99c9417426dba0a12441e7191c6]) successfully deleted
I20260130 02:58:10.908897 20422 ts_tablet_manager.cc:1916] T f54addb1a4304ce69c711ae0cc3b452c P 371a36d219164974abdf848841528152: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260130 02:58:10.911720 19526 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a8a5038e9def44eb8b3b1e4aeae25652" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
I20260130 02:58:10.912367 19625 ts_tablet_manager.cc:1434] T 04f4ac27800c4ac3a79ae12d35479757 P d870164263bb44b6b780a557daea3656: Time spent starting tablet: real 0.020s user 0.004s sys 0.000s
I20260130 02:58:10.913205 19625 tablet_bootstrap.cc:492] T b89ed298e3dc49d789d166c613ad7fb8 P d870164263bb44b6b780a557daea3656: Bootstrap starting.
I20260130 02:58:10.918232 19625 tablet_bootstrap.cc:654] T b89ed298e3dc49d789d166c613ad7fb8 P d870164263bb44b6b780a557daea3656: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:10.921927 20244 consensus_queue.cc:1048] T 127a84346b9f473d9e7f739b1e65feb8 P 371a36d219164974abdf848841528152 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260130 02:58:10.863848 20426 tablet.cc:1722] T 341041cb0be2456e88403367c0814704 P 371a36d219164974abdf848841528152: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260130 02:58:10.889285 19225 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a8a5038e9def44eb8b3b1e4aeae25652" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152" is_pre_election: true
I20260130 02:58:10.890242 19807 raft_consensus.cc:385] T d27ca7dd87f44102b7680e1f4a88954a P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:10.932477 19807 raft_consensus.cc:740] T d27ca7dd87f44102b7680e1f4a88954a P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 371a36d219164974abdf848841528152, State: Initialized, Role: FOLLOWER
I20260130 02:58:10.933192 19807 consensus_queue.cc:260] T d27ca7dd87f44102b7680e1f4a88954a P 371a36d219164974abdf848841528152 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
W20260130 02:58:10.940003 19314 leader_election.cc:343] T a8a5038e9def44eb8b3b1e4aeae25652 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:10.964704 19807 ts_tablet_manager.cc:1434] T d27ca7dd87f44102b7680e1f4a88954a P 371a36d219164974abdf848841528152: Time spent starting tablet: real 0.145s user 0.004s sys 0.002s
I20260130 02:58:10.965610 19807 tablet_bootstrap.cc:492] T 753344737b5941d9a93b6779e72f63d4 P 371a36d219164974abdf848841528152: Bootstrap starting.
I20260130 02:58:10.970196 19947 ts_tablet_manager.cc:1916] T 6573198830ad45628e71952a7d9c00a0 P d870164263bb44b6b780a557daea3656: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260130 02:58:10.973181 19313 leader_election.cc:343] T a8a5038e9def44eb8b3b1e4aeae25652 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:10.973554 19313 leader_election.cc:304] T a8a5038e9def44eb8b3b1e4aeae25652 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78c2c46c7520492db35d51c244fb4f74; no voters: 371a36d219164974abdf848841528152, d870164263bb44b6b780a557daea3656
I20260130 02:58:10.974810 19625 tablet_bootstrap.cc:492] T b89ed298e3dc49d789d166c613ad7fb8 P d870164263bb44b6b780a557daea3656: No bootstrap required, opened a new log
I20260130 02:58:10.975237 19625 ts_tablet_manager.cc:1403] T b89ed298e3dc49d789d166c613ad7fb8 P d870164263bb44b6b780a557daea3656: Time spent bootstrapping tablet: real 0.062s user 0.009s sys 0.000s
I20260130 02:58:10.977381 19625 raft_consensus.cc:359] T b89ed298e3dc49d789d166c613ad7fb8 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:10.977989 19625 raft_consensus.cc:385] T b89ed298e3dc49d789d166c613ad7fb8 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:10.978222 19625 raft_consensus.cc:740] T b89ed298e3dc49d789d166c613ad7fb8 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d870164263bb44b6b780a557daea3656, State: Initialized, Role: FOLLOWER
I20260130 02:58:10.978833 20383 raft_consensus.cc:493] T dc09a0f0cecf49738bab04596944c0f3 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:10.979285 20383 raft_consensus.cc:515] T dc09a0f0cecf49738bab04596944c0f3 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:10.981065 20383 leader_election.cc:290] T dc09a0f0cecf49738bab04596944c0f3 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), d870164263bb44b6b780a557daea3656 (127.18.100.67:39571)
I20260130 02:58:10.987202 19225 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc09a0f0cecf49738bab04596944c0f3" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152" is_pre_election: true
W20260130 02:58:10.988281 19314 leader_election.cc:343] T dc09a0f0cecf49738bab04596944c0f3 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:10.990556 20383 raft_consensus.cc:2749] T a8a5038e9def44eb8b3b1e4aeae25652 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260130 02:58:10.992326 19526 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc09a0f0cecf49738bab04596944c0f3" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
I20260130 02:58:10.996294 19840 tablet_bootstrap.cc:654] T 16691c9315e94253bb9b3fb5058e5636 P 78c2c46c7520492db35d51c244fb4f74: Neither blocks nor log segments found. Creating new log.
W20260130 02:58:10.997578 19313 leader_election.cc:343] T dc09a0f0cecf49738bab04596944c0f3 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:10.997964 19313 leader_election.cc:304] T dc09a0f0cecf49738bab04596944c0f3 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78c2c46c7520492db35d51c244fb4f74; no voters: 371a36d219164974abdf848841528152, d870164263bb44b6b780a557daea3656
I20260130 02:58:11.010798 20383 raft_consensus.cc:2749] T dc09a0f0cecf49738bab04596944c0f3 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260130 02:58:11.016952 19159 consensus_peers.cc:597] T 3ae5472cba1e4613abe4cda42ccb2496 P 371a36d219164974abdf848841528152 -> Peer d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): Couldn't send request to peer d870164263bb44b6b780a557daea3656. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260130 02:58:11.021704 20435 raft_consensus.cc:493] T 04f4ac27800c4ac3a79ae12d35479757 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:11.022188 20435 raft_consensus.cc:515] T 04f4ac27800c4ac3a79ae12d35479757 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:11.024060 20435 leader_election.cc:290] T 04f4ac27800c4ac3a79ae12d35479757 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767)
I20260130 02:58:11.025920 19225 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "04f4ac27800c4ac3a79ae12d35479757" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152" is_pre_election: true
I20260130 02:58:11.032176 19378 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "04f4ac27800c4ac3a79ae12d35479757" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74" is_pre_election: true
I20260130 02:58:11.033418 19807 tablet_bootstrap.cc:654] T 753344737b5941d9a93b6779e72f63d4 P 371a36d219164974abdf848841528152: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:11.033609 19462 leader_election.cc:304] T 04f4ac27800c4ac3a79ae12d35479757 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d870164263bb44b6b780a557daea3656; no voters: 371a36d219164974abdf848841528152, 78c2c46c7520492db35d51c244fb4f74
W20260130 02:58:11.034440 19159 consensus_peers.cc:597] T b89ed298e3dc49d789d166c613ad7fb8 P 371a36d219164974abdf848841528152 -> Peer d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): Couldn't send request to peer d870164263bb44b6b780a557daea3656. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20260130 02:58:11.034546 20435 raft_consensus.cc:2749] T 04f4ac27800c4ac3a79ae12d35479757 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260130 02:58:11.038939 19625 consensus_queue.cc:260] T b89ed298e3dc49d789d166c613ad7fb8 P d870164263bb44b6b780a557daea3656 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:11.042521 19625 ts_tablet_manager.cc:1434] T b89ed298e3dc49d789d166c613ad7fb8 P d870164263bb44b6b780a557daea3656: Time spent starting tablet: real 0.067s user 0.006s sys 0.000s
I20260130 02:58:11.051720 18868 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:50284 (request call id 218) took 1884 ms. Trace:
W20260130 02:58:11.055313 18871 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 49efcfced0b74c48962393149ac2a500 has no consensus state
I20260130 02:58:11.055845 18871 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:50284 (request call id 220) took 1543 ms. Trace:
I20260130 02:58:11.057636 20422 ts_tablet_manager.cc:1929] T f54addb1a4304ce69c711ae0cc3b452c P 371a36d219164974abdf848841528152: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260130 02:58:11.058013 20422 log.cc:1199] T f54addb1a4304ce69c711ae0cc3b452c P 371a36d219164974abdf848841528152: Deleting WAL directory at /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-0/wal/wals/f54addb1a4304ce69c711ae0cc3b452c
I20260130 02:58:11.059661 19625 tablet_bootstrap.cc:492] T 3ae5472cba1e4613abe4cda42ccb2496 P d870164263bb44b6b780a557daea3656: Bootstrap starting.
I20260130 02:58:11.052042 18868 rpcz_store.cc:276] 0130 02:58:09.167300 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:58:10.672471 (+1505171us) service_pool.cc:225] Handling call
0130 02:58:10.763771 (+ 91300us) catalog_manager.cc:2101] Acquired catalog manager lock
0130 02:58:10.776996 (+ 13225us) catalog_manager.cc:2147] Created new table and tablet info
0130 02:58:11.051064 (+274068us) catalog_manager.cc:2207] Wrote table and tablets to system table
0130 02:58:11.051371 (+ 307us) catalog_manager.cc:2241] Inserted table and tablets into CatalogManager maps
0130 02:58:11.051635 (+ 264us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0130 02:58:10.850726 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0130 02:58:10.851220 (+ 494us) write_op.cc:432] Acquiring schema lock in shared mode
0130 02:58:10.851247 (+ 27us) write_op.cc:435] Acquired schema lock
0130 02:58:10.851262 (+ 15us) tablet.cc:660] Decoding operations
0130 02:58:10.851573 (+ 311us) write_op.cc:620] Acquiring the partition lock for write op
0130 02:58:10.851615 (+ 42us) write_op.cc:641] Partition lock acquired for write op
0130 02:58:10.851639 (+ 24us) tablet.cc:683] Acquiring locks for 2 operations
0130 02:58:10.851849 (+ 210us) tablet.cc:699] Row locks acquired
0130 02:58:10.851865 (+ 16us) write_op.cc:260] PREPARE: finished
0130 02:58:10.852011 (+ 146us) write_op.cc:270] Start()
0130 02:58:10.852116 (+ 105us) write_op.cc:276] Timestamp: P: 1769741890851914 usec, L: 0
0130 02:58:10.852134 (+ 18us) op_driver.cc:348] REPLICATION: starting
0130 02:58:10.852748 (+ 614us) log.cc:844] Serialized 433 byte log entry
0130 02:58:11.024728 (+171980us) op_driver.cc:464] REPLICATION: finished
0130 02:58:11.030010 (+ 5282us) write_op.cc:301] APPLY: starting
0130 02:58:11.030070 (+ 60us) tablet.cc:1365] starting BulkCheckPresence
0130 02:58:11.030166 (+ 96us) tablet.cc:1368] finished BulkCheckPresence
0130 02:58:11.030265 (+ 99us) tablet.cc:1370] starting ApplyRowOperation cycle
0130 02:58:11.050052 (+ 19787us) tablet.cc:1381] finished ApplyRowOperation cycle
0130 02:58:11.050197 (+ 145us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0130 02:58:11.050240 (+ 43us) write_op.cc:312] APPLY: finished
0130 02:58:11.050431 (+ 191us) log.cc:844] Serialized 29 byte log entry
0130 02:58:11.050611 (+ 180us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:58:11.050676 (+ 65us) write_op.cc:454] Released schema lock
0130 02:58:11.050710 (+ 34us) write_op.cc:341] FINISH: Updating metrics
0130 02:58:11.050938 (+ 228us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:58:11.050961 (+ 23us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":5093,"apply.run_cpu_time_us":1688,"apply.run_wall_time_us":20923,"num_ops":2,"prepare.queue_time_us":28445,"prepare.run_cpu_time_us":2274,"prepare.run_wall_time_us":2275,"replication_time_us":172488,"thread_start_us":18266,"threads_started":1}]]}
I20260130 02:58:11.062171 18868 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50284:
name: "default.Table_aa4e5fd22945475da1dad48e1508595e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260130 02:58:11.063944 18868 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_aa4e5fd22945475da1dad48e1508595e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260130 02:58:11.056100 18871 rpcz_store.cc:276] 0130 02:58:09.512142 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:58:10.826774 (+1314632us) service_pool.cc:225] Handling call
0130 02:58:10.827805 (+ 1031us) catalog_manager.cc:2393] Looking up, locking, and authorizing table
0130 02:58:10.828451 (+ 646us) catalog_manager.cc:3708] Apply alter schema
0130 02:58:10.829007 (+ 556us) catalog_manager.cc:3740] Acquired catalog manager lock
0130 02:58:10.829230 (+ 223us) catalog_manager.cc:3898] Updating metadata on disk
0130 02:58:11.054789 (+225559us) catalog_manager.cc:3929] Committing alterations to in-memory state
0130 02:58:11.055768 (+ 979us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0130 02:58:10.882515 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0130 02:58:10.883043 (+ 528us) write_op.cc:432] Acquiring schema lock in shared mode
0130 02:58:10.883066 (+ 23us) write_op.cc:435] Acquired schema lock
0130 02:58:10.883081 (+ 15us) tablet.cc:660] Decoding operations
0130 02:58:10.883387 (+ 306us) write_op.cc:620] Acquiring the partition lock for write op
0130 02:58:10.883425 (+ 38us) write_op.cc:641] Partition lock acquired for write op
0130 02:58:10.883449 (+ 24us) tablet.cc:683] Acquiring locks for 1 operations
0130 02:58:10.883608 (+ 159us) tablet.cc:699] Row locks acquired
0130 02:58:10.883619 (+ 11us) write_op.cc:260] PREPARE: finished
0130 02:58:10.883740 (+ 121us) write_op.cc:270] Start()
0130 02:58:10.883835 (+ 95us) write_op.cc:276] Timestamp: P: 1769741890883665 usec, L: 0
0130 02:58:10.883852 (+ 17us) op_driver.cc:348] REPLICATION: starting
0130 02:58:10.884391 (+ 539us) log.cc:844] Serialized 405 byte log entry
0130 02:58:11.043959 (+159568us) op_driver.cc:464] REPLICATION: finished
0130 02:58:11.051200 (+ 7241us) write_op.cc:301] APPLY: starting
0130 02:58:11.051254 (+ 54us) tablet.cc:1365] starting BulkCheckPresence
0130 02:58:11.051351 (+ 97us) tablet.cc:1368] finished BulkCheckPresence
0130 02:58:11.051441 (+ 90us) tablet.cc:1370] starting ApplyRowOperation cycle
0130 02:58:11.051615 (+ 174us) tablet.cc:1381] finished ApplyRowOperation cycle
0130 02:58:11.051742 (+ 127us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0130 02:58:11.051781 (+ 39us) write_op.cc:312] APPLY: finished
0130 02:58:11.051933 (+ 152us) log.cc:844] Serialized 23 byte log entry
0130 02:58:11.052071 (+ 138us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:58:11.052129 (+ 58us) write_op.cc:454] Released schema lock
0130 02:58:11.052168 (+ 39us) write_op.cc:341] FINISH: Updating metrics
0130 02:58:11.052369 (+ 201us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:58:11.052393 (+ 24us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":7103,"apply.run_cpu_time_us":1133,"apply.run_wall_time_us":1133,"num_ops":1,"prepare.queue_time_us":26784,"prepare.run_cpu_time_us":2083,"prepare.run_wall_time_us":3075,"replication_time_us":160012}]]}
I20260130 02:58:11.067854 19807 tablet_bootstrap.cc:492] T 753344737b5941d9a93b6779e72f63d4 P 371a36d219164974abdf848841528152: No bootstrap required, opened a new log
I20260130 02:58:11.067922 19625 tablet_bootstrap.cc:654] T 3ae5472cba1e4613abe4cda42ccb2496 P d870164263bb44b6b780a557daea3656: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:11.068333 18871 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50284:
name: "default.Table_7e6ab9964f4c4e0f906d875a937a781d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260130 02:58:11.070815 18871 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7e6ab9964f4c4e0f906d875a937a781d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260130 02:58:11.071275 19313 consensus_peers.cc:597] T 43cec19c51ec4f05b63800c07f9534ef P 78c2c46c7520492db35d51c244fb4f74 -> Peer d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): Couldn't send request to peer d870164263bb44b6b780a557daea3656. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260130 02:58:11.077131 20422 ts_tablet_manager.cc:1950] T f54addb1a4304ce69c711ae0cc3b452c P 371a36d219164974abdf848841528152: Deleting consensus metadata
I20260130 02:58:11.080581 20422 tablet_replica.cc:333] T b2ac91da01954c2390b6b7dcc06c394f P 371a36d219164974abdf848841528152: stopping tablet replica
I20260130 02:58:11.080574 18861 catalog_manager.cc:4985] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021): tablet f54addb1a4304ce69c711ae0cc3b452c (table default.Table_2d345d0f0957486c86c2cf2e34b00941 [id=4c3e0d3ecd5642a3b7fc7d20e2521418]) successfully deleted
I20260130 02:58:11.081724 19840 tablet_bootstrap.cc:492] T 16691c9315e94253bb9b3fb5058e5636 P 78c2c46c7520492db35d51c244fb4f74: No bootstrap required, opened a new log
I20260130 02:58:11.082295 19840 ts_tablet_manager.cc:1403] T 16691c9315e94253bb9b3fb5058e5636 P 78c2c46c7520492db35d51c244fb4f74: Time spent bootstrapping tablet: real 0.238s user 0.010s sys 0.000s
I20260130 02:58:11.083930 20422 raft_consensus.cc:2243] T b2ac91da01954c2390b6b7dcc06c394f P 371a36d219164974abdf848841528152 [term 1 LEADER]: Raft consensus shutting down.
I20260130 02:58:11.084492 19840 raft_consensus.cc:359] T 16691c9315e94253bb9b3fb5058e5636 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:11.085088 19840 raft_consensus.cc:385] T 16691c9315e94253bb9b3fb5058e5636 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:11.077826 19807 ts_tablet_manager.cc:1403] T 753344737b5941d9a93b6779e72f63d4 P 371a36d219164974abdf848841528152: Time spent bootstrapping tablet: real 0.112s user 0.007s sys 0.007s
I20260130 02:58:11.085331 19840 raft_consensus.cc:740] T 16691c9315e94253bb9b3fb5058e5636 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78c2c46c7520492db35d51c244fb4f74, State: Initialized, Role: FOLLOWER
I20260130 02:58:11.085963 19840 consensus_queue.cc:260] T 16691c9315e94253bb9b3fb5058e5636 P 78c2c46c7520492db35d51c244fb4f74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:11.087512 19807 raft_consensus.cc:359] T 753344737b5941d9a93b6779e72f63d4 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:11.088160 19807 raft_consensus.cc:385] T 753344737b5941d9a93b6779e72f63d4 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:11.088255 19840 ts_tablet_manager.cc:1434] T 16691c9315e94253bb9b3fb5058e5636 P 78c2c46c7520492db35d51c244fb4f74: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260130 02:58:11.088486 19807 raft_consensus.cc:740] T 753344737b5941d9a93b6779e72f63d4 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 371a36d219164974abdf848841528152, State: Initialized, Role: FOLLOWER
I20260130 02:58:11.089133 19840 tablet_bootstrap.cc:492] T 753344737b5941d9a93b6779e72f63d4 P 78c2c46c7520492db35d51c244fb4f74: Bootstrap starting.
I20260130 02:58:11.090806 20422 raft_consensus.cc:2272] T b2ac91da01954c2390b6b7dcc06c394f P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260130 02:58:11.092038 18861 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet 49efcfced0b74c48962393149ac2a500 (table default.Table_2e81f0a0cbc449a397141b717db17025 [id=a9c919f57c0c437cb35a6921103e45a6], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 49efcfced0b74c48962393149ac2a500 has no consensus state
I20260130 02:58:11.089540 19807 consensus_queue.cc:260] T 753344737b5941d9a93b6779e72f63d4 P 371a36d219164974abdf848841528152 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:11.101236 19625 tablet_bootstrap.cc:492] T 3ae5472cba1e4613abe4cda42ccb2496 P d870164263bb44b6b780a557daea3656: No bootstrap required, opened a new log
I20260130 02:58:11.101629 19625 ts_tablet_manager.cc:1403] T 3ae5472cba1e4613abe4cda42ccb2496 P d870164263bb44b6b780a557daea3656: Time spent bootstrapping tablet: real 0.042s user 0.005s sys 0.003s
I20260130 02:58:11.104041 19625 raft_consensus.cc:359] T 3ae5472cba1e4613abe4cda42ccb2496 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:11.104662 19625 raft_consensus.cc:385] T 3ae5472cba1e4613abe4cda42ccb2496 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:11.104913 19625 raft_consensus.cc:740] T 3ae5472cba1e4613abe4cda42ccb2496 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d870164263bb44b6b780a557daea3656, State: Initialized, Role: FOLLOWER
I20260130 02:58:11.106496 20422 ts_tablet_manager.cc:1916] T b2ac91da01954c2390b6b7dcc06c394f P 371a36d219164974abdf848841528152: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260130 02:58:11.110231 19947 ts_tablet_manager.cc:1929] T 6573198830ad45628e71952a7d9c00a0 P d870164263bb44b6b780a557daea3656: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260130 02:58:11.101274 19807 ts_tablet_manager.cc:1434] T 753344737b5941d9a93b6779e72f63d4 P 371a36d219164974abdf848841528152: Time spent starting tablet: real 0.016s user 0.000s sys 0.004s
I20260130 02:58:11.111476 19625 consensus_queue.cc:260] T 3ae5472cba1e4613abe4cda42ccb2496 P d870164263bb44b6b780a557daea3656 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:11.112462 19807 tablet_bootstrap.cc:492] T a8a5038e9def44eb8b3b1e4aeae25652 P 371a36d219164974abdf848841528152: Bootstrap starting.
I20260130 02:58:11.131891 19625 ts_tablet_manager.cc:1434] T 3ae5472cba1e4613abe4cda42ccb2496 P d870164263bb44b6b780a557daea3656: Time spent starting tablet: real 0.030s user 0.006s sys 0.001s
I20260130 02:58:11.132707 19807 tablet_bootstrap.cc:654] T a8a5038e9def44eb8b3b1e4aeae25652 P 371a36d219164974abdf848841528152: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:11.134383 19625 tablet_bootstrap.cc:492] T 0e8018c094024c4788c484e8f5466e00 P d870164263bb44b6b780a557daea3656: Bootstrap starting.
I20260130 02:58:11.144111 19625 tablet_bootstrap.cc:654] T 0e8018c094024c4788c484e8f5466e00 P d870164263bb44b6b780a557daea3656: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:11.143934 19501 tablet_service.cc:1505] Processing CreateTablet for tablet 2c79a3aecab844698e70834999e1caac (DEFAULT_TABLE table=default.Table_12c2fe20cc3e45d29b7be37f900f68a9 [id=67ac4f4eda324e799b5867b2b011a16b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:11.144971 19501 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2c79a3aecab844698e70834999e1caac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:11.146186 20243 raft_consensus.cc:493] T 3b02480d9c6f4328af83f7940716d698 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:11.146665 20243 raft_consensus.cc:515] T 3b02480d9c6f4328af83f7940716d698 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:11.148550 20243 leader_election.cc:290] T 3b02480d9c6f4328af83f7940716d698 P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d870164263bb44b6b780a557daea3656 (127.18.100.67:39571), 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767)
I20260130 02:58:11.151501 19526 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3b02480d9c6f4328af83f7940716d698" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
I20260130 02:58:11.152570 19376 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3b02480d9c6f4328af83f7940716d698" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74" is_pre_election: true
W20260130 02:58:11.154469 19159 leader_election.cc:343] T 3b02480d9c6f4328af83f7940716d698 P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): Illegal state: must be running to vote when last-logged opid is not known
W20260130 02:58:11.155110 19159 leader_election.cc:343] T 3b02480d9c6f4328af83f7940716d698 P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:11.155428 19159 leader_election.cc:304] T 3b02480d9c6f4328af83f7940716d698 P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 371a36d219164974abdf848841528152; no voters: 78c2c46c7520492db35d51c244fb4f74, d870164263bb44b6b780a557daea3656
I20260130 02:58:11.156283 20331 raft_consensus.cc:493] T d27ca7dd87f44102b7680e1f4a88954a P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:11.156683 20331 raft_consensus.cc:515] T d27ca7dd87f44102b7680e1f4a88954a P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:11.158706 20244 raft_consensus.cc:493] T 753344737b5941d9a93b6779e72f63d4 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:11.159097 20244 raft_consensus.cc:515] T 753344737b5941d9a93b6779e72f63d4 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:11.160804 20244 leader_election.cc:290] T 753344737b5941d9a93b6779e72f63d4 P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767), d870164263bb44b6b780a557daea3656 (127.18.100.67:39571)
I20260130 02:58:11.161234 20244 raft_consensus.cc:2749] T 3b02480d9c6f4328af83f7940716d698 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260130 02:58:11.158398 20331 leader_election.cc:290] T d27ca7dd87f44102b7680e1f4a88954a P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767), d870164263bb44b6b780a557daea3656 (127.18.100.67:39571)
I20260130 02:58:11.169541 19521 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "753344737b5941d9a93b6779e72f63d4" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
I20260130 02:58:11.171708 19378 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "753344737b5941d9a93b6779e72f63d4" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74" is_pre_election: true
I20260130 02:58:11.173751 19526 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d27ca7dd87f44102b7680e1f4a88954a" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
I20260130 02:58:11.174692 20414 ts_tablet_manager.cc:1929] T 6573198830ad45628e71952a7d9c00a0 P 78c2c46c7520492db35d51c244fb4f74: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260130 02:58:11.174921 19376 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d27ca7dd87f44102b7680e1f4a88954a" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74" is_pre_election: true
I20260130 02:58:11.175206 20414 log.cc:1199] T 6573198830ad45628e71952a7d9c00a0 P 78c2c46c7520492db35d51c244fb4f74: Deleting WAL directory at /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-1/wal/wals/6573198830ad45628e71952a7d9c00a0
I20260130 02:58:11.175518 19376 raft_consensus.cc:2468] T d27ca7dd87f44102b7680e1f4a88954a P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 371a36d219164974abdf848841528152 in term 0.
I20260130 02:58:11.176124 20414 ts_tablet_manager.cc:1950] T 6573198830ad45628e71952a7d9c00a0 P 78c2c46c7520492db35d51c244fb4f74: Deleting consensus metadata
I20260130 02:58:11.176738 19521 raft_consensus.cc:3060] T ee7c8ff92ca9465fa18be6eae63d78ca P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:11.177634 18874 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:50284 (request call id 219) took 1960 ms. Trace:
I20260130 02:58:11.177937 18874 rpcz_store.cc:276] 0130 02:58:09.217139 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:58:10.707465 (+1490326us) service_pool.cc:225] Handling call
0130 02:58:11.177557 (+470092us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260130 02:58:11.178977 20414 tablet_replica.cc:333] T f54addb1a4304ce69c711ae0cc3b452c P 78c2c46c7520492db35d51c244fb4f74: stopping tablet replica
I20260130 02:58:11.178997 18874 master_service.cc:954] ReplaceTablet: received request to replace tablet 7b80b22bd44c4233837c951aa97e6996 from {username='slave'} at 127.0.0.1:50284
I20260130 02:58:11.179865 20414 raft_consensus.cc:2243] T f54addb1a4304ce69c711ae0cc3b452c P 78c2c46c7520492db35d51c244fb4f74 [term 1 LEADER]: Raft consensus shutting down.
I20260130 02:58:11.180763 20414 raft_consensus.cc:2272] T f54addb1a4304ce69c711ae0cc3b452c P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260130 02:58:11.142460 19840 tablet_bootstrap.cc:654] T 753344737b5941d9a93b6779e72f63d4 P 78c2c46c7520492db35d51c244fb4f74: Neither blocks nor log segments found. Creating new log.
W20260130 02:58:11.181353 19159 leader_election.cc:343] T 753344737b5941d9a93b6779e72f63d4 P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:11.184128 19159 leader_election.cc:304] T d27ca7dd87f44102b7680e1f4a88954a P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 371a36d219164974abdf848841528152, 78c2c46c7520492db35d51c244fb4f74; no voters:
I20260130 02:58:11.181190 18871 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:50284 (request call id 222) took 1261 ms. Trace:
I20260130 02:58:11.184675 20244 consensus_queue.cc:1048] T 3ae5472cba1e4613abe4cda42ccb2496 P 371a36d219164974abdf848841528152 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260130 02:58:11.187109 20243 raft_consensus.cc:2804] T d27ca7dd87f44102b7680e1f4a88954a P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260130 02:58:11.183511 18868 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:50284 (request call id 221) took 1351 ms. Trace:
I20260130 02:58:11.187448 20243 raft_consensus.cc:493] T d27ca7dd87f44102b7680e1f4a88954a P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260130 02:58:11.187744 20243 raft_consensus.cc:3060] T d27ca7dd87f44102b7680e1f4a88954a P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Advancing to term 1
W20260130 02:58:11.191051 19159 leader_election.cc:343] T 753344737b5941d9a93b6779e72f63d4 P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:11.185236 18871 rpcz_store.cc:276] 0130 02:58:09.919698 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:58:11.067872 (+1148174us) service_pool.cc:225] Handling call
0130 02:58:11.071250 (+ 3378us) catalog_manager.cc:2101] Acquired catalog manager lock
0130 02:58:11.073888 (+ 2638us) catalog_manager.cc:2147] Created new table and tablet info
0130 02:58:11.180612 (+106724us) catalog_manager.cc:2207] Wrote table and tablets to system table
0130 02:58:11.180872 (+ 260us) catalog_manager.cc:2241] Inserted table and tablets into CatalogManager maps
0130 02:58:11.181124 (+ 252us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0130 02:58:11.096115 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0130 02:58:11.096562 (+ 447us) write_op.cc:432] Acquiring schema lock in shared mode
0130 02:58:11.096588 (+ 26us) write_op.cc:435] Acquired schema lock
0130 02:58:11.096603 (+ 15us) tablet.cc:660] Decoding operations
0130 02:58:11.096910 (+ 307us) write_op.cc:620] Acquiring the partition lock for write op
0130 02:58:11.096947 (+ 37us) write_op.cc:641] Partition lock acquired for write op
0130 02:58:11.096975 (+ 28us) tablet.cc:683] Acquiring locks for 2 operations
0130 02:58:11.097192 (+ 217us) tablet.cc:699] Row locks acquired
0130 02:58:11.097209 (+ 17us) write_op.cc:260] PREPARE: finished
0130 02:58:11.097338 (+ 129us) write_op.cc:270] Start()
0130 02:58:11.097426 (+ 88us) write_op.cc:276] Timestamp: P: 1769741891097293 usec, L: 0
0130 02:58:11.097448 (+ 22us) op_driver.cc:348] REPLICATION: starting
0130 02:58:11.097947 (+ 499us) log.cc:844] Serialized 433 byte log entry
0130 02:58:11.169282 (+ 71335us) op_driver.cc:464] REPLICATION: finished
0130 02:58:11.170465 (+ 1183us) write_op.cc:301] APPLY: starting
0130 02:58:11.170517 (+ 52us) tablet.cc:1365] starting BulkCheckPresence
0130 02:58:11.170653 (+ 136us) tablet.cc:1368] finished BulkCheckPresence
0130 02:58:11.170757 (+ 104us) tablet.cc:1370] starting ApplyRowOperation cycle
0130 02:58:11.171162 (+ 405us) tablet.cc:1381] finished ApplyRowOperation cycle
0130 02:58:11.171309 (+ 147us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0130 02:58:11.171351 (+ 42us) write_op.cc:312] APPLY: finished
0130 02:58:11.171545 (+ 194us) log.cc:844] Serialized 29 byte log entry
0130 02:58:11.171695 (+ 150us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:58:11.171760 (+ 65us) write_op.cc:454] Released schema lock
0130 02:58:11.171791 (+ 31us) write_op.cc:341] FINISH: Updating metrics
0130 02:58:11.172024 (+ 233us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:58:11.172049 (+ 25us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":1014,"apply.run_cpu_time_us":1512,"apply.run_wall_time_us":1521,"mutex_wait_us":20838,"num_ops":2,"prepare.queue_time_us":21019,"prepare.run_cpu_time_us":2045,"prepare.run_wall_time_us":2948,"replication_time_us":71722}]]}
I20260130 02:58:11.191480 19159 leader_election.cc:304] T 753344737b5941d9a93b6779e72f63d4 P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 371a36d219164974abdf848841528152; no voters: 78c2c46c7520492db35d51c244fb4f74, d870164263bb44b6b780a557daea3656
W20260130 02:58:11.195547 19159 leader_election.cc:343] T d27ca7dd87f44102b7680e1f4a88954a P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:11.195986 20244 raft_consensus.cc:2749] T 753344737b5941d9a93b6779e72f63d4 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260130 02:58:11.197649 19625 tablet_bootstrap.cc:492] T 0e8018c094024c4788c484e8f5466e00 P d870164263bb44b6b780a557daea3656: No bootstrap required, opened a new log
I20260130 02:58:11.198048 19625 ts_tablet_manager.cc:1403] T 0e8018c094024c4788c484e8f5466e00 P d870164263bb44b6b780a557daea3656: Time spent bootstrapping tablet: real 0.064s user 0.014s sys 0.002s
I20260130 02:58:11.187601 18868 rpcz_store.cc:276] 0130 02:58:09.831694 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:58:11.061283 (+1229589us) service_pool.cc:225] Handling call
0130 02:58:11.064660 (+ 3377us) catalog_manager.cc:2101] Acquired catalog manager lock
0130 02:58:11.066218 (+ 1558us) catalog_manager.cc:2147] Created new table and tablet info
0130 02:58:11.182910 (+116692us) catalog_manager.cc:2207] Wrote table and tablets to system table
0130 02:58:11.183177 (+ 267us) catalog_manager.cc:2241] Inserted table and tablets into CatalogManager maps
0130 02:58:11.183446 (+ 269us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0130 02:58:11.099184 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0130 02:58:11.099616 (+ 432us) write_op.cc:432] Acquiring schema lock in shared mode
0130 02:58:11.099640 (+ 24us) write_op.cc:435] Acquired schema lock
0130 02:58:11.099656 (+ 16us) tablet.cc:660] Decoding operations
0130 02:58:11.099946 (+ 290us) write_op.cc:620] Acquiring the partition lock for write op
0130 02:58:11.099985 (+ 39us) write_op.cc:641] Partition lock acquired for write op
0130 02:58:11.100011 (+ 26us) tablet.cc:683] Acquiring locks for 2 operations
0130 02:58:11.100222 (+ 211us) tablet.cc:699] Row locks acquired
0130 02:58:11.100246 (+ 24us) write_op.cc:260] PREPARE: finished
0130 02:58:11.100355 (+ 109us) write_op.cc:270] Start()
0130 02:58:11.100434 (+ 79us) write_op.cc:276] Timestamp: P: 1769741891100314 usec, L: 0
0130 02:58:11.100455 (+ 21us) op_driver.cc:348] REPLICATION: starting
0130 02:58:11.100923 (+ 468us) log.cc:844] Serialized 433 byte log entry
0130 02:58:11.169574 (+ 68651us) op_driver.cc:464] REPLICATION: finished
0130 02:58:11.172282 (+ 2708us) write_op.cc:301] APPLY: starting
0130 02:58:11.172331 (+ 49us) tablet.cc:1365] starting BulkCheckPresence
0130 02:58:11.172422 (+ 91us) tablet.cc:1368] finished BulkCheckPresence
0130 02:58:11.172510 (+ 88us) tablet.cc:1370] starting ApplyRowOperation cycle
0130 02:58:11.172894 (+ 384us) tablet.cc:1381] finished ApplyRowOperation cycle
0130 02:58:11.173024 (+ 130us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0130 02:58:11.173062 (+ 38us) write_op.cc:312] APPLY: finished
0130 02:58:11.173226 (+ 164us) log.cc:844] Serialized 29 byte log entry
0130 02:58:11.173360 (+ 134us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:58:11.173419 (+ 59us) write_op.cc:454] Released schema lock
0130 02:58:11.173446 (+ 27us) write_op.cc:341] FINISH: Updating metrics
0130 02:58:11.173648 (+ 202us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:58:11.173669 (+ 21us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":2577,"apply.run_cpu_time_us":1338,"apply.run_wall_time_us":1338,"mutex_wait_us":17475,"num_ops":2,"prepare.queue_time_us":20609,"prepare.run_cpu_time_us":1947,"prepare.run_wall_time_us":2957,"replication_time_us":69030}]]}
I20260130 02:58:11.200275 19625 raft_consensus.cc:359] T 0e8018c094024c4788c484e8f5466e00 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:11.203835 19625 raft_consensus.cc:385] T 0e8018c094024c4788c484e8f5466e00 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20260130 02:58:11.205485 19313 consensus_peers.cc:597] T 8e57ae31ab3941f789ac69627384e3b0 P 78c2c46c7520492db35d51c244fb4f74 -> Peer d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): Couldn't send request to peer d870164263bb44b6b780a557daea3656. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260130 02:58:11.204499 19625 raft_consensus.cc:740] T 0e8018c094024c4788c484e8f5466e00 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d870164263bb44b6b780a557daea3656, State: Initialized, Role: FOLLOWER
I20260130 02:58:11.206247 19625 consensus_queue.cc:260] T 0e8018c094024c4788c484e8f5466e00 P d870164263bb44b6b780a557daea3656 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:11.209497 19625 ts_tablet_manager.cc:1434] T 0e8018c094024c4788c484e8f5466e00 P d870164263bb44b6b780a557daea3656: Time spent starting tablet: real 0.011s user 0.005s sys 0.000s
I20260130 02:58:11.215348 19625 tablet_bootstrap.cc:492] T d27ca7dd87f44102b7680e1f4a88954a P d870164263bb44b6b780a557daea3656: Bootstrap starting.
I20260130 02:58:11.212998 20243 raft_consensus.cc:515] T d27ca7dd87f44102b7680e1f4a88954a P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:11.222118 19375 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d27ca7dd87f44102b7680e1f4a88954a" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74"
I20260130 02:58:11.222765 19375 raft_consensus.cc:3060] T d27ca7dd87f44102b7680e1f4a88954a P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:11.223488 19205 tablet_service.cc:1505] Processing CreateTablet for tablet 2c79a3aecab844698e70834999e1caac (DEFAULT_TABLE table=default.Table_12c2fe20cc3e45d29b7be37f900f68a9 [id=67ac4f4eda324e799b5867b2b011a16b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:11.224440 19205 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2c79a3aecab844698e70834999e1caac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:11.226519 19840 tablet_bootstrap.cc:492] T 753344737b5941d9a93b6779e72f63d4 P 78c2c46c7520492db35d51c244fb4f74: No bootstrap required, opened a new log
I20260130 02:58:11.226951 19840 ts_tablet_manager.cc:1403] T 753344737b5941d9a93b6779e72f63d4 P 78c2c46c7520492db35d51c244fb4f74: Time spent bootstrapping tablet: real 0.138s user 0.011s sys 0.000s
I20260130 02:58:11.230515 19625 tablet_bootstrap.cc:654] T d27ca7dd87f44102b7680e1f4a88954a P d870164263bb44b6b780a557daea3656: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:11.232713 19375 raft_consensus.cc:2468] T d27ca7dd87f44102b7680e1f4a88954a P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 371a36d219164974abdf848841528152 in term 1.
I20260130 02:58:11.234901 18860 catalog_manager.cc:4985] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767): tablet 6573198830ad45628e71952a7d9c00a0 (table default.Table_1c2b251034c0418cba750ae4d1308b9f [id=9a5fe99c9417426dba0a12441e7191c6]) successfully deleted
I20260130 02:58:11.235917 19159 leader_election.cc:304] T d27ca7dd87f44102b7680e1f4a88954a P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 371a36d219164974abdf848841528152, 78c2c46c7520492db35d51c244fb4f74; no voters:
I20260130 02:58:11.236709 20244 raft_consensus.cc:2804] T d27ca7dd87f44102b7680e1f4a88954a P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Leader election won for term 1
I20260130 02:58:11.237180 20244 raft_consensus.cc:697] T d27ca7dd87f44102b7680e1f4a88954a P 371a36d219164974abdf848841528152 [term 1 LEADER]: Becoming Leader. State: Replica: 371a36d219164974abdf848841528152, State: Running, Role: LEADER
I20260130 02:58:11.237920 20244 consensus_queue.cc:237] T d27ca7dd87f44102b7680e1f4a88954a P 371a36d219164974abdf848841528152 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:11.244539 19201 tablet_service.cc:1505] Processing CreateTablet for tablet f7607df5efd543438fb2dff3667cc8c8 (DEFAULT_TABLE table=default.Table_e8fd7516280f4dc4a73eb1f88fe916a2 [id=6efb836791414f3888f714aa8f65f0a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:11.236927 19840 raft_consensus.cc:359] T 753344737b5941d9a93b6779e72f63d4 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:11.245532 19201 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f7607df5efd543438fb2dff3667cc8c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:11.245820 19840 raft_consensus.cc:385] T 753344737b5941d9a93b6779e72f63d4 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:11.247457 19840 raft_consensus.cc:740] T 753344737b5941d9a93b6779e72f63d4 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78c2c46c7520492db35d51c244fb4f74, State: Initialized, Role: FOLLOWER
I20260130 02:58:11.265074 19350 tablet_service.cc:1505] Processing CreateTablet for tablet 2c79a3aecab844698e70834999e1caac (DEFAULT_TABLE table=default.Table_12c2fe20cc3e45d29b7be37f900f68a9 [id=67ac4f4eda324e799b5867b2b011a16b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:11.266273 19350 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2c79a3aecab844698e70834999e1caac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:11.269146 18871 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50284:
name: "default.Table_6773a0ad5d6a429086a5818d731f3f4a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260130 02:58:11.271986 18871 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6773a0ad5d6a429086a5818d731f3f4a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260130 02:58:11.273402 19807 tablet_bootstrap.cc:492] T a8a5038e9def44eb8b3b1e4aeae25652 P 371a36d219164974abdf848841528152: No bootstrap required, opened a new log
I20260130 02:58:11.273838 19807 ts_tablet_manager.cc:1403] T a8a5038e9def44eb8b3b1e4aeae25652 P 371a36d219164974abdf848841528152: Time spent bootstrapping tablet: real 0.162s user 0.008s sys 0.001s
I20260130 02:58:11.274736 18868 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50284:
table { table_name: "default.Table_4c2511f2cf3f40ee9c134faafc7c8383" } new_table_name: "default.Table_8f311c5878744ef692b496ecfdfc3ede" modify_external_catalogs: true
I20260130 02:58:11.275820 20243 leader_election.cc:290] T d27ca7dd87f44102b7680e1f4a88954a P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 election: Requested vote from peers 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767), d870164263bb44b6b780a557daea3656 (127.18.100.67:39571)
I20260130 02:58:11.278632 19526 raft_consensus.cc:3060] T 7d96e93d95f14af79cfb268626559b1e P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:11.265071 19840 consensus_queue.cc:260] T 753344737b5941d9a93b6779e72f63d4 P 78c2c46c7520492db35d51c244fb4f74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:11.285382 19496 tablet_service.cc:1505] Processing CreateTablet for tablet f7607df5efd543438fb2dff3667cc8c8 (DEFAULT_TABLE table=default.Table_e8fd7516280f4dc4a73eb1f88fe916a2 [id=6efb836791414f3888f714aa8f65f0a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:11.286466 19496 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f7607df5efd543438fb2dff3667cc8c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:11.288687 19521 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d27ca7dd87f44102b7680e1f4a88954a" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656"
W20260130 02:58:11.290035 19159 leader_election.cc:343] T d27ca7dd87f44102b7680e1f4a88954a P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:11.291765 19840 ts_tablet_manager.cc:1434] T 753344737b5941d9a93b6779e72f63d4 P 78c2c46c7520492db35d51c244fb4f74: Time spent starting tablet: real 0.057s user 0.005s sys 0.000s
I20260130 02:58:11.292537 19807 raft_consensus.cc:359] T a8a5038e9def44eb8b3b1e4aeae25652 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:11.293147 19807 raft_consensus.cc:385] T a8a5038e9def44eb8b3b1e4aeae25652 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:11.293406 19807 raft_consensus.cc:740] T a8a5038e9def44eb8b3b1e4aeae25652 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 371a36d219164974abdf848841528152, State: Initialized, Role: FOLLOWER
I20260130 02:58:11.294101 19840 tablet_bootstrap.cc:492] T 0e8018c094024c4788c484e8f5466e00 P 78c2c46c7520492db35d51c244fb4f74: Bootstrap starting.
I20260130 02:58:11.294034 19807 consensus_queue.cc:260] T a8a5038e9def44eb8b3b1e4aeae25652 P 371a36d219164974abdf848841528152 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:11.296339 19807 ts_tablet_manager.cc:1434] T a8a5038e9def44eb8b3b1e4aeae25652 P 371a36d219164974abdf848841528152: Time spent starting tablet: real 0.022s user 0.003s sys 0.002s
I20260130 02:58:11.297123 19807 tablet_bootstrap.cc:492] T 16691c9315e94253bb9b3fb5058e5636 P 371a36d219164974abdf848841528152: Bootstrap starting.
I20260130 02:58:11.305490 20435 raft_consensus.cc:493] T b89ed298e3dc49d789d166c613ad7fb8 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:11.305945 20435 raft_consensus.cc:515] T b89ed298e3dc49d789d166c613ad7fb8 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:11.286504 20422 ts_tablet_manager.cc:1929] T b2ac91da01954c2390b6b7dcc06c394f P 371a36d219164974abdf848841528152: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260130 02:58:11.308705 20422 log.cc:1199] T b2ac91da01954c2390b6b7dcc06c394f P 371a36d219164974abdf848841528152: Deleting WAL directory at /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-0/wal/wals/b2ac91da01954c2390b6b7dcc06c394f
I20260130 02:58:11.309438 20422 ts_tablet_manager.cc:1950] T b2ac91da01954c2390b6b7dcc06c394f P 371a36d219164974abdf848841528152: Deleting consensus metadata
I20260130 02:58:11.312001 18861 catalog_manager.cc:4985] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021): tablet b2ac91da01954c2390b6b7dcc06c394f (table default.Table_f2f790e8e46840ecb199c34fae899ada [id=60b8e98e3f864716a6c1aa3816612211]) successfully deleted
W20260130 02:58:11.316641 18861 catalog_manager.cc:4952] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021): delete failed for tablet 1bbc55bbf2254d8bb6b721d7e8f69d85 because the tablet was not found. No further retry: Not found: Tablet not found: 1bbc55bbf2254d8bb6b721d7e8f69d85
I20260130 02:58:11.333961 19358 tablet_service.cc:1505] Processing CreateTablet for tablet f7607df5efd543438fb2dff3667cc8c8 (DEFAULT_TABLE table=default.Table_e8fd7516280f4dc4a73eb1f88fe916a2 [id=6efb836791414f3888f714aa8f65f0a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:11.335023 19358 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f7607df5efd543438fb2dff3667cc8c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:11.345468 19840 tablet_bootstrap.cc:654] T 0e8018c094024c4788c484e8f5466e00 P 78c2c46c7520492db35d51c244fb4f74: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:11.356289 19807 tablet_bootstrap.cc:654] T 16691c9315e94253bb9b3fb5058e5636 P 371a36d219164974abdf848841528152: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:11.378240 19205 tablet_service.cc:1505] Processing CreateTablet for tablet 133e9bfe175f44aab1a53114abe67e4d (DEFAULT_TABLE table=default.Table_d3f82845a18f4a4a830460cb1cbbb2de [id=c3d0c57ad165460eae9c20c32557fd2e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:11.325023 19947 log.cc:1199] T 6573198830ad45628e71952a7d9c00a0 P d870164263bb44b6b780a557daea3656: Deleting WAL directory at /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-2/wal/wals/6573198830ad45628e71952a7d9c00a0
I20260130 02:58:11.379521 19947 ts_tablet_manager.cc:1950] T 6573198830ad45628e71952a7d9c00a0 P d870164263bb44b6b780a557daea3656: Deleting consensus metadata
I20260130 02:58:11.379521 19505 tablet_service.cc:1505] Processing CreateTablet for tablet 133e9bfe175f44aab1a53114abe67e4d (DEFAULT_TABLE table=default.Table_d3f82845a18f4a4a830460cb1cbbb2de [id=c3d0c57ad165460eae9c20c32557fd2e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:11.380146 19840 tablet_bootstrap.cc:492] T 0e8018c094024c4788c484e8f5466e00 P 78c2c46c7520492db35d51c244fb4f74: No bootstrap required, opened a new log
I20260130 02:58:11.380507 19840 ts_tablet_manager.cc:1403] T 0e8018c094024c4788c484e8f5466e00 P 78c2c46c7520492db35d51c244fb4f74: Time spent bootstrapping tablet: real 0.087s user 0.008s sys 0.000s
I20260130 02:58:11.381121 19505 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 133e9bfe175f44aab1a53114abe67e4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:11.382640 19840 raft_consensus.cc:359] T 0e8018c094024c4788c484e8f5466e00 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:11.383251 19840 raft_consensus.cc:385] T 0e8018c094024c4788c484e8f5466e00 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:11.383496 19840 raft_consensus.cc:740] T 0e8018c094024c4788c484e8f5466e00 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78c2c46c7520492db35d51c244fb4f74, State: Initialized, Role: FOLLOWER
I20260130 02:58:11.384162 20414 ts_tablet_manager.cc:1916] T f54addb1a4304ce69c711ae0cc3b452c P 78c2c46c7520492db35d51c244fb4f74: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260130 02:58:11.384052 19840 consensus_queue.cc:260] T 0e8018c094024c4788c484e8f5466e00 P 78c2c46c7520492db35d51c244fb4f74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:11.397172 19840 ts_tablet_manager.cc:1434] T 0e8018c094024c4788c484e8f5466e00 P 78c2c46c7520492db35d51c244fb4f74: Time spent starting tablet: real 0.016s user 0.007s sys 0.000s
I20260130 02:58:11.400781 19840 tablet_bootstrap.cc:492] T 9fe0448bcc4a4346b3c67fc3f626ed3c P 78c2c46c7520492db35d51c244fb4f74: Bootstrap starting.
I20260130 02:58:11.403406 18867 catalog_manager.cc:5746] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571) does not have the latest schema for tablet 341041cb0be2456e88403367c0814704 (table default.Table_1703c69112744c0e951c1ac9667bbac6 [id=8d76f4328edc491291491c78ba205d30]). Expected version 1 got 0
I20260130 02:58:11.404227 18867 catalog_manager.cc:5746] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571) does not have the latest schema for tablet ee7c8ff92ca9465fa18be6eae63d78ca (table default.Table_dd22e9dc63f84d8688c0071f427f0b25 [id=77979d498aa040828de415129cb15a06]). Expected version 1 got 0
I20260130 02:58:11.404937 18867 catalog_manager.cc:5746] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571) does not have the latest schema for tablet 54be22352e404ff9b798a075693f7f21 (table default.Table_3a858a3cb886427cb8268f42c953ad4c [id=17150cb73e374ff9a1e92c43e0c3f2c1]). Expected version 1 got 0
I20260130 02:58:11.406044 19840 tablet_bootstrap.cc:654] T 9fe0448bcc4a4346b3c67fc3f626ed3c P 78c2c46c7520492db35d51c244fb4f74: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:11.428979 19201 tablet_service.cc:1505] Processing CreateTablet for tablet ca06f07bb98f45a7bf63aebc28348e08 (DEFAULT_TABLE table=default.Table_829d1e187e8b4db5b8f4f835e84246e5 [id=a24b4761037d47a383f454a82517131f]), partition=RANGE (key) PARTITION UNBOUNDED
W20260130 02:58:11.431066 18868 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 16691c9315e94253bb9b3fb5058e5636 consensus state has no leader
I20260130 02:58:11.436769 19807 tablet_bootstrap.cc:492] T 16691c9315e94253bb9b3fb5058e5636 P 371a36d219164974abdf848841528152: No bootstrap required, opened a new log
I20260130 02:58:11.437202 19807 ts_tablet_manager.cc:1403] T 16691c9315e94253bb9b3fb5058e5636 P 371a36d219164974abdf848841528152: Time spent bootstrapping tablet: real 0.140s user 0.010s sys 0.011s
I20260130 02:58:11.437875 19350 tablet_service.cc:1505] Processing CreateTablet for tablet 133e9bfe175f44aab1a53114abe67e4d (DEFAULT_TABLE table=default.Table_d3f82845a18f4a4a830460cb1cbbb2de [id=c3d0c57ad165460eae9c20c32557fd2e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:11.439378 19350 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 133e9bfe175f44aab1a53114abe67e4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:11.439353 19807 raft_consensus.cc:359] T 16691c9315e94253bb9b3fb5058e5636 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:11.439944 19807 raft_consensus.cc:385] T 16691c9315e94253bb9b3fb5058e5636 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:11.440173 19807 raft_consensus.cc:740] T 16691c9315e94253bb9b3fb5058e5636 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 371a36d219164974abdf848841528152, State: Initialized, Role: FOLLOWER
I20260130 02:58:11.440884 19807 consensus_queue.cc:260] T 16691c9315e94253bb9b3fb5058e5636 P 371a36d219164974abdf848841528152 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:11.430440 18871 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:50284 (request call id 224) took 1387 ms. Trace:
I20260130 02:58:11.453235 18871 rpcz_store.cc:276] 0130 02:58:10.043264 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:58:11.191700 (+1148436us) service_pool.cc:225] Handling call
0130 02:58:11.269019 (+ 77319us) spinlock_profiling.cc:243] Waited 75.2 ms on lock 0x7b5400030554. stack: 00007fb0bd3f7373 00007fb0bd3f7270 00007fb0bcf6c574 0000000000508744 00000000005086d9 00007fb0c44490c0 00007fb0c44e7bba 00007fb0c61e6eac 00007fb0c62e2656 00007fb0c3f05524 00007fb0c3f054b2 00007fb0c3f053e1 00007fb0c3f0535c 00007fb0c3f04612 00007fb0c21dd34c 00007fb0c21dc786
0130 02:58:11.273708 (+ 4689us) catalog_manager.cc:2101] Acquired catalog manager lock
0130 02:58:11.360370 (+ 86662us) catalog_manager.cc:2147] Created new table and tablet info
0130 02:58:11.414329 (+ 53959us) catalog_manager.cc:2207] Wrote table and tablets to system table
0130 02:58:11.430099 (+ 15770us) catalog_manager.cc:2241] Inserted table and tablets into CatalogManager maps
0130 02:58:11.430377 (+ 278us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0130 02:58:11.384719 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0130 02:58:11.385174 (+ 455us) write_op.cc:432] Acquiring schema lock in shared mode
0130 02:58:11.385201 (+ 27us) write_op.cc:435] Acquired schema lock
0130 02:58:11.385229 (+ 28us) tablet.cc:660] Decoding operations
0130 02:58:11.385565 (+ 336us) write_op.cc:620] Acquiring the partition lock for write op
0130 02:58:11.385610 (+ 45us) write_op.cc:641] Partition lock acquired for write op
0130 02:58:11.385634 (+ 24us) tablet.cc:683] Acquiring locks for 2 operations
0130 02:58:11.385841 (+ 207us) tablet.cc:699] Row locks acquired
0130 02:58:11.385858 (+ 17us) write_op.cc:260] PREPARE: finished
0130 02:58:11.385999 (+ 141us) write_op.cc:270] Start()
0130 02:58:11.386077 (+ 78us) write_op.cc:276] Timestamp: P: 1769741891385923 usec, L: 0
0130 02:58:11.386090 (+ 13us) op_driver.cc:348] REPLICATION: starting
0130 02:58:11.386564 (+ 474us) log.cc:844] Serialized 433 byte log entry
0130 02:58:11.394665 (+ 8101us) op_driver.cc:464] REPLICATION: finished
0130 02:58:11.400703 (+ 6038us) write_op.cc:301] APPLY: starting
0130 02:58:11.400755 (+ 52us) tablet.cc:1365] starting BulkCheckPresence
0130 02:58:11.401080 (+ 325us) tablet.cc:1368] finished BulkCheckPresence
0130 02:58:11.401193 (+ 113us) tablet.cc:1370] starting ApplyRowOperation cycle
0130 02:58:11.401641 (+ 448us) tablet.cc:1381] finished ApplyRowOperation cycle
0130 02:58:11.401776 (+ 135us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0130 02:58:11.401812 (+ 36us) write_op.cc:312] APPLY: finished
0130 02:58:11.401976 (+ 164us) log.cc:844] Serialized 29 byte log entry
0130 02:58:11.402560 (+ 584us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:58:11.402640 (+ 80us) write_op.cc:454] Released schema lock
0130 02:58:11.402679 (+ 39us) write_op.cc:341] FINISH: Updating metrics
0130 02:58:11.402920 (+ 241us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:58:11.402943 (+ 23us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":173039360,"child_traces":[["op",{"apply.queue_time_us":5868,"apply.run_cpu_time_us":1562,"apply.run_wall_time_us":2194,"num_ops":2,"prepare.queue_time_us":23096,"prepare.run_cpu_time_us":2190,"prepare.run_wall_time_us":2191,"raft.queue_time_us":2816,"raft.run_cpu_time_us":648,"raft.run_wall_time_us":650,"replication_time_us":8444}]]}
I20260130 02:58:11.461741 19625 tablet_bootstrap.cc:492] T d27ca7dd87f44102b7680e1f4a88954a P d870164263bb44b6b780a557daea3656: No bootstrap required, opened a new log
I20260130 02:58:11.462152 19625 ts_tablet_manager.cc:1403] T d27ca7dd87f44102b7680e1f4a88954a P d870164263bb44b6b780a557daea3656: Time spent bootstrapping tablet: real 0.247s user 0.008s sys 0.007s
I20260130 02:58:11.474165 19505 tablet_service.cc:1505] Processing CreateTablet for tablet ca06f07bb98f45a7bf63aebc28348e08 (DEFAULT_TABLE table=default.Table_829d1e187e8b4db5b8f4f835e84246e5 [id=a24b4761037d47a383f454a82517131f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:11.475368 19505 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ca06f07bb98f45a7bf63aebc28348e08. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:11.442902 19807 ts_tablet_manager.cc:1434] T 16691c9315e94253bb9b3fb5058e5636 P 371a36d219164974abdf848841528152: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260130 02:58:11.482013 20383 consensus_queue.cc:1048] T 43cec19c51ec4f05b63800c07f9534ef P 78c2c46c7520492db35d51c244fb4f74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260130 02:58:11.488179 18860 catalog_manager.cc:4985] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): tablet 6573198830ad45628e71952a7d9c00a0 (table default.Table_1c2b251034c0418cba750ae4d1308b9f [id=9a5fe99c9417426dba0a12441e7191c6]) successfully deleted
I20260130 02:58:11.490917 20383 consensus_queue.cc:1048] T 8e57ae31ab3941f789ac69627384e3b0 P 78c2c46c7520492db35d51c244fb4f74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260130 02:58:11.474252 19625 raft_consensus.cc:359] T d27ca7dd87f44102b7680e1f4a88954a P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:11.492079 19625 raft_consensus.cc:385] T d27ca7dd87f44102b7680e1f4a88954a P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:11.492413 19625 raft_consensus.cc:740] T d27ca7dd87f44102b7680e1f4a88954a P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d870164263bb44b6b780a557daea3656, State: Initialized, Role: FOLLOWER
I20260130 02:58:11.492720 18874 catalog_manager.cc:5941] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4: Sending DeleteTablet for 3 replicas of tablet 7b80b22bd44c4233837c951aa97e6996
I20260130 02:58:11.493479 19625 consensus_queue.cc:260] T d27ca7dd87f44102b7680e1f4a88954a P d870164263bb44b6b780a557daea3656 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:11.493943 18874 catalog_manager.cc:6724] ReplaceTablet: tablet 7b80b22bd44c4233837c951aa97e6996 deleted and replaced by tablet 1fc8f09d6329407c8078542b4123ef4d
I20260130 02:58:11.494266 18874 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:50284 (request call id 223) took 1468 ms. Trace:
I20260130 02:58:11.495517 19197 tablet_service.cc:1552] Processing DeleteTablet for tablet 7b80b22bd44c4233837c951aa97e6996 with delete_type TABLET_DATA_DELETED (replaced by tablet 1fc8f09d6329407c8078542b4123ef4d) from {username='slave'} at 127.0.0.1:42870
I20260130 02:58:11.475520 20435 leader_election.cc:290] T b89ed298e3dc49d789d166c613ad7fb8 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767)
I20260130 02:58:11.494525 18874 rpcz_store.cc:276] 0130 02:58:10.025980 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:58:11.178878 (+1152898us) service_pool.cc:225] Handling call
0130 02:58:11.267794 (+ 88916us) spinlock_profiling.cc:243] Waited 88 ms on lock 0x7b5400030554. stack: 00007fb0bd3f7373 00007fb0bd3f7270 00007fb0bcf6c574 0000000000508744 00000000005086d9 00007fb0c44490c0 00007fb0c44e7bba 00007fb0c61e6eac 00007fb0c62e51b1 00007fb0c3f26227 00007fb0c3f261b2 00007fb0c3f260e1 00007fb0c3f2605c 00007fb0c3f25312 00007fb0c21dd34c 00007fb0c21dc786
0130 02:58:11.494206 (+226412us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0130 02:58:11.454875 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0130 02:58:11.455265 (+ 390us) write_op.cc:432] Acquiring schema lock in shared mode
0130 02:58:11.455287 (+ 22us) write_op.cc:435] Acquired schema lock
0130 02:58:11.455299 (+ 12us) tablet.cc:660] Decoding operations
0130 02:58:11.455581 (+ 282us) write_op.cc:620] Acquiring the partition lock for write op
0130 02:58:11.455619 (+ 38us) write_op.cc:641] Partition lock acquired for write op
0130 02:58:11.455641 (+ 22us) tablet.cc:683] Acquiring locks for 2 operations
0130 02:58:11.455818 (+ 177us) tablet.cc:699] Row locks acquired
0130 02:58:11.455832 (+ 14us) write_op.cc:260] PREPARE: finished
0130 02:58:11.455942 (+ 110us) write_op.cc:270] Start()
0130 02:58:11.456034 (+ 92us) write_op.cc:276] Timestamp: P: 1769741891455862 usec, L: 0
0130 02:58:11.456049 (+ 15us) op_driver.cc:348] REPLICATION: starting
0130 02:58:11.456518 (+ 469us) log.cc:844] Serialized 662 byte log entry
0130 02:58:11.489800 (+ 33282us) op_driver.cc:464] REPLICATION: finished
0130 02:58:11.490310 (+ 510us) write_op.cc:301] APPLY: starting
0130 02:58:11.490365 (+ 55us) tablet.cc:1365] starting BulkCheckPresence
0130 02:58:11.490462 (+ 97us) tablet.cc:1368] finished BulkCheckPresence
0130 02:58:11.490593 (+ 131us) tablet.cc:1370] starting ApplyRowOperation cycle
0130 02:58:11.490941 (+ 348us) tablet.cc:1381] finished ApplyRowOperation cycle
0130 02:58:11.491153 (+ 212us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0130 02:58:11.491194 (+ 41us) write_op.cc:312] APPLY: finished
0130 02:58:11.491383 (+ 189us) log.cc:844] Serialized 29 byte log entry
0130 02:58:11.491970 (+ 587us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:58:11.492037 (+ 67us) write_op.cc:454] Released schema lock
0130 02:58:11.492067 (+ 30us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"spinlock_wait_cycles":202412928,"child_traces":[["op",{"apply.queue_time_us":358,"num_ops":2,"prepare.queue_time_us":302,"prepare.run_cpu_time_us":1937,"prepare.run_wall_time_us":2877,"raft.queue_time_us":87,"raft.run_cpu_time_us":10,"raft.run_wall_time_us":12,"replication_time_us":33656,"spinlock_wait_cycles":32512}]]}
I20260130 02:58:11.501885 18874 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50284:
table { table_name: "default.Table_4f2d6e4d9c8e465087d65efc9c2de442" } new_table_name: "default.Table_1cbb7ef2df504c1083290b333440c9f7" modify_external_catalogs: true
I20260130 02:58:11.503794 20383 raft_consensus.cc:493] T 0e8018c094024c4788c484e8f5466e00 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:11.504323 20383 raft_consensus.cc:515] T 0e8018c094024c4788c484e8f5466e00 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:11.506290 20383 leader_election.cc:290] T 0e8018c094024c4788c484e8f5466e00 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), d870164263bb44b6b780a557daea3656 (127.18.100.67:39571)
I20260130 02:58:11.507133 19222 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0e8018c094024c4788c484e8f5466e00" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152" is_pre_election: true
W20260130 02:58:11.508610 19314 leader_election.cc:343] T 0e8018c094024c4788c484e8f5466e00 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:11.521183 19197 tablet_service.cc:1505] Processing CreateTablet for tablet dfbbe9045e7c4aa5b520218687e7e4c1 (DEFAULT_TABLE table=default.Table_e03bdbb221e94bd6b535ddc02c863bad [id=2382eeebdc9540d5b588835705b2a4f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:11.523737 19840 tablet_bootstrap.cc:492] T 9fe0448bcc4a4346b3c67fc3f626ed3c P 78c2c46c7520492db35d51c244fb4f74: No bootstrap required, opened a new log
I20260130 02:58:11.523720 19198 tablet_service.cc:1505] Processing CreateTablet for tablet a3258852aed24ed582a79f77860e6f0d (DEFAULT_TABLE table=default.Table_7c48a0d054354501947d11c5fd7b1864 [id=aa4ad1394ab344df8c54ba0031af5590]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:11.523989 19205 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 133e9bfe175f44aab1a53114abe67e4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:11.524160 19840 ts_tablet_manager.cc:1403] T 9fe0448bcc4a4346b3c67fc3f626ed3c P 78c2c46c7520492db35d51c244fb4f74: Time spent bootstrapping tablet: real 0.124s user 0.016s sys 0.005s
I20260130 02:58:11.524718 19198 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a3258852aed24ed582a79f77860e6f0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:11.528108 19201 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ca06f07bb98f45a7bf63aebc28348e08. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:11.528543 19807 tablet_bootstrap.cc:492] T dc09a0f0cecf49738bab04596944c0f3 P 371a36d219164974abdf848841528152: Bootstrap starting.
I20260130 02:58:11.530632 20422 tablet_replica.cc:333] T 7b80b22bd44c4233837c951aa97e6996 P 371a36d219164974abdf848841528152: stopping tablet replica
I20260130 02:58:11.531518 20422 raft_consensus.cc:2243] T 7b80b22bd44c4233837c951aa97e6996 P 371a36d219164974abdf848841528152 [term 1 LEADER]: Raft consensus shutting down.
I20260130 02:58:11.532404 20422 raft_consensus.cc:2272] T 7b80b22bd44c4233837c951aa97e6996 P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260130 02:58:11.537869 19807 tablet_bootstrap.cc:654] T dc09a0f0cecf49738bab04596944c0f3 P 371a36d219164974abdf848841528152: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:11.539119 19197 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dfbbe9045e7c4aa5b520218687e7e4c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:11.540449 19625 ts_tablet_manager.cc:1434] T d27ca7dd87f44102b7680e1f4a88954a P d870164263bb44b6b780a557daea3656: Time spent starting tablet: real 0.078s user 0.005s sys 0.005s
I20260130 02:58:11.541343 19625 tablet_bootstrap.cc:492] T 3b02480d9c6f4328af83f7940716d698 P d870164263bb44b6b780a557daea3656: Bootstrap starting.
I20260130 02:58:11.534364 20422 ts_tablet_manager.cc:1916] T 7b80b22bd44c4233837c951aa97e6996 P 371a36d219164974abdf848841528152: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260130 02:58:11.544409 18860 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.100.125:39711, user_credentials={real_user=slave}} blocked reactor thread for 55114.7us
I20260130 02:58:11.545255 19223 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b89ed298e3dc49d789d166c613ad7fb8" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152" is_pre_election: true
I20260130 02:58:11.547209 19358 tablet_service.cc:1552] Processing DeleteTablet for tablet 7b80b22bd44c4233837c951aa97e6996 with delete_type TABLET_DATA_DELETED (replaced by tablet 1fc8f09d6329407c8078542b4123ef4d) from {username='slave'} at 127.0.0.1:60110
I20260130 02:58:11.547924 19496 tablet_service.cc:1552] Processing DeleteTablet for tablet 7b80b22bd44c4233837c951aa97e6996 with delete_type TABLET_DATA_DELETED (replaced by tablet 1fc8f09d6329407c8078542b4123ef4d) from {username='slave'} at 127.0.0.1:59068
I20260130 02:58:11.548635 19947 tablet_replica.cc:333] T 7b80b22bd44c4233837c951aa97e6996 P d870164263bb44b6b780a557daea3656: stopping tablet replica
I20260130 02:58:11.549363 19947 raft_consensus.cc:2243] T 7b80b22bd44c4233837c951aa97e6996 P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260130 02:58:11.549100 18871 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50284:
name: "default.Table_748cb9c252084fb48ceac74812b82a64"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260130 02:58:11.549795 19947 raft_consensus.cc:2272] T 7b80b22bd44c4233837c951aa97e6996 P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260130 02:58:11.550752 18871 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_748cb9c252084fb48ceac74812b82a64 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260130 02:58:11.551805 19947 ts_tablet_manager.cc:1916] T 7b80b22bd44c4233837c951aa97e6996 P d870164263bb44b6b780a557daea3656: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260130 02:58:11.555269 18866 catalog_manager.cc:5746] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767) does not have the latest schema for tablet 341041cb0be2456e88403367c0814704 (table default.Table_1703c69112744c0e951c1ac9667bbac6 [id=8d76f4328edc491291491c78ba205d30]). Expected version 1 got 0
I20260130 02:58:11.556135 18866 catalog_manager.cc:5746] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767) does not have the latest schema for tablet ee7c8ff92ca9465fa18be6eae63d78ca (table default.Table_dd22e9dc63f84d8688c0071f427f0b25 [id=77979d498aa040828de415129cb15a06]). Expected version 1 got 0
I20260130 02:58:11.557150 18866 catalog_manager.cc:5746] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767) does not have the latest schema for tablet 54be22352e404ff9b798a075693f7f21 (table default.Table_3a858a3cb886427cb8268f42c953ad4c [id=17150cb73e374ff9a1e92c43e0c3f2c1]). Expected version 1 got 0
I20260130 02:58:11.526372 19840 raft_consensus.cc:359] T 9fe0448bcc4a4346b3c67fc3f626ed3c P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:11.557708 19840 raft_consensus.cc:385] T 9fe0448bcc4a4346b3c67fc3f626ed3c P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:11.557979 19840 raft_consensus.cc:740] T 9fe0448bcc4a4346b3c67fc3f626ed3c P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78c2c46c7520492db35d51c244fb4f74, State: Initialized, Role: FOLLOWER
W20260130 02:58:11.559027 19313 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.100.67:39571, user_credentials={real_user=slave}} blocked reactor thread for 80684.4us
I20260130 02:58:11.560375 19526 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0e8018c094024c4788c484e8f5466e00" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
I20260130 02:58:11.560992 19526 raft_consensus.cc:2468] T 0e8018c094024c4788c484e8f5466e00 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78c2c46c7520492db35d51c244fb4f74 in term 0.
I20260130 02:58:11.565443 19840 consensus_queue.cc:260] T 9fe0448bcc4a4346b3c67fc3f626ed3c P 78c2c46c7520492db35d51c244fb4f74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:11.574947 19376 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b89ed298e3dc49d789d166c613ad7fb8" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74" is_pre_election: true
I20260130 02:58:11.581525 19358 tablet_service.cc:1505] Processing CreateTablet for tablet dfbbe9045e7c4aa5b520218687e7e4c1 (DEFAULT_TABLE table=default.Table_e03bdbb221e94bd6b535ddc02c863bad [id=2382eeebdc9540d5b588835705b2a4f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:11.582939 19840 ts_tablet_manager.cc:1434] T 9fe0448bcc4a4346b3c67fc3f626ed3c P 78c2c46c7520492db35d51c244fb4f74: Time spent starting tablet: real 0.059s user 0.007s sys 0.000s
I20260130 02:58:11.584860 19840 tablet_bootstrap.cc:492] T 4c139aa8cd184f488cec4e05ef7dcada P 78c2c46c7520492db35d51c244fb4f74: Bootstrap starting.
I20260130 02:58:11.581030 19313 leader_election.cc:304] T 0e8018c094024c4788c484e8f5466e00 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 78c2c46c7520492db35d51c244fb4f74, d870164263bb44b6b780a557daea3656; no voters: 371a36d219164974abdf848841528152
I20260130 02:58:11.588182 19462 leader_election.cc:304] T b89ed298e3dc49d789d166c613ad7fb8 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d870164263bb44b6b780a557daea3656; no voters: 371a36d219164974abdf848841528152, 78c2c46c7520492db35d51c244fb4f74
I20260130 02:58:11.591185 19358 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dfbbe9045e7c4aa5b520218687e7e4c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:11.592314 20374 raft_consensus.cc:2804] T 0e8018c094024c4788c484e8f5466e00 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260130 02:58:11.592641 20374 raft_consensus.cc:493] T 0e8018c094024c4788c484e8f5466e00 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260130 02:58:11.592931 20374 raft_consensus.cc:3060] T 0e8018c094024c4788c484e8f5466e00 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:11.593516 19840 tablet_bootstrap.cc:654] T 4c139aa8cd184f488cec4e05ef7dcada P 78c2c46c7520492db35d51c244fb4f74: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:11.595443 19496 tablet_service.cc:1505] Processing CreateTablet for tablet dfbbe9045e7c4aa5b520218687e7e4c1 (DEFAULT_TABLE table=default.Table_e03bdbb221e94bd6b535ddc02c863bad [id=2382eeebdc9540d5b588835705b2a4f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:11.596756 20476 raft_consensus.cc:2749] T b89ed298e3dc49d789d166c613ad7fb8 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260130 02:58:11.599886 19625 tablet_bootstrap.cc:654] T 3b02480d9c6f4328af83f7940716d698 P d870164263bb44b6b780a557daea3656: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:11.603927 19501 tablet_service.cc:1505] Processing CreateTablet for tablet a3258852aed24ed582a79f77860e6f0d (DEFAULT_TABLE table=default.Table_7c48a0d054354501947d11c5fd7b1864 [id=aa4ad1394ab344df8c54ba0031af5590]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:11.604426 20374 raft_consensus.cc:515] T 0e8018c094024c4788c484e8f5466e00 P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:11.606885 19225 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0e8018c094024c4788c484e8f5466e00" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152"
W20260130 02:58:11.608433 19314 leader_election.cc:343] T 0e8018c094024c4788c484e8f5466e00 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:11.610894 19501 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a3258852aed24ed582a79f77860e6f0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:11.612563 19496 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dfbbe9045e7c4aa5b520218687e7e4c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:11.615392 20374 leader_election.cc:290] T 0e8018c094024c4788c484e8f5466e00 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 election: Requested vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), d870164263bb44b6b780a557daea3656 (127.18.100.67:39571)
I20260130 02:58:11.616078 19807 tablet_bootstrap.cc:492] T dc09a0f0cecf49738bab04596944c0f3 P 371a36d219164974abdf848841528152: No bootstrap required, opened a new log
I20260130 02:58:11.616478 19807 ts_tablet_manager.cc:1403] T dc09a0f0cecf49738bab04596944c0f3 P 371a36d219164974abdf848841528152: Time spent bootstrapping tablet: real 0.088s user 0.007s sys 0.001s
I20260130 02:58:11.622121 19526 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0e8018c094024c4788c484e8f5466e00" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656"
I20260130 02:58:11.622722 19526 raft_consensus.cc:3060] T 0e8018c094024c4788c484e8f5466e00 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:11.618547 19807 raft_consensus.cc:359] T dc09a0f0cecf49738bab04596944c0f3 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:11.632369 19521 raft_consensus.cc:3060] T 04f4ac27800c4ac3a79ae12d35479757 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:11.636415 20414 ts_tablet_manager.cc:1929] T f54addb1a4304ce69c711ae0cc3b452c P 78c2c46c7520492db35d51c244fb4f74: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260130 02:58:11.641355 19526 raft_consensus.cc:2468] T 0e8018c094024c4788c484e8f5466e00 P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78c2c46c7520492db35d51c244fb4f74 in term 1.
I20260130 02:58:11.647804 19313 leader_election.cc:304] T 0e8018c094024c4788c484e8f5466e00 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 78c2c46c7520492db35d51c244fb4f74, d870164263bb44b6b780a557daea3656; no voters: 371a36d219164974abdf848841528152
I20260130 02:58:11.648694 20474 raft_consensus.cc:2804] T 0e8018c094024c4788c484e8f5466e00 P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Leader election won for term 1
I20260130 02:58:11.649292 20474 raft_consensus.cc:697] T 0e8018c094024c4788c484e8f5466e00 P 78c2c46c7520492db35d51c244fb4f74 [term 1 LEADER]: Becoming Leader. State: Replica: 78c2c46c7520492db35d51c244fb4f74, State: Running, Role: LEADER
I20260130 02:58:11.627190 19807 raft_consensus.cc:385] T dc09a0f0cecf49738bab04596944c0f3 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:11.650710 19807 raft_consensus.cc:740] T dc09a0f0cecf49738bab04596944c0f3 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 371a36d219164974abdf848841528152, State: Initialized, Role: FOLLOWER
I20260130 02:58:11.650200 20474 consensus_queue.cc:237] T 0e8018c094024c4788c484e8f5466e00 P 78c2c46c7520492db35d51c244fb4f74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:11.648025 20414 log.cc:1199] T f54addb1a4304ce69c711ae0cc3b452c P 78c2c46c7520492db35d51c244fb4f74: Deleting WAL directory at /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-1/wal/wals/f54addb1a4304ce69c711ae0cc3b452c
I20260130 02:58:11.651394 19807 consensus_queue.cc:260] T dc09a0f0cecf49738bab04596944c0f3 P 371a36d219164974abdf848841528152 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:11.653472 20414 ts_tablet_manager.cc:1950] T f54addb1a4304ce69c711ae0cc3b452c P 78c2c46c7520492db35d51c244fb4f74: Deleting consensus metadata
I20260130 02:58:11.653609 19807 ts_tablet_manager.cc:1434] T dc09a0f0cecf49738bab04596944c0f3 P 371a36d219164974abdf848841528152: Time spent starting tablet: real 0.037s user 0.001s sys 0.005s
I20260130 02:58:11.655471 19351 tablet_service.cc:1505] Processing CreateTablet for tablet a3258852aed24ed582a79f77860e6f0d (DEFAULT_TABLE table=default.Table_7c48a0d054354501947d11c5fd7b1864 [id=aa4ad1394ab344df8c54ba0031af5590]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:11.656528 19351 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a3258852aed24ed582a79f77860e6f0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:11.657110 20414 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60110 (request call id 222) took 1155 ms. Trace:
I20260130 02:58:11.657451 20414 rpcz_store.cc:276] 0130 02:58:10.502203 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:58:10.505675 (+ 3472us) service_pool.cc:225] Handling call
0130 02:58:11.178794 (+673119us) ts_tablet_manager.cc:1151] Deleting tablet f54addb1a4304ce69c711ae0cc3b452c
0130 02:58:11.178839 (+ 45us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0130 02:58:11.477052 (+298213us) tablet_metadata.cc:641] Metadata flushed
0130 02:58:11.636192 (+159140us) tablet_metadata.cc:641] Metadata flushed
0130 02:58:11.657042 (+ 20850us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":672498}
I20260130 02:58:11.660195 19625 tablet_bootstrap.cc:492] T 3b02480d9c6f4328af83f7940716d698 P d870164263bb44b6b780a557daea3656: No bootstrap required, opened a new log
I20260130 02:58:11.660593 19625 ts_tablet_manager.cc:1403] T 3b02480d9c6f4328af83f7940716d698 P d870164263bb44b6b780a557daea3656: Time spent bootstrapping tablet: real 0.119s user 0.008s sys 0.000s
I20260130 02:58:11.669585 20481 raft_consensus.cc:493] T 16691c9315e94253bb9b3fb5058e5636 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:11.669854 20422 ts_tablet_manager.cc:1929] T 7b80b22bd44c4233837c951aa97e6996 P 371a36d219164974abdf848841528152: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260130 02:58:11.670027 20481 raft_consensus.cc:515] T 16691c9315e94253bb9b3fb5058e5636 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:11.673319 20481 leader_election.cc:290] T 16691c9315e94253bb9b3fb5058e5636 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), d870164263bb44b6b780a557daea3656 (127.18.100.67:39571)
I20260130 02:58:11.673326 18866 catalog_manager.cc:5746] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767) does not have the latest schema for tablet 127a84346b9f473d9e7f739b1e65feb8 (table default.Table_4f2d6e4d9c8e465087d65efc9c2de442 [id=1515db389f284e2f9ed1a3cd0061d0f1]). Expected version 5 got 0
I20260130 02:58:11.676628 19350 tablet_service.cc:1505] Processing CreateTablet for tablet ca06f07bb98f45a7bf63aebc28348e08 (DEFAULT_TABLE table=default.Table_829d1e187e8b4db5b8f4f835e84246e5 [id=a24b4761037d47a383f454a82517131f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:11.682842 19625 raft_consensus.cc:359] T 3b02480d9c6f4328af83f7940716d698 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:11.683473 19625 raft_consensus.cc:385] T 3b02480d9c6f4328af83f7940716d698 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:11.683702 19625 raft_consensus.cc:740] T 3b02480d9c6f4328af83f7940716d698 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d870164263bb44b6b780a557daea3656, State: Initialized, Role: FOLLOWER
I20260130 02:58:11.684224 19625 consensus_queue.cc:260] T 3b02480d9c6f4328af83f7940716d698 P d870164263bb44b6b780a557daea3656 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:11.672580 19225 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "16691c9315e94253bb9b3fb5058e5636" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152" is_pre_election: true
I20260130 02:58:11.688046 19225 raft_consensus.cc:2468] T 16691c9315e94253bb9b3fb5058e5636 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78c2c46c7520492db35d51c244fb4f74 in term 0.
I20260130 02:58:11.688885 20422 log.cc:1199] T 7b80b22bd44c4233837c951aa97e6996 P 371a36d219164974abdf848841528152: Deleting WAL directory at /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-0/wal/wals/7b80b22bd44c4233837c951aa97e6996
I20260130 02:58:11.690014 19807 tablet_bootstrap.cc:492] T 4c139aa8cd184f488cec4e05ef7dcada P 371a36d219164974abdf848841528152: Bootstrap starting.
I20260130 02:58:11.690593 20422 ts_tablet_manager.cc:1950] T 7b80b22bd44c4233837c951aa97e6996 P 371a36d219164974abdf848841528152: Deleting consensus metadata
I20260130 02:58:11.697103 18861 catalog_manager.cc:4985] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021): tablet 7b80b22bd44c4233837c951aa97e6996 (table default.Table_7d1c27ab3515406ba5b5333c90df093e [id=b6fabeeae79941dc8094d0f2b51704f9]) successfully deleted
I20260130 02:58:11.698721 18860 catalog_manager.cc:4985] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767): tablet f54addb1a4304ce69c711ae0cc3b452c (table default.Table_2d345d0f0957486c86c2cf2e34b00941 [id=4c3e0d3ecd5642a3b7fc7d20e2521418]) successfully deleted
I20260130 02:58:11.683365 20414 tablet_replica.cc:333] T b2ac91da01954c2390b6b7dcc06c394f P 78c2c46c7520492db35d51c244fb4f74: stopping tablet replica
I20260130 02:58:11.690143 19314 leader_election.cc:304] T 16691c9315e94253bb9b3fb5058e5636 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 371a36d219164974abdf848841528152, 78c2c46c7520492db35d51c244fb4f74; no voters:
I20260130 02:58:11.707341 19350 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ca06f07bb98f45a7bf63aebc28348e08. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:11.691099 20481 raft_consensus.cc:493] T 753344737b5941d9a93b6779e72f63d4 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:11.711304 20481 raft_consensus.cc:515] T 753344737b5941d9a93b6779e72f63d4 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:11.713052 20481 leader_election.cc:290] T 753344737b5941d9a93b6779e72f63d4 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), d870164263bb44b6b780a557daea3656 (127.18.100.67:39571)
I20260130 02:58:11.713475 20481 raft_consensus.cc:2804] T 16691c9315e94253bb9b3fb5058e5636 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260130 02:58:11.713757 20481 raft_consensus.cc:493] T 16691c9315e94253bb9b3fb5058e5636 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260130 02:58:11.714052 20481 raft_consensus.cc:3060] T 16691c9315e94253bb9b3fb5058e5636 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:11.718297 19197 tablet_service.cc:1505] Processing CreateTablet for tablet d480bf18140245e3870a62582d5a8318 (DEFAULT_TABLE table=default.Table_109c8b6c17d4442b939bde9441d0476f [id=dcf82ea15d6044c2b023f81f0a579ade]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:11.719087 19223 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "753344737b5941d9a93b6779e72f63d4" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152" is_pre_election: true
I20260130 02:58:11.718709 18866 catalog_manager.cc:5654] T 04f4ac27800c4ac3a79ae12d35479757 P 78c2c46c7520492db35d51c244fb4f74 reported cstate change: term changed from 0 to 1, leader changed from <none> to 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66). New cstate: current_term: 1 leader_uuid: "78c2c46c7520492db35d51c244fb4f74" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } health_report { overall_health: UNKNOWN } } }
I20260130 02:58:11.719507 19197 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d480bf18140245e3870a62582d5a8318. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:11.722615 19807 tablet_bootstrap.cc:654] T 4c139aa8cd184f488cec4e05ef7dcada P 371a36d219164974abdf848841528152: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:11.719578 19223 raft_consensus.cc:2468] T 753344737b5941d9a93b6779e72f63d4 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78c2c46c7520492db35d51c244fb4f74 in term 0.
I20260130 02:58:11.723299 19840 tablet_bootstrap.cc:492] T 4c139aa8cd184f488cec4e05ef7dcada P 78c2c46c7520492db35d51c244fb4f74: No bootstrap required, opened a new log
I20260130 02:58:11.723706 19840 ts_tablet_manager.cc:1403] T 4c139aa8cd184f488cec4e05ef7dcada P 78c2c46c7520492db35d51c244fb4f74: Time spent bootstrapping tablet: real 0.139s user 0.011s sys 0.000s
I20260130 02:58:11.724079 19314 leader_election.cc:304] T 753344737b5941d9a93b6779e72f63d4 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 371a36d219164974abdf848841528152, 78c2c46c7520492db35d51c244fb4f74; no voters:
I20260130 02:58:11.724889 19198 tablet_service.cc:1505] Processing CreateTablet for tablet 657e9b0d8c05413f904129dce745710b (DEFAULT_TABLE table=default.Table_526677c8ef4144bd9de7a54a5e78e2be [id=b16908b75587459597a978c71756451c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:11.725342 20479 raft_consensus.cc:2804] T 753344737b5941d9a93b6779e72f63d4 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260130 02:58:11.725632 20479 raft_consensus.cc:493] T 753344737b5941d9a93b6779e72f63d4 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260130 02:58:11.725809 19198 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 657e9b0d8c05413f904129dce745710b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:11.725972 20479 raft_consensus.cc:3060] T 753344737b5941d9a93b6779e72f63d4 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:11.727259 19356 tablet_service.cc:1505] Processing CreateTablet for tablet 657e9b0d8c05413f904129dce745710b (DEFAULT_TABLE table=default.Table_526677c8ef4144bd9de7a54a5e78e2be [id=b16908b75587459597a978c71756451c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:11.727404 19355 tablet_service.cc:1505] Processing CreateTablet for tablet d480bf18140245e3870a62582d5a8318 (DEFAULT_TABLE table=default.Table_109c8b6c17d4442b939bde9441d0476f [id=dcf82ea15d6044c2b023f81f0a579ade]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:11.728261 19356 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 657e9b0d8c05413f904129dce745710b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:11.728791 19355 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d480bf18140245e3870a62582d5a8318. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:11.732427 20469 consensus_queue.cc:1048] T d55291d7cea64a0d9a345ce712fbfa80 P 78c2c46c7520492db35d51c244fb4f74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260130 02:58:11.735536 20414 raft_consensus.cc:2243] T b2ac91da01954c2390b6b7dcc06c394f P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260130 02:58:11.736455 20414 raft_consensus.cc:2272] T b2ac91da01954c2390b6b7dcc06c394f P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260130 02:58:11.738816 20414 ts_tablet_manager.cc:1916] T b2ac91da01954c2390b6b7dcc06c394f P 78c2c46c7520492db35d51c244fb4f74: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260130 02:58:11.748149 20481 raft_consensus.cc:515] T 16691c9315e94253bb9b3fb5058e5636 P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:11.753021 19223 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "16691c9315e94253bb9b3fb5058e5636" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152"
I20260130 02:58:11.753538 19223 raft_consensus.cc:3060] T 16691c9315e94253bb9b3fb5058e5636 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:11.762171 19223 raft_consensus.cc:2468] T 16691c9315e94253bb9b3fb5058e5636 P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78c2c46c7520492db35d51c244fb4f74 in term 1.
I20260130 02:58:11.763708 19314 leader_election.cc:304] T 16691c9315e94253bb9b3fb5058e5636 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 371a36d219164974abdf848841528152, 78c2c46c7520492db35d51c244fb4f74; no voters:
I20260130 02:58:11.764532 20474 raft_consensus.cc:2804] T 16691c9315e94253bb9b3fb5058e5636 P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Leader election won for term 1
I20260130 02:58:11.765091 20474 raft_consensus.cc:697] T 16691c9315e94253bb9b3fb5058e5636 P 78c2c46c7520492db35d51c244fb4f74 [term 1 LEADER]: Becoming Leader. State: Replica: 78c2c46c7520492db35d51c244fb4f74, State: Running, Role: LEADER
I20260130 02:58:11.765889 20474 consensus_queue.cc:237] T 16691c9315e94253bb9b3fb5058e5636 P 78c2c46c7520492db35d51c244fb4f74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:11.775959 18871 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50284:
name: "default.Table_fa63c9eff99d4ac1b185a4c4ce2df031"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260130 02:58:11.783806 18871 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fa63c9eff99d4ac1b185a4c4ce2df031 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260130 02:58:11.792750 19354 tablet_service.cc:1505] Processing CreateTablet for tablet aae68390dbbe4dc58ce3d3bd01725ae6 (DEFAULT_TABLE table=default.Table_f3e214a955f6447482099175b92ecb5c [id=76a688022fd9421a84de059b3c1da77a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:11.793788 19354 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aae68390dbbe4dc58ce3d3bd01725ae6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:11.749500 19840 raft_consensus.cc:359] T 4c139aa8cd184f488cec4e05ef7dcada P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:11.804998 19840 raft_consensus.cc:385] T 4c139aa8cd184f488cec4e05ef7dcada P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:11.805301 19840 raft_consensus.cc:740] T 4c139aa8cd184f488cec4e05ef7dcada P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78c2c46c7520492db35d51c244fb4f74, State: Initialized, Role: FOLLOWER
I20260130 02:58:11.750423 20479 raft_consensus.cc:515] T 753344737b5941d9a93b6779e72f63d4 P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:11.808444 20479 leader_election.cc:290] T 753344737b5941d9a93b6779e72f63d4 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 election: Requested vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), d870164263bb44b6b780a557daea3656 (127.18.100.67:39571)
I20260130 02:58:11.809481 19223 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "753344737b5941d9a93b6779e72f63d4" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152"
I20260130 02:58:11.809990 19223 raft_consensus.cc:3060] T 753344737b5941d9a93b6779e72f63d4 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:11.816376 20481 leader_election.cc:290] T 16691c9315e94253bb9b3fb5058e5636 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 election: Requested vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), d870164263bb44b6b780a557daea3656 (127.18.100.67:39571)
I20260130 02:58:11.826622 19840 consensus_queue.cc:260] T 4c139aa8cd184f488cec4e05ef7dcada P 78c2c46c7520492db35d51c244fb4f74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:11.828285 19807 tablet_bootstrap.cc:492] T 4c139aa8cd184f488cec4e05ef7dcada P 371a36d219164974abdf848841528152: No bootstrap required, opened a new log
I20260130 02:58:11.828681 19807 ts_tablet_manager.cc:1403] T 4c139aa8cd184f488cec4e05ef7dcada P 371a36d219164974abdf848841528152: Time spent bootstrapping tablet: real 0.139s user 0.006s sys 0.003s
I20260130 02:58:11.828737 19840 ts_tablet_manager.cc:1434] T 4c139aa8cd184f488cec4e05ef7dcada P 78c2c46c7520492db35d51c244fb4f74: Time spent starting tablet: real 0.105s user 0.006s sys 0.000s
I20260130 02:58:11.832263 19502 tablet_service.cc:1505] Processing CreateTablet for tablet d480bf18140245e3870a62582d5a8318 (DEFAULT_TABLE table=default.Table_109c8b6c17d4442b939bde9441d0476f [id=dcf82ea15d6044c2b023f81f0a579ade]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:11.834350 19503 tablet_service.cc:1505] Processing CreateTablet for tablet aae68390dbbe4dc58ce3d3bd01725ae6 (DEFAULT_TABLE table=default.Table_f3e214a955f6447482099175b92ecb5c [id=76a688022fd9421a84de059b3c1da77a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:11.834976 19524 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "16691c9315e94253bb9b3fb5058e5636" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656"
I20260130 02:58:11.832263 19508 tablet_service.cc:1505] Processing CreateTablet for tablet 657e9b0d8c05413f904129dce745710b (DEFAULT_TABLE table=default.Table_526677c8ef4144bd9de7a54a5e78e2be [id=b16908b75587459597a978c71756451c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:11.837913 19526 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "753344737b5941d9a93b6779e72f63d4" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
I20260130 02:58:11.833117 19521 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "16691c9315e94253bb9b3fb5058e5636" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
I20260130 02:58:11.845417 19502 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d480bf18140245e3870a62582d5a8318. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:11.834408 19523 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "753344737b5941d9a93b6779e72f63d4" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656"
I20260130 02:58:11.849143 19223 raft_consensus.cc:2468] T 753344737b5941d9a93b6779e72f63d4 P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78c2c46c7520492db35d51c244fb4f74 in term 1.
I20260130 02:58:11.850239 19314 leader_election.cc:304] T 753344737b5941d9a93b6779e72f63d4 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 371a36d219164974abdf848841528152, 78c2c46c7520492db35d51c244fb4f74; no voters:
I20260130 02:58:11.851115 20481 raft_consensus.cc:2804] T 753344737b5941d9a93b6779e72f63d4 P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Leader election won for term 1
I20260130 02:58:11.851215 19503 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aae68390dbbe4dc58ce3d3bd01725ae6. 1 dirs total, 0 dirs full, 0 dirs failed
W20260130 02:58:11.851956 19313 leader_election.cc:343] T 16691c9315e94253bb9b3fb5058e5636 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): Illegal state: must be running to vote when last-logged opid is not known
W20260130 02:58:11.852923 19313 consensus_peers.cc:597] T d55291d7cea64a0d9a345ce712fbfa80 P 78c2c46c7520492db35d51c244fb4f74 -> Peer d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): Couldn't send request to peer d870164263bb44b6b780a557daea3656. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260130 02:58:11.853245 20481 raft_consensus.cc:697] T 753344737b5941d9a93b6779e72f63d4 P 78c2c46c7520492db35d51c244fb4f74 [term 1 LEADER]: Becoming Leader. State: Replica: 78c2c46c7520492db35d51c244fb4f74, State: Running, Role: LEADER
W20260130 02:58:11.853549 19313 leader_election.cc:343] T 753344737b5941d9a93b6779e72f63d4 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): Illegal state: must be running to vote when last-logged opid is not known
W20260130 02:58:11.854420 19313 leader_election.cc:343] T 16691c9315e94253bb9b3fb5058e5636 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): Illegal state: must be running to vote when last-logged opid is not known
W20260130 02:58:11.856827 19313 leader_election.cc:343] T 753344737b5941d9a93b6779e72f63d4 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:11.859138 19508 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 657e9b0d8c05413f904129dce745710b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:11.869303 19205 tablet_service.cc:1505] Processing CreateTablet for tablet aae68390dbbe4dc58ce3d3bd01725ae6 (DEFAULT_TABLE table=default.Table_f3e214a955f6447482099175b92ecb5c [id=76a688022fd9421a84de059b3c1da77a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:11.870393 19205 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aae68390dbbe4dc58ce3d3bd01725ae6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:11.879639 19625 ts_tablet_manager.cc:1434] T 3b02480d9c6f4328af83f7940716d698 P d870164263bb44b6b780a557daea3656: Time spent starting tablet: real 0.219s user 0.006s sys 0.003s
I20260130 02:58:11.881103 19625 tablet_bootstrap.cc:492] T a8a5038e9def44eb8b3b1e4aeae25652 P d870164263bb44b6b780a557daea3656: Bootstrap starting.
I20260130 02:58:11.854054 20481 consensus_queue.cc:237] T 753344737b5941d9a93b6779e72f63d4 P 78c2c46c7520492db35d51c244fb4f74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:11.892386 20435 raft_consensus.cc:493] T 3ae5472cba1e4613abe4cda42ccb2496 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:11.893143 19947 ts_tablet_manager.cc:1929] T 7b80b22bd44c4233837c951aa97e6996 P d870164263bb44b6b780a557daea3656: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260130 02:58:11.893039 20435 raft_consensus.cc:515] T 3ae5472cba1e4613abe4cda42ccb2496 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:11.893643 19947 log.cc:1199] T 7b80b22bd44c4233837c951aa97e6996 P d870164263bb44b6b780a557daea3656: Deleting WAL directory at /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-2/wal/wals/7b80b22bd44c4233837c951aa97e6996
I20260130 02:58:11.894546 19947 ts_tablet_manager.cc:1950] T 7b80b22bd44c4233837c951aa97e6996 P d870164263bb44b6b780a557daea3656: Deleting consensus metadata
I20260130 02:58:11.895341 20435 leader_election.cc:290] T 3ae5472cba1e4613abe4cda42ccb2496 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767)
I20260130 02:58:11.899833 19198 tablet_service.cc:1505] Processing CreateTablet for tablet 49efcfced0b74c48962393149ac2a500 (DEFAULT_TABLE table=default.Table_2e81f0a0cbc449a397141b717db17025 [id=a9c919f57c0c437cb35a6921103e45a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:11.900029 19376 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3ae5472cba1e4613abe4cda42ccb2496" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74" is_pre_election: true
I20260130 02:58:11.900939 19198 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 49efcfced0b74c48962393149ac2a500. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:11.900619 19807 raft_consensus.cc:359] T 4c139aa8cd184f488cec4e05ef7dcada P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:11.901389 19807 raft_consensus.cc:385] T 4c139aa8cd184f488cec4e05ef7dcada P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:11.901651 19807 raft_consensus.cc:740] T 4c139aa8cd184f488cec4e05ef7dcada P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 371a36d219164974abdf848841528152, State: Initialized, Role: FOLLOWER
I20260130 02:58:11.902305 19807 consensus_queue.cc:260] T 4c139aa8cd184f488cec4e05ef7dcada P 371a36d219164974abdf848841528152 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:11.907925 19840 tablet_bootstrap.cc:492] T 3389db58493448aea3cf00aca411047c P 78c2c46c7520492db35d51c244fb4f74: Bootstrap starting.
I20260130 02:58:11.909691 19625 tablet_bootstrap.cc:654] T a8a5038e9def44eb8b3b1e4aeae25652 P d870164263bb44b6b780a557daea3656: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:11.911764 18870 catalog_manager.cc:5654] T b89ed298e3dc49d789d166c613ad7fb8 P 371a36d219164974abdf848841528152 reported cstate change: term changed from 0 to 1, leader changed from <none> to 371a36d219164974abdf848841528152 (127.18.100.65). New cstate: current_term: 1 leader_uuid: "371a36d219164974abdf848841528152" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } health_report { overall_health: UNKNOWN } } }
I20260130 02:58:11.918352 19501 tablet_service.cc:1505] Processing CreateTablet for tablet 49efcfced0b74c48962393149ac2a500 (DEFAULT_TABLE table=default.Table_2e81f0a0cbc449a397141b717db17025 [id=a9c919f57c0c437cb35a6921103e45a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:11.919502 19501 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 49efcfced0b74c48962393149ac2a500. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:11.922551 19840 tablet_bootstrap.cc:654] T 3389db58493448aea3cf00aca411047c P 78c2c46c7520492db35d51c244fb4f74: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:11.927029 19351 tablet_service.cc:1505] Processing CreateTablet for tablet 49efcfced0b74c48962393149ac2a500 (DEFAULT_TABLE table=default.Table_2e81f0a0cbc449a397141b717db17025 [id=a9c919f57c0c437cb35a6921103e45a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:11.939852 19351 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 49efcfced0b74c48962393149ac2a500. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:11.940689 20414 ts_tablet_manager.cc:1929] T b2ac91da01954c2390b6b7dcc06c394f P 78c2c46c7520492db35d51c244fb4f74: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260130 02:58:11.949846 19223 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3ae5472cba1e4613abe4cda42ccb2496" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152" is_pre_election: true
I20260130 02:58:11.954243 20490 raft_consensus.cc:493] T 3b02480d9c6f4328af83f7940716d698 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:11.954721 20490 raft_consensus.cc:515] T 3b02480d9c6f4328af83f7940716d698 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:11.957283 20490 leader_election.cc:290] T 3b02480d9c6f4328af83f7940716d698 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767)
I20260130 02:58:11.957479 19376 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3b02480d9c6f4328af83f7940716d698" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74" is_pre_election: true
I20260130 02:58:11.960769 20414 log.cc:1199] T b2ac91da01954c2390b6b7dcc06c394f P 78c2c46c7520492db35d51c244fb4f74: Deleting WAL directory at /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-1/wal/wals/b2ac91da01954c2390b6b7dcc06c394f
W20260130 02:58:11.960969 19462 leader_election.cc:343] T 3b02480d9c6f4328af83f7940716d698 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:11.963688 20414 ts_tablet_manager.cc:1950] T b2ac91da01954c2390b6b7dcc06c394f P 78c2c46c7520492db35d51c244fb4f74: Deleting consensus metadata
I20260130 02:58:11.970166 20414 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60110 (request call id 226) took 1426 ms. Trace:
I20260130 02:58:11.977419 20414 rpcz_store.cc:276] 0130 02:58:10.543533 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:58:10.543658 (+ 125us) service_pool.cc:225] Handling call
0130 02:58:11.683207 (+1139549us) ts_tablet_manager.cc:1151] Deleting tablet b2ac91da01954c2390b6b7dcc06c394f
0130 02:58:11.683255 (+ 48us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0130 02:58:11.880947 (+197692us) tablet_metadata.cc:641] Metadata flushed
0130 02:58:11.940471 (+ 59524us) tablet_metadata.cc:641] Metadata flushed
0130 02:58:11.970101 (+ 29630us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1138950}
I20260130 02:58:11.983695 20414 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60110 (request call id 227) took 1430 ms. Trace:
I20260130 02:58:11.984444 18860 catalog_manager.cc:4985] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767): tablet b2ac91da01954c2390b6b7dcc06c394f (table default.Table_f2f790e8e46840ecb199c34fae899ada [id=60b8e98e3f864716a6c1aa3816612211]) successfully deleted
I20260130 02:58:11.985055 18860 catalog_manager.cc:4985] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): tablet 7b80b22bd44c4233837c951aa97e6996 (table default.Table_7d1c27ab3515406ba5b5333c90df093e [id=b6fabeeae79941dc8094d0f2b51704f9]) successfully deleted
I20260130 02:58:11.986315 20414 rpcz_store.cc:276] 0130 02:58:10.552846 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:58:10.552973 (+ 127us) service_pool.cc:225] Handling call
0130 02:58:11.983427 (+1430454us) ts_tablet_manager.cc:1151] Deleting tablet 5ca9919f80954e74a4d91c94b74de0e4
0130 02:58:11.983468 (+ 41us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0130 02:58:11.983635 (+ 167us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1429844}
I20260130 02:58:11.994407 20414 tablet_replica.cc:333] T 7d96e93d95f14af79cfb268626559b1e P 78c2c46c7520492db35d51c244fb4f74: stopping tablet replica
I20260130 02:58:11.995287 20414 raft_consensus.cc:2243] T 7d96e93d95f14af79cfb268626559b1e P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260130 02:58:11.995843 20414 raft_consensus.cc:2272] T 7d96e93d95f14af79cfb268626559b1e P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260130 02:58:11.997864 20414 ts_tablet_manager.cc:1916] T 7d96e93d95f14af79cfb268626559b1e P 78c2c46c7520492db35d51c244fb4f74: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260130 02:58:11.998030 19223 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3b02480d9c6f4328af83f7940716d698" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152" is_pre_election: true
I20260130 02:58:11.998533 19223 raft_consensus.cc:2468] T 3b02480d9c6f4328af83f7940716d698 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d870164263bb44b6b780a557daea3656 in term 0.
I20260130 02:58:12.001951 19463 leader_election.cc:304] T 3ae5472cba1e4613abe4cda42ccb2496 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d870164263bb44b6b780a557daea3656; no voters: 371a36d219164974abdf848841528152, 78c2c46c7520492db35d51c244fb4f74
I20260130 02:58:12.002745 20490 raft_consensus.cc:2749] T 3ae5472cba1e4613abe4cda42ccb2496 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260130 02:58:12.004271 19625 tablet_bootstrap.cc:492] T a8a5038e9def44eb8b3b1e4aeae25652 P d870164263bb44b6b780a557daea3656: No bootstrap required, opened a new log
I20260130 02:58:12.004668 19625 ts_tablet_manager.cc:1403] T a8a5038e9def44eb8b3b1e4aeae25652 P d870164263bb44b6b780a557daea3656: Time spent bootstrapping tablet: real 0.124s user 0.005s sys 0.005s
I20260130 02:58:12.008728 19625 raft_consensus.cc:359] T a8a5038e9def44eb8b3b1e4aeae25652 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:12.009341 19625 raft_consensus.cc:385] T a8a5038e9def44eb8b3b1e4aeae25652 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:12.009590 19625 raft_consensus.cc:740] T a8a5038e9def44eb8b3b1e4aeae25652 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d870164263bb44b6b780a557daea3656, State: Initialized, Role: FOLLOWER
I20260130 02:58:12.010322 19625 consensus_queue.cc:260] T a8a5038e9def44eb8b3b1e4aeae25652 P d870164263bb44b6b780a557daea3656 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:12.020149 19807 ts_tablet_manager.cc:1434] T 4c139aa8cd184f488cec4e05ef7dcada P 371a36d219164974abdf848841528152: Time spent starting tablet: real 0.191s user 0.003s sys 0.004s
I20260130 02:58:12.022295 19526 raft_consensus.cc:3060] T b89ed298e3dc49d789d166c613ad7fb8 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:12.025884 19625 ts_tablet_manager.cc:1434] T a8a5038e9def44eb8b3b1e4aeae25652 P d870164263bb44b6b780a557daea3656: Time spent starting tablet: real 0.021s user 0.007s sys 0.000s
I20260130 02:58:12.021086 19807 tablet_bootstrap.cc:492] T 9fe0448bcc4a4346b3c67fc3f626ed3c P 371a36d219164974abdf848841528152: Bootstrap starting.
I20260130 02:58:12.044823 19463 leader_election.cc:304] T 3b02480d9c6f4328af83f7940716d698 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 371a36d219164974abdf848841528152, d870164263bb44b6b780a557daea3656; no voters: 78c2c46c7520492db35d51c244fb4f74
I20260130 02:58:12.048153 19625 tablet_bootstrap.cc:492] T dc09a0f0cecf49738bab04596944c0f3 P d870164263bb44b6b780a557daea3656: Bootstrap starting.
I20260130 02:58:12.050019 20490 raft_consensus.cc:2804] T 3b02480d9c6f4328af83f7940716d698 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260130 02:58:12.050387 20490 raft_consensus.cc:493] T 3b02480d9c6f4328af83f7940716d698 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260130 02:58:12.050750 20490 raft_consensus.cc:3060] T 3b02480d9c6f4328af83f7940716d698 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Advancing to term 1
W20260130 02:58:12.051400 19159 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.100.67:39571, user_credentials={real_user=slave}} blocked reactor thread for 47937.9us
I20260130 02:58:12.054476 19840 tablet_bootstrap.cc:492] T 3389db58493448aea3cf00aca411047c P 78c2c46c7520492db35d51c244fb4f74: No bootstrap required, opened a new log
I20260130 02:58:12.057371 20490 raft_consensus.cc:515] T 3b02480d9c6f4328af83f7940716d698 P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:12.057755 19840 ts_tablet_manager.cc:1403] T 3389db58493448aea3cf00aca411047c P 78c2c46c7520492db35d51c244fb4f74: Time spent bootstrapping tablet: real 0.150s user 0.014s sys 0.005s
I20260130 02:58:12.065052 19625 tablet_bootstrap.cc:654] T dc09a0f0cecf49738bab04596944c0f3 P d870164263bb44b6b780a557daea3656: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:12.069154 19840 raft_consensus.cc:359] T 3389db58493448aea3cf00aca411047c P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:12.069764 19840 raft_consensus.cc:385] T 3389db58493448aea3cf00aca411047c P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:12.070019 19840 raft_consensus.cc:740] T 3389db58493448aea3cf00aca411047c P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78c2c46c7520492db35d51c244fb4f74, State: Initialized, Role: FOLLOWER
I20260130 02:58:12.079483 19840 consensus_queue.cc:260] T 3389db58493448aea3cf00aca411047c P 78c2c46c7520492db35d51c244fb4f74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:12.086478 20504 tablet.cc:1722] T 54be22352e404ff9b798a075693f7f21 P 371a36d219164974abdf848841528152: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260130 02:58:12.090830 18870 catalog_manager.cc:5746] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021) does not have the latest schema for tablet 341041cb0be2456e88403367c0814704 (table default.Table_1703c69112744c0e951c1ac9667bbac6 [id=8d76f4328edc491291491c78ba205d30]). Expected version 1 got 0
I20260130 02:58:12.091497 19840 ts_tablet_manager.cc:1434] T 3389db58493448aea3cf00aca411047c P 78c2c46c7520492db35d51c244fb4f74: Time spent starting tablet: real 0.033s user 0.005s sys 0.000s
I20260130 02:58:12.101150 20490 leader_election.cc:290] T 3b02480d9c6f4328af83f7940716d698 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 election: Requested vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767)
I20260130 02:58:12.102566 19223 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3b02480d9c6f4328af83f7940716d698" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152"
I20260130 02:58:12.103102 19223 raft_consensus.cc:3060] T 3b02480d9c6f4328af83f7940716d698 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:12.103506 20414 ts_tablet_manager.cc:1929] T 7d96e93d95f14af79cfb268626559b1e P 78c2c46c7520492db35d51c244fb4f74: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260130 02:58:12.103874 20414 log.cc:1199] T 7d96e93d95f14af79cfb268626559b1e P 78c2c46c7520492db35d51c244fb4f74: Deleting WAL directory at /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-1/wal/wals/7d96e93d95f14af79cfb268626559b1e
I20260130 02:58:12.104538 20414 ts_tablet_manager.cc:1950] T 7d96e93d95f14af79cfb268626559b1e P 78c2c46c7520492db35d51c244fb4f74: Deleting consensus metadata
I20260130 02:58:12.108578 18870 catalog_manager.cc:5746] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021) does not have the latest schema for tablet ee7c8ff92ca9465fa18be6eae63d78ca (table default.Table_dd22e9dc63f84d8688c0071f427f0b25 [id=77979d498aa040828de415129cb15a06]). Expected version 1 got 0
I20260130 02:58:12.109429 18870 catalog_manager.cc:5654] T 3ae5472cba1e4613abe4cda42ccb2496 P 371a36d219164974abdf848841528152 reported cstate change: term changed from 0 to 1, leader changed from <none> to 371a36d219164974abdf848841528152 (127.18.100.65). New cstate: current_term: 1 leader_uuid: "371a36d219164974abdf848841528152" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } health_report { overall_health: UNKNOWN } } }
I20260130 02:58:12.110261 19223 raft_consensus.cc:2468] T 3b02480d9c6f4328af83f7940716d698 P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d870164263bb44b6b780a557daea3656 in term 1.
I20260130 02:58:12.111584 19463 leader_election.cc:304] T 3b02480d9c6f4328af83f7940716d698 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 371a36d219164974abdf848841528152, d870164263bb44b6b780a557daea3656; no voters:
I20260130 02:58:12.112603 20490 raft_consensus.cc:2804] T 3b02480d9c6f4328af83f7940716d698 P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Leader election won for term 1
I20260130 02:58:12.113318 19807 tablet_bootstrap.cc:654] T 9fe0448bcc4a4346b3c67fc3f626ed3c P 371a36d219164974abdf848841528152: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:12.113677 20490 raft_consensus.cc:697] T 3b02480d9c6f4328af83f7940716d698 P d870164263bb44b6b780a557daea3656 [term 1 LEADER]: Becoming Leader. State: Replica: d870164263bb44b6b780a557daea3656, State: Running, Role: LEADER
I20260130 02:58:12.114609 20490 consensus_queue.cc:237] T 3b02480d9c6f4328af83f7940716d698 P d870164263bb44b6b780a557daea3656 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:12.116289 20505 tablet.cc:1722] T 54be22352e404ff9b798a075693f7f21 P 78c2c46c7520492db35d51c244fb4f74: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260130 02:58:12.154798 19625 tablet_bootstrap.cc:492] T dc09a0f0cecf49738bab04596944c0f3 P d870164263bb44b6b780a557daea3656: No bootstrap required, opened a new log
I20260130 02:58:12.123333 19376 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3b02480d9c6f4328af83f7940716d698" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74"
I20260130 02:58:12.155207 19625 ts_tablet_manager.cc:1403] T dc09a0f0cecf49738bab04596944c0f3 P d870164263bb44b6b780a557daea3656: Time spent bootstrapping tablet: real 0.107s user 0.015s sys 0.005s
W20260130 02:58:12.156059 19462 leader_election.cc:343] T 3b02480d9c6f4328af83f7940716d698 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:12.124481 19840 tablet_bootstrap.cc:492] T 4907294442c145c6b03eb1de057c66a8 P 78c2c46c7520492db35d51c244fb4f74: Bootstrap starting.
I20260130 02:58:12.157421 19625 raft_consensus.cc:359] T dc09a0f0cecf49738bab04596944c0f3 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:12.158032 19625 raft_consensus.cc:385] T dc09a0f0cecf49738bab04596944c0f3 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:12.158277 19625 raft_consensus.cc:740] T dc09a0f0cecf49738bab04596944c0f3 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d870164263bb44b6b780a557daea3656, State: Initialized, Role: FOLLOWER
I20260130 02:58:12.161255 19625 consensus_queue.cc:260] T dc09a0f0cecf49738bab04596944c0f3 P d870164263bb44b6b780a557daea3656 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:12.162645 19840 tablet_bootstrap.cc:654] T 4907294442c145c6b03eb1de057c66a8 P 78c2c46c7520492db35d51c244fb4f74: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:12.166227 19625 ts_tablet_manager.cc:1434] T dc09a0f0cecf49738bab04596944c0f3 P d870164263bb44b6b780a557daea3656: Time spent starting tablet: real 0.011s user 0.005s sys 0.000s
I20260130 02:58:12.167147 19625 tablet_bootstrap.cc:492] T 753344737b5941d9a93b6779e72f63d4 P d870164263bb44b6b780a557daea3656: Bootstrap starting.
I20260130 02:58:12.170607 18870 catalog_manager.cc:5746] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021) does not have the latest schema for tablet 54be22352e404ff9b798a075693f7f21 (table default.Table_3a858a3cb886427cb8268f42c953ad4c [id=17150cb73e374ff9a1e92c43e0c3f2c1]). Expected version 1 got 0
W20260130 02:58:12.104516 18860 catalog_manager.cc:4952] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767): delete failed for tablet 5ca9919f80954e74a4d91c94b74de0e4 because the tablet was not found. No further retry: Not found: Tablet not found: 5ca9919f80954e74a4d91c94b74de0e4
W20260130 02:58:12.171270 18860 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.18.100.66:40767, user_credentials={real_user=slave}} blocked reactor thread for 66789.9us
I20260130 02:58:12.179214 18870 catalog_manager.cc:5654] T 127a84346b9f473d9e7f739b1e65feb8 P 371a36d219164974abdf848841528152 reported cstate change: term changed from 0 to 1, leader changed from <none> to 371a36d219164974abdf848841528152 (127.18.100.65). New cstate: current_term: 1 leader_uuid: "371a36d219164974abdf848841528152" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } health_report { overall_health: UNKNOWN } } }
I20260130 02:58:12.180500 18870 catalog_manager.cc:5746] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021) does not have the latest schema for tablet 127a84346b9f473d9e7f739b1e65feb8 (table default.Table_4f2d6e4d9c8e465087d65efc9c2de442 [id=1515db389f284e2f9ed1a3cd0061d0f1]). Expected version 5 got 0
I20260130 02:58:12.184341 20497 raft_consensus.cc:493] T a8a5038e9def44eb8b3b1e4aeae25652 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:12.184765 20497 raft_consensus.cc:515] T a8a5038e9def44eb8b3b1e4aeae25652 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:12.166452 20508 tablet.cc:1722] T 54be22352e404ff9b798a075693f7f21 P d870164263bb44b6b780a557daea3656: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260130 02:58:12.186883 20497 leader_election.cc:290] T a8a5038e9def44eb8b3b1e4aeae25652 P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767), d870164263bb44b6b780a557daea3656 (127.18.100.67:39571)
W20260130 02:58:12.189565 19159 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.100.66:40767, user_credentials={real_user=slave}} blocked reactor thread for 46725.4us
I20260130 02:58:12.169318 20414 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60110 (request call id 228) took 1582 ms. Trace:
I20260130 02:58:12.203290 19625 tablet_bootstrap.cc:654] T 753344737b5941d9a93b6779e72f63d4 P d870164263bb44b6b780a557daea3656: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:12.202951 20414 rpcz_store.cc:276] 0130 02:58:10.587205 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:58:10.587374 (+ 169us) service_pool.cc:225] Handling call
0130 02:58:11.993159 (+1405785us) ts_tablet_manager.cc:1151] Deleting tablet 7d96e93d95f14af79cfb268626559b1e
0130 02:58:11.994296 (+ 1137us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0130 02:58:12.040956 (+ 46660us) tablet_metadata.cc:641] Metadata flushed
0130 02:58:12.103270 (+ 62314us) tablet_metadata.cc:641] Metadata flushed
0130 02:58:12.169247 (+ 65977us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1404145}
I20260130 02:58:12.204741 20414 tablet_replica.cc:333] T 7b80b22bd44c4233837c951aa97e6996 P 78c2c46c7520492db35d51c244fb4f74: stopping tablet replica
I20260130 02:58:12.205642 20414 raft_consensus.cc:2243] T 7b80b22bd44c4233837c951aa97e6996 P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260130 02:58:12.209328 19526 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a8a5038e9def44eb8b3b1e4aeae25652" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
I20260130 02:58:12.209842 19526 raft_consensus.cc:2468] T a8a5038e9def44eb8b3b1e4aeae25652 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 371a36d219164974abdf848841528152 in term 0.
I20260130 02:58:12.211609 19523 raft_consensus.cc:3060] T 3ae5472cba1e4613abe4cda42ccb2496 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:12.225992 19807 tablet_bootstrap.cc:492] T 9fe0448bcc4a4346b3c67fc3f626ed3c P 371a36d219164974abdf848841528152: No bootstrap required, opened a new log
I20260130 02:58:12.226508 19807 ts_tablet_manager.cc:1403] T 9fe0448bcc4a4346b3c67fc3f626ed3c P 371a36d219164974abdf848841528152: Time spent bootstrapping tablet: real 0.206s user 0.008s sys 0.002s
I20260130 02:58:12.207176 19376 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a8a5038e9def44eb8b3b1e4aeae25652" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74" is_pre_election: true
I20260130 02:58:12.228662 20479 raft_consensus.cc:493] T 9fe0448bcc4a4346b3c67fc3f626ed3c P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:12.231639 19376 raft_consensus.cc:2468] T a8a5038e9def44eb8b3b1e4aeae25652 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 371a36d219164974abdf848841528152 in term 0.
I20260130 02:58:12.231592 20479 raft_consensus.cc:515] T 9fe0448bcc4a4346b3c67fc3f626ed3c P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:12.233033 20414 raft_consensus.cc:2272] T 7b80b22bd44c4233837c951aa97e6996 P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260130 02:58:12.233551 20479 leader_election.cc:290] T 9fe0448bcc4a4346b3c67fc3f626ed3c P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), d870164263bb44b6b780a557daea3656 (127.18.100.67:39571)
I20260130 02:58:12.216279 18860 catalog_manager.cc:4985] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767): tablet 7d96e93d95f14af79cfb268626559b1e (table default.Table_6eb0a92b1926487cbf0b978d26d6a7b1 [id=de7a420d40864f55b156e149308ba12f]) successfully deleted
I20260130 02:58:12.235806 19223 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9fe0448bcc4a4346b3c67fc3f626ed3c" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152" is_pre_election: true
I20260130 02:58:12.236763 19521 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9fe0448bcc4a4346b3c67fc3f626ed3c" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
W20260130 02:58:12.238018 19313 leader_election.cc:343] T 9fe0448bcc4a4346b3c67fc3f626ed3c P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:12.239871 19159 leader_election.cc:304] T a8a5038e9def44eb8b3b1e4aeae25652 P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 371a36d219164974abdf848841528152, 78c2c46c7520492db35d51c244fb4f74; no voters:
W20260130 02:58:12.240684 19159 consensus_peers.cc:597] T 7d96e93d95f14af79cfb268626559b1e P 371a36d219164974abdf848841528152 -> Peer 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767): Couldn't send request to peer 78c2c46c7520492db35d51c244fb4f74. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7d96e93d95f14af79cfb268626559b1e. This is attempt 1: this message will repeat every 5th retry.
I20260130 02:58:12.241389 20498 raft_consensus.cc:2804] T a8a5038e9def44eb8b3b1e4aeae25652 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260130 02:58:12.241720 20498 raft_consensus.cc:493] T a8a5038e9def44eb8b3b1e4aeae25652 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260130 02:58:12.242015 20498 raft_consensus.cc:3060] T a8a5038e9def44eb8b3b1e4aeae25652 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:12.244812 19840 tablet_bootstrap.cc:492] T 4907294442c145c6b03eb1de057c66a8 P 78c2c46c7520492db35d51c244fb4f74: No bootstrap required, opened a new log
I20260130 02:58:12.245226 19840 ts_tablet_manager.cc:1403] T 4907294442c145c6b03eb1de057c66a8 P 78c2c46c7520492db35d51c244fb4f74: Time spent bootstrapping tablet: real 0.121s user 0.008s sys 0.000s
I20260130 02:58:12.248482 19840 raft_consensus.cc:359] T 4907294442c145c6b03eb1de057c66a8 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:12.249111 19840 raft_consensus.cc:385] T 4907294442c145c6b03eb1de057c66a8 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:12.249356 19840 raft_consensus.cc:740] T 4907294442c145c6b03eb1de057c66a8 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78c2c46c7520492db35d51c244fb4f74, State: Initialized, Role: FOLLOWER
I20260130 02:58:12.250556 20498 raft_consensus.cc:515] T a8a5038e9def44eb8b3b1e4aeae25652 P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:12.251976 19840 consensus_queue.cc:260] T 4907294442c145c6b03eb1de057c66a8 P 78c2c46c7520492db35d51c244fb4f74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:12.253655 19376 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a8a5038e9def44eb8b3b1e4aeae25652" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74"
I20260130 02:58:12.252388 20498 leader_election.cc:290] T a8a5038e9def44eb8b3b1e4aeae25652 P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 election: Requested vote from peers 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767), d870164263bb44b6b780a557daea3656 (127.18.100.67:39571)
I20260130 02:58:12.254179 19376 raft_consensus.cc:3060] T a8a5038e9def44eb8b3b1e4aeae25652 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:12.260926 19376 raft_consensus.cc:2468] T a8a5038e9def44eb8b3b1e4aeae25652 P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 371a36d219164974abdf848841528152 in term 1.
I20260130 02:58:12.262310 19159 leader_election.cc:304] T a8a5038e9def44eb8b3b1e4aeae25652 P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 371a36d219164974abdf848841528152, 78c2c46c7520492db35d51c244fb4f74; no voters:
W20260130 02:58:12.263923 18861 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet 49efcfced0b74c48962393149ac2a500 (table default.Table_2e81f0a0cbc449a397141b717db17025 [id=a9c919f57c0c437cb35a6921103e45a6], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 49efcfced0b74c48962393149ac2a500 consensus state has no leader
I20260130 02:58:12.264526 20498 raft_consensus.cc:2804] T a8a5038e9def44eb8b3b1e4aeae25652 P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Leader election won for term 1
I20260130 02:58:12.265429 20498 raft_consensus.cc:697] T a8a5038e9def44eb8b3b1e4aeae25652 P 371a36d219164974abdf848841528152 [term 1 LEADER]: Becoming Leader. State: Replica: 371a36d219164974abdf848841528152, State: Running, Role: LEADER
I20260130 02:58:12.265738 19840 ts_tablet_manager.cc:1434] T 4907294442c145c6b03eb1de057c66a8 P 78c2c46c7520492db35d51c244fb4f74: Time spent starting tablet: real 0.020s user 0.007s sys 0.000s
I20260130 02:58:12.266731 19523 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a8a5038e9def44eb8b3b1e4aeae25652" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656"
I20260130 02:58:12.267279 19523 raft_consensus.cc:3060] T a8a5038e9def44eb8b3b1e4aeae25652 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:12.266688 20414 ts_tablet_manager.cc:1916] T 7b80b22bd44c4233837c951aa97e6996 P 78c2c46c7520492db35d51c244fb4f74: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260130 02:58:12.273177 19840 tablet_bootstrap.cc:492] T 3b02480d9c6f4328af83f7940716d698 P 78c2c46c7520492db35d51c244fb4f74: Bootstrap starting.
I20260130 02:58:12.274798 20498 consensus_queue.cc:237] T a8a5038e9def44eb8b3b1e4aeae25652 P 371a36d219164974abdf848841528152 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:12.290545 19523 raft_consensus.cc:2468] T a8a5038e9def44eb8b3b1e4aeae25652 P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 371a36d219164974abdf848841528152 in term 1.
W20260130 02:58:02.728628 19999 debug-util.cc:398] Leaking SignalData structure 0x7b080002ec60 after lost signal to thread 19872
W20260130 02:58:12.288049 19872 thread.cc:641] GCE (cloud detector) Time spent creating pthread: real 10.887s user 0.327s sys 0.538s
W20260130 02:58:12.292846 19872 thread.cc:608] GCE (cloud detector) Time spent starting thread: real 10.892s user 0.328s sys 0.538s
W20260130 02:58:01.532364 20002 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260130 02:58:12.312330 18868 catalog_manager.cc:5746] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767) does not have the latest schema for tablet a8a5038e9def44eb8b3b1e4aeae25652 (table default.Table_dee034aec0b24982b8dc2a3fa1b326f1 [id=66033e75b3814a8584311cf7e4a07ad3]). Expected version 1 got 0
I20260130 02:58:12.317173 19872 server_base.cc:1060] running on GCE node
I20260130 02:58:12.318704 19840 tablet_bootstrap.cc:654] T 3b02480d9c6f4328af83f7940716d698 P 78c2c46c7520492db35d51c244fb4f74: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:12.228838 19807 raft_consensus.cc:359] T 9fe0448bcc4a4346b3c67fc3f626ed3c P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:12.318665 18868 catalog_manager.cc:5654] T 8e57ae31ab3941f789ac69627384e3b0 P 78c2c46c7520492db35d51c244fb4f74 reported cstate change: term changed from 0 to 1, leader changed from <none> to 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66). New cstate: current_term: 1 leader_uuid: "78c2c46c7520492db35d51c244fb4f74" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } health_report { overall_health: UNKNOWN } } }
I20260130 02:58:12.320315 19807 raft_consensus.cc:385] T 9fe0448bcc4a4346b3c67fc3f626ed3c P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:12.320586 19807 raft_consensus.cc:740] T 9fe0448bcc4a4346b3c67fc3f626ed3c P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 371a36d219164974abdf848841528152, State: Initialized, Role: FOLLOWER
W20260130 02:58:12.323865 20517 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260130 02:58:12.325656 19872 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260130 02:58:12.313021 19625 tablet_bootstrap.cc:492] T 753344737b5941d9a93b6779e72f63d4 P d870164263bb44b6b780a557daea3656: No bootstrap required, opened a new log
I20260130 02:58:12.326901 19625 ts_tablet_manager.cc:1403] T 753344737b5941d9a93b6779e72f63d4 P d870164263bb44b6b780a557daea3656: Time spent bootstrapping tablet: real 0.160s user 0.010s sys 0.000s
I20260130 02:58:12.327205 18868 catalog_manager.cc:5654] T 43cec19c51ec4f05b63800c07f9534ef P 78c2c46c7520492db35d51c244fb4f74 reported cstate change: term changed from 0 to 1, leader changed from <none> to 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66). New cstate: current_term: 1 leader_uuid: "78c2c46c7520492db35d51c244fb4f74" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } health_report { overall_health: UNKNOWN } } }
I20260130 02:58:12.329172 19625 raft_consensus.cc:359] T 753344737b5941d9a93b6779e72f63d4 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:12.329851 19625 raft_consensus.cc:385] T 753344737b5941d9a93b6779e72f63d4 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:12.330111 19625 raft_consensus.cc:740] T 753344737b5941d9a93b6779e72f63d4 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d870164263bb44b6b780a557daea3656, State: Initialized, Role: FOLLOWER
I20260130 02:58:12.330765 20414 ts_tablet_manager.cc:1929] T 7b80b22bd44c4233837c951aa97e6996 P 78c2c46c7520492db35d51c244fb4f74: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260130 02:58:12.332540 18868 catalog_manager.cc:5654] T d55291d7cea64a0d9a345ce712fbfa80 P 78c2c46c7520492db35d51c244fb4f74 reported cstate change: term changed from 0 to 1, leader changed from <none> to 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66). New cstate: current_term: 1 leader_uuid: "78c2c46c7520492db35d51c244fb4f74" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } health_report { overall_health: UNKNOWN } } }
I20260130 02:58:12.333611 20414 log.cc:1199] T 7b80b22bd44c4233837c951aa97e6996 P 78c2c46c7520492db35d51c244fb4f74: Deleting WAL directory at /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-1/wal/wals/7b80b22bd44c4233837c951aa97e6996
I20260130 02:58:12.334400 20414 ts_tablet_manager.cc:1950] T 7b80b22bd44c4233837c951aa97e6996 P 78c2c46c7520492db35d51c244fb4f74: Deleting consensus metadata
I20260130 02:58:12.337893 18868 catalog_manager.cc:5746] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767) does not have the latest schema for tablet 54be22352e404ff9b798a075693f7f21 (table default.Table_3a858a3cb886427cb8268f42c953ad4c [id=17150cb73e374ff9a1e92c43e0c3f2c1]). Expected version 1 got 0
I20260130 02:58:12.339025 18860 catalog_manager.cc:4985] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767): tablet 7b80b22bd44c4233837c951aa97e6996 (table default.Table_7d1c27ab3515406ba5b5333c90df093e [id=b6fabeeae79941dc8094d0f2b51704f9]) successfully deleted
I20260130 02:58:12.341579 18868 catalog_manager.cc:5746] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767) does not have the latest schema for tablet 16691c9315e94253bb9b3fb5058e5636 (table default.Table_8f311c5878744ef692b496ecfdfc3ede [id=4f499357303d464aafc17679ffcaaa07]). Expected version 1 got 0
I20260130 02:58:12.342156 18868 catalog_manager.cc:5746] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767) does not have the latest schema for tablet 127a84346b9f473d9e7f739b1e65feb8 (table default.Table_4f2d6e4d9c8e465087d65efc9c2de442 [id=1515db389f284e2f9ed1a3cd0061d0f1]). Expected version 5 got 0
I20260130 02:58:12.344368 18868 catalog_manager.cc:5746] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767) does not have the latest schema for tablet ee7c8ff92ca9465fa18be6eae63d78ca (table default.Table_dd22e9dc63f84d8688c0071f427f0b25 [id=77979d498aa040828de415129cb15a06]). Expected version 1 got 0
I20260130 02:58:12.348068 19807 consensus_queue.cc:260] T 9fe0448bcc4a4346b3c67fc3f626ed3c P 371a36d219164974abdf848841528152 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:12.348879 19223 raft_consensus.cc:2468] T 9fe0448bcc4a4346b3c67fc3f626ed3c P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78c2c46c7520492db35d51c244fb4f74 in term 0.
I20260130 02:58:12.353236 19807 ts_tablet_manager.cc:1434] T 9fe0448bcc4a4346b3c67fc3f626ed3c P 371a36d219164974abdf848841528152: Time spent starting tablet: real 0.126s user 0.004s sys 0.000s
I20260130 02:58:12.353250 19314 leader_election.cc:304] T 9fe0448bcc4a4346b3c67fc3f626ed3c P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 371a36d219164974abdf848841528152, 78c2c46c7520492db35d51c244fb4f74; no voters: d870164263bb44b6b780a557daea3656
I20260130 02:58:12.355226 20479 raft_consensus.cc:2804] T 9fe0448bcc4a4346b3c67fc3f626ed3c P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260130 02:58:12.355541 20479 raft_consensus.cc:493] T 9fe0448bcc4a4346b3c67fc3f626ed3c P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260130 02:58:12.355822 20479 raft_consensus.cc:3060] T 9fe0448bcc4a4346b3c67fc3f626ed3c P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:12.358640 19807 tablet_bootstrap.cc:492] T 4907294442c145c6b03eb1de057c66a8 P 371a36d219164974abdf848841528152: Bootstrap starting.
I20260130 02:58:12.359354 19625 consensus_queue.cc:260] T 753344737b5941d9a93b6779e72f63d4 P d870164263bb44b6b780a557daea3656 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:12.372529 19807 tablet_bootstrap.cc:654] T 4907294442c145c6b03eb1de057c66a8 P 371a36d219164974abdf848841528152: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:12.377127 19625 ts_tablet_manager.cc:1434] T 753344737b5941d9a93b6779e72f63d4 P d870164263bb44b6b780a557daea3656: Time spent starting tablet: real 0.050s user 0.005s sys 0.000s
I20260130 02:58:12.377734 19872 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260130 02:58:12.379866 20498 raft_consensus.cc:493] T dc09a0f0cecf49738bab04596944c0f3 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:12.380306 20498 raft_consensus.cc:515] T dc09a0f0cecf49738bab04596944c0f3 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:12.382817 20498 leader_election.cc:290] T dc09a0f0cecf49738bab04596944c0f3 P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767), d870164263bb44b6b780a557daea3656 (127.18.100.67:39571)
I20260130 02:58:12.383038 19376 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc09a0f0cecf49738bab04596944c0f3" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74" is_pre_election: true
I20260130 02:58:12.383571 19376 raft_consensus.cc:2468] T dc09a0f0cecf49738bab04596944c0f3 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 371a36d219164974abdf848841528152 in term 0.
I20260130 02:58:12.384876 19159 leader_election.cc:304] T dc09a0f0cecf49738bab04596944c0f3 P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 371a36d219164974abdf848841528152, 78c2c46c7520492db35d51c244fb4f74; no voters:
I20260130 02:58:12.385308 19523 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc09a0f0cecf49738bab04596944c0f3" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
I20260130 02:58:12.385843 19523 raft_consensus.cc:2468] T dc09a0f0cecf49738bab04596944c0f3 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 371a36d219164974abdf848841528152 in term 0.
I20260130 02:58:12.387835 20498 raft_consensus.cc:2804] T dc09a0f0cecf49738bab04596944c0f3 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260130 02:58:12.388219 20498 raft_consensus.cc:493] T dc09a0f0cecf49738bab04596944c0f3 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260130 02:58:12.388574 20498 raft_consensus.cc:3060] T dc09a0f0cecf49738bab04596944c0f3 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:12.399468 20498 raft_consensus.cc:515] T dc09a0f0cecf49738bab04596944c0f3 P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:12.388020 19840 tablet_bootstrap.cc:492] T 3b02480d9c6f4328af83f7940716d698 P 78c2c46c7520492db35d51c244fb4f74: No bootstrap required, opened a new log
I20260130 02:58:12.400799 19840 ts_tablet_manager.cc:1403] T 3b02480d9c6f4328af83f7940716d698 P 78c2c46c7520492db35d51c244fb4f74: Time spent bootstrapping tablet: real 0.128s user 0.003s sys 0.023s
I20260130 02:58:12.410451 19625 tablet_bootstrap.cc:492] T 3389db58493448aea3cf00aca411047c P d870164263bb44b6b780a557daea3656: Bootstrap starting.
I20260130 02:58:12.412459 19840 raft_consensus.cc:359] T 3b02480d9c6f4328af83f7940716d698 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:12.416860 19840 raft_consensus.cc:385] T 3b02480d9c6f4328af83f7940716d698 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:12.417232 19840 raft_consensus.cc:740] T 3b02480d9c6f4328af83f7940716d698 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78c2c46c7520492db35d51c244fb4f74, State: Initialized, Role: FOLLOWER
I20260130 02:58:12.417951 19840 consensus_queue.cc:260] T 3b02480d9c6f4328af83f7940716d698 P 78c2c46c7520492db35d51c244fb4f74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:12.418838 19872 hybrid_clock.cc:648] HybridClock initialized: now 1769741892418697 us; error 121 us; skew 500 ppm
I20260130 02:58:12.420060 19872 server_base.cc:860] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260130 02:58:12.421454 19840 ts_tablet_manager.cc:1434] T 3b02480d9c6f4328af83f7940716d698 P 78c2c46c7520492db35d51c244fb4f74: Time spent starting tablet: real 0.020s user 0.003s sys 0.001s
I20260130 02:58:12.386385 20479 raft_consensus.cc:515] T 9fe0448bcc4a4346b3c67fc3f626ed3c P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:12.425868 20479 leader_election.cc:290] T 9fe0448bcc4a4346b3c67fc3f626ed3c P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 election: Requested vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), d870164263bb44b6b780a557daea3656 (127.18.100.67:39571)
I20260130 02:58:12.427778 19523 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9fe0448bcc4a4346b3c67fc3f626ed3c" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656"
W20260130 02:58:12.430099 19313 leader_election.cc:343] T 9fe0448bcc4a4346b3c67fc3f626ed3c P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:12.431197 19840 tablet_bootstrap.cc:492] T 7caf7cbad5054396b5617094e4d536a0 P 78c2c46c7520492db35d51c244fb4f74: Bootstrap starting.
I20260130 02:58:12.436214 19625 tablet_bootstrap.cc:654] T 3389db58493448aea3cf00aca411047c P d870164263bb44b6b780a557daea3656: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:12.446358 19840 tablet_bootstrap.cc:654] T 7caf7cbad5054396b5617094e4d536a0 P 78c2c46c7520492db35d51c244fb4f74: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:12.423836 19223 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9fe0448bcc4a4346b3c67fc3f626ed3c" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152"
I20260130 02:58:12.452631 19223 raft_consensus.cc:3060] T 9fe0448bcc4a4346b3c67fc3f626ed3c P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:12.459280 19223 raft_consensus.cc:2468] T 9fe0448bcc4a4346b3c67fc3f626ed3c P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78c2c46c7520492db35d51c244fb4f74 in term 1.
I20260130 02:58:12.443985 19807 tablet_bootstrap.cc:492] T 4907294442c145c6b03eb1de057c66a8 P 371a36d219164974abdf848841528152: No bootstrap required, opened a new log
I20260130 02:58:12.460341 19807 ts_tablet_manager.cc:1403] T 4907294442c145c6b03eb1de057c66a8 P 371a36d219164974abdf848841528152: Time spent bootstrapping tablet: real 0.102s user 0.013s sys 0.007s
I20260130 02:58:12.461905 19525 raft_consensus.cc:3060] T d27ca7dd87f44102b7680e1f4a88954a P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:12.463023 19807 raft_consensus.cc:359] T 4907294442c145c6b03eb1de057c66a8 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:12.463816 19807 raft_consensus.cc:385] T 4907294442c145c6b03eb1de057c66a8 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:12.464103 19807 raft_consensus.cc:740] T 4907294442c145c6b03eb1de057c66a8 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 371a36d219164974abdf848841528152, State: Initialized, Role: FOLLOWER
I20260130 02:58:12.464771 19807 consensus_queue.cc:260] T 4907294442c145c6b03eb1de057c66a8 P 371a36d219164974abdf848841528152 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:12.466074 20490 raft_consensus.cc:493] T 753344737b5941d9a93b6779e72f63d4 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:12.466475 19314 leader_election.cc:304] T 9fe0448bcc4a4346b3c67fc3f626ed3c P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 371a36d219164974abdf848841528152, 78c2c46c7520492db35d51c244fb4f74; no voters: d870164263bb44b6b780a557daea3656
I20260130 02:58:12.466507 20490 raft_consensus.cc:515] T 753344737b5941d9a93b6779e72f63d4 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:12.467720 20479 raft_consensus.cc:2804] T 9fe0448bcc4a4346b3c67fc3f626ed3c P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Leader election won for term 1
I20260130 02:58:12.468286 20490 leader_election.cc:290] T 753344737b5941d9a93b6779e72f63d4 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767)
I20260130 02:58:12.476550 19807 ts_tablet_manager.cc:1434] T 4907294442c145c6b03eb1de057c66a8 P 371a36d219164974abdf848841528152: Time spent starting tablet: real 0.016s user 0.001s sys 0.004s
I20260130 02:58:12.477319 20479 raft_consensus.cc:697] T 9fe0448bcc4a4346b3c67fc3f626ed3c P 78c2c46c7520492db35d51c244fb4f74 [term 1 LEADER]: Becoming Leader. State: Replica: 78c2c46c7520492db35d51c244fb4f74, State: Running, Role: LEADER
I20260130 02:58:12.478349 20479 consensus_queue.cc:237] T 9fe0448bcc4a4346b3c67fc3f626ed3c P 78c2c46c7520492db35d51c244fb4f74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:12.478325 20462 consensus_queue.cc:1048] T d27ca7dd87f44102b7680e1f4a88954a P 371a36d219164974abdf848841528152 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260130 02:58:12.485226 19223 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "753344737b5941d9a93b6779e72f63d4" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152" is_pre_election: true
I20260130 02:58:12.487186 19223 raft_consensus.cc:2393] T 753344737b5941d9a93b6779e72f63d4 P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d870164263bb44b6b780a557daea3656 in current term 1: Already voted for candidate 78c2c46c7520492db35d51c244fb4f74 in this term.
I20260130 02:58:12.487685 20490 raft_consensus.cc:493] T d27ca7dd87f44102b7680e1f4a88954a P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:12.481161 19376 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "753344737b5941d9a93b6779e72f63d4" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74" is_pre_election: true
I20260130 02:58:12.488072 20490 raft_consensus.cc:515] T d27ca7dd87f44102b7680e1f4a88954a P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:12.492204 20490 leader_election.cc:290] T d27ca7dd87f44102b7680e1f4a88954a P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767)
I20260130 02:58:12.486944 20462 consensus_queue.cc:1048] T d27ca7dd87f44102b7680e1f4a88954a P 371a36d219164974abdf848841528152 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260130 02:58:12.490419 19225 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d27ca7dd87f44102b7680e1f4a88954a" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152" is_pre_election: true
I20260130 02:58:12.494812 19463 leader_election.cc:304] T 753344737b5941d9a93b6779e72f63d4 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d870164263bb44b6b780a557daea3656; no voters: 371a36d219164974abdf848841528152, 78c2c46c7520492db35d51c244fb4f74
I20260130 02:58:12.496340 20490 raft_consensus.cc:3060] T 753344737b5941d9a93b6779e72f63d4 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:12.503150 19378 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d27ca7dd87f44102b7680e1f4a88954a" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74" is_pre_election: true
I20260130 02:58:12.506793 19462 leader_election.cc:304] T d27ca7dd87f44102b7680e1f4a88954a P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d870164263bb44b6b780a557daea3656; no voters: 371a36d219164974abdf848841528152, 78c2c46c7520492db35d51c244fb4f74
I20260130 02:58:12.514223 19625 tablet_bootstrap.cc:492] T 3389db58493448aea3cf00aca411047c P d870164263bb44b6b780a557daea3656: No bootstrap required, opened a new log
I20260130 02:58:12.514683 19625 ts_tablet_manager.cc:1403] T 3389db58493448aea3cf00aca411047c P d870164263bb44b6b780a557daea3656: Time spent bootstrapping tablet: real 0.105s user 0.008s sys 0.003s
I20260130 02:58:12.517020 19625 raft_consensus.cc:359] T 3389db58493448aea3cf00aca411047c P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:12.517629 19625 raft_consensus.cc:385] T 3389db58493448aea3cf00aca411047c P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:12.517891 19625 raft_consensus.cc:740] T 3389db58493448aea3cf00aca411047c P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d870164263bb44b6b780a557daea3656, State: Initialized, Role: FOLLOWER
I20260130 02:58:12.518127 20490 raft_consensus.cc:2749] T 753344737b5941d9a93b6779e72f63d4 P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260130 02:58:12.518471 19625 consensus_queue.cc:260] T 3389db58493448aea3cf00aca411047c P d870164263bb44b6b780a557daea3656 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:12.519593 19872 webserver.cc:492] Webserver started at http://127.18.100.124:37741/ using document root <none> and password file <none>
I20260130 02:58:12.520800 19376 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc09a0f0cecf49738bab04596944c0f3" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74"
I20260130 02:58:12.521399 19376 raft_consensus.cc:3060] T dc09a0f0cecf49738bab04596944c0f3 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:12.526227 19872 fs_manager.cc:362] Metadata directory not provided
I20260130 02:58:12.492077 19807 tablet_bootstrap.cc:492] T 3389db58493448aea3cf00aca411047c P 371a36d219164974abdf848841528152: Bootstrap starting.
I20260130 02:58:12.527985 19872 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260130 02:58:12.530172 19625 ts_tablet_manager.cc:1434] T 3389db58493448aea3cf00aca411047c P d870164263bb44b6b780a557daea3656: Time spent starting tablet: real 0.015s user 0.006s sys 0.000s
I20260130 02:58:12.531100 19625 tablet_bootstrap.cc:492] T 345eaaa5625c48d099280b7fbb2e514b P d870164263bb44b6b780a557daea3656: Bootstrap starting.
I20260130 02:58:12.533346 19521 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc09a0f0cecf49738bab04596944c0f3" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656"
I20260130 02:58:12.533866 19521 raft_consensus.cc:3060] T dc09a0f0cecf49738bab04596944c0f3 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:12.538245 19625 tablet_bootstrap.cc:654] T 345eaaa5625c48d099280b7fbb2e514b P d870164263bb44b6b780a557daea3656: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:12.542692 20491 raft_consensus.cc:2749] T d27ca7dd87f44102b7680e1f4a88954a P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260130 02:58:12.563226 20481 raft_consensus.cc:493] T 3389db58493448aea3cf00aca411047c P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:12.564846 20481 raft_consensus.cc:515] T 3389db58493448aea3cf00aca411047c P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:12.565995 19625 tablet_bootstrap.cc:492] T 345eaaa5625c48d099280b7fbb2e514b P d870164263bb44b6b780a557daea3656: No bootstrap required, opened a new log
I20260130 02:58:12.566390 19625 ts_tablet_manager.cc:1403] T 345eaaa5625c48d099280b7fbb2e514b P d870164263bb44b6b780a557daea3656: Time spent bootstrapping tablet: real 0.036s user 0.008s sys 0.001s
I20260130 02:58:12.567978 19223 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3389db58493448aea3cf00aca411047c" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152" is_pre_election: true
W20260130 02:58:12.569202 19314 leader_election.cc:343] T 3389db58493448aea3cf00aca411047c P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:12.570523 20481 leader_election.cc:290] T 3389db58493448aea3cf00aca411047c P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), d870164263bb44b6b780a557daea3656 (127.18.100.67:39571)
I20260130 02:58:12.571198 19525 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3389db58493448aea3cf00aca411047c" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
I20260130 02:58:12.571841 19525 raft_consensus.cc:2468] T 3389db58493448aea3cf00aca411047c P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78c2c46c7520492db35d51c244fb4f74 in term 0.
I20260130 02:58:12.573148 19313 leader_election.cc:304] T 3389db58493448aea3cf00aca411047c P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 78c2c46c7520492db35d51c244fb4f74, d870164263bb44b6b780a557daea3656; no voters: 371a36d219164974abdf848841528152
I20260130 02:58:12.579101 20474 raft_consensus.cc:2804] T 3389db58493448aea3cf00aca411047c P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260130 02:58:12.579452 20474 raft_consensus.cc:493] T 3389db58493448aea3cf00aca411047c P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260130 02:58:12.579768 20474 raft_consensus.cc:3060] T 3389db58493448aea3cf00aca411047c P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:12.580987 19625 raft_consensus.cc:359] T 345eaaa5625c48d099280b7fbb2e514b P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:12.581620 19625 raft_consensus.cc:385] T 345eaaa5625c48d099280b7fbb2e514b P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:12.581866 19625 raft_consensus.cc:740] T 345eaaa5625c48d099280b7fbb2e514b P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d870164263bb44b6b780a557daea3656, State: Initialized, Role: FOLLOWER
I20260130 02:58:12.582407 19625 consensus_queue.cc:260] T 345eaaa5625c48d099280b7fbb2e514b P d870164263bb44b6b780a557daea3656 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:12.584389 19625 ts_tablet_manager.cc:1434] T 345eaaa5625c48d099280b7fbb2e514b P d870164263bb44b6b780a557daea3656: Time spent starting tablet: real 0.018s user 0.004s sys 0.000s
I20260130 02:58:12.585270 19625 tablet_bootstrap.cc:492] T 7caf7cbad5054396b5617094e4d536a0 P d870164263bb44b6b780a557daea3656: Bootstrap starting.
I20260130 02:58:12.588173 20474 raft_consensus.cc:515] T 3389db58493448aea3cf00aca411047c P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:12.590675 19223 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3389db58493448aea3cf00aca411047c" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152"
W20260130 02:58:12.591822 19314 leader_election.cc:343] T 3389db58493448aea3cf00aca411047c P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:12.592734 19376 raft_consensus.cc:2468] T dc09a0f0cecf49738bab04596944c0f3 P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 371a36d219164974abdf848841528152 in term 1.
W20260130 02:58:12.601452 19314 consensus_peers.cc:597] T 0e8018c094024c4788c484e8f5466e00 P 78c2c46c7520492db35d51c244fb4f74 -> Peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Couldn't send request to peer 371a36d219164974abdf848841528152. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260130 02:58:12.602377 19625 tablet_bootstrap.cc:654] T 7caf7cbad5054396b5617094e4d536a0 P d870164263bb44b6b780a557daea3656: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:12.606479 19840 tablet_bootstrap.cc:492] T 7caf7cbad5054396b5617094e4d536a0 P 78c2c46c7520492db35d51c244fb4f74: No bootstrap required, opened a new log
I20260130 02:58:12.606904 19840 ts_tablet_manager.cc:1403] T 7caf7cbad5054396b5617094e4d536a0 P 78c2c46c7520492db35d51c244fb4f74: Time spent bootstrapping tablet: real 0.176s user 0.034s sys 0.018s
I20260130 02:58:12.607123 19159 leader_election.cc:304] T dc09a0f0cecf49738bab04596944c0f3 P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 371a36d219164974abdf848841528152, 78c2c46c7520492db35d51c244fb4f74; no voters:
I20260130 02:58:12.608026 20497 raft_consensus.cc:2804] T dc09a0f0cecf49738bab04596944c0f3 P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Leader election won for term 1
I20260130 02:58:12.608440 20497 raft_consensus.cc:697] T dc09a0f0cecf49738bab04596944c0f3 P 371a36d219164974abdf848841528152 [term 1 LEADER]: Becoming Leader. State: Replica: 371a36d219164974abdf848841528152, State: Running, Role: LEADER
I20260130 02:58:12.609196 19840 raft_consensus.cc:359] T 7caf7cbad5054396b5617094e4d536a0 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:12.609143 20497 consensus_queue.cc:237] T dc09a0f0cecf49738bab04596944c0f3 P 371a36d219164974abdf848841528152 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:12.609784 19840 raft_consensus.cc:385] T 7caf7cbad5054396b5617094e4d536a0 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:12.609999 19840 raft_consensus.cc:740] T 7caf7cbad5054396b5617094e4d536a0 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78c2c46c7520492db35d51c244fb4f74, State: Initialized, Role: FOLLOWER
I20260130 02:58:12.610514 19840 consensus_queue.cc:260] T 7caf7cbad5054396b5617094e4d536a0 P 78c2c46c7520492db35d51c244fb4f74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:12.612522 19840 ts_tablet_manager.cc:1434] T 7caf7cbad5054396b5617094e4d536a0 P 78c2c46c7520492db35d51c244fb4f74: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260130 02:58:12.613350 19840 tablet_bootstrap.cc:492] T ba91a508b1cb4430a4d8ca26946aecc8 P 78c2c46c7520492db35d51c244fb4f74: Bootstrap starting.
I20260130 02:58:12.562764 19521 raft_consensus.cc:2468] T dc09a0f0cecf49738bab04596944c0f3 P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 371a36d219164974abdf848841528152 in term 1.
I20260130 02:58:12.621630 19840 tablet_bootstrap.cc:654] T ba91a508b1cb4430a4d8ca26946aecc8 P 78c2c46c7520492db35d51c244fb4f74: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:12.620404 19872 fs_manager.cc:714] Time spent opening directory manager: real 0.051s user 0.006s sys 0.002s
I20260130 02:58:12.624537 20481 raft_consensus.cc:493] T 4c139aa8cd184f488cec4e05ef7dcada P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:12.624958 20481 raft_consensus.cc:515] T 4c139aa8cd184f488cec4e05ef7dcada P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:12.626902 20481 leader_election.cc:290] T 4c139aa8cd184f488cec4e05ef7dcada P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), d870164263bb44b6b780a557daea3656 (127.18.100.67:39571)
I20260130 02:58:12.627910 19223 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4c139aa8cd184f488cec4e05ef7dcada" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152" is_pre_election: true
I20260130 02:58:12.628202 19523 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4c139aa8cd184f488cec4e05ef7dcada" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
I20260130 02:58:12.628686 19223 raft_consensus.cc:2468] T 4c139aa8cd184f488cec4e05ef7dcada P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78c2c46c7520492db35d51c244fb4f74 in term 0.
W20260130 02:58:12.629498 19313 leader_election.cc:343] T 4c139aa8cd184f488cec4e05ef7dcada P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:12.630421 19314 leader_election.cc:304] T 4c139aa8cd184f488cec4e05ef7dcada P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 371a36d219164974abdf848841528152, 78c2c46c7520492db35d51c244fb4f74; no voters: d870164263bb44b6b780a557daea3656
I20260130 02:58:12.631561 19525 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3389db58493448aea3cf00aca411047c" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656"
I20260130 02:58:12.632242 19525 raft_consensus.cc:3060] T 3389db58493448aea3cf00aca411047c P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:12.638507 19840 tablet_bootstrap.cc:492] T ba91a508b1cb4430a4d8ca26946aecc8 P 78c2c46c7520492db35d51c244fb4f74: No bootstrap required, opened a new log
I20260130 02:58:12.638933 19840 ts_tablet_manager.cc:1403] T ba91a508b1cb4430a4d8ca26946aecc8 P 78c2c46c7520492db35d51c244fb4f74: Time spent bootstrapping tablet: real 0.026s user 0.009s sys 0.000s
I20260130 02:58:12.643693 20474 leader_election.cc:290] T 3389db58493448aea3cf00aca411047c P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 election: Requested vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), d870164263bb44b6b780a557daea3656 (127.18.100.67:39571)
I20260130 02:58:12.668747 19625 tablet_bootstrap.cc:492] T 7caf7cbad5054396b5617094e4d536a0 P d870164263bb44b6b780a557daea3656: No bootstrap required, opened a new log
I20260130 02:58:12.669196 19625 ts_tablet_manager.cc:1403] T 7caf7cbad5054396b5617094e4d536a0 P d870164263bb44b6b780a557daea3656: Time spent bootstrapping tablet: real 0.084s user 0.017s sys 0.001s
I20260130 02:58:12.678846 20474 raft_consensus.cc:2804] T 4c139aa8cd184f488cec4e05ef7dcada P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260130 02:58:12.679366 20474 raft_consensus.cc:493] T 4c139aa8cd184f488cec4e05ef7dcada P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260130 02:58:12.680233 19840 raft_consensus.cc:359] T ba91a508b1cb4430a4d8ca26946aecc8 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:12.680881 19840 raft_consensus.cc:385] T ba91a508b1cb4430a4d8ca26946aecc8 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:12.681133 19840 raft_consensus.cc:740] T ba91a508b1cb4430a4d8ca26946aecc8 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78c2c46c7520492db35d51c244fb4f74, State: Initialized, Role: FOLLOWER
I20260130 02:58:12.680359 20474 raft_consensus.cc:3060] T 4c139aa8cd184f488cec4e05ef7dcada P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:12.681656 19840 consensus_queue.cc:260] T ba91a508b1cb4430a4d8ca26946aecc8 P 78c2c46c7520492db35d51c244fb4f74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:12.684873 19840 ts_tablet_manager.cc:1434] T ba91a508b1cb4430a4d8ca26946aecc8 P 78c2c46c7520492db35d51c244fb4f74: Time spent starting tablet: real 0.046s user 0.005s sys 0.000s
I20260130 02:58:12.685416 20538 consensus_queue.cc:1048] T 753344737b5941d9a93b6779e72f63d4 P 78c2c46c7520492db35d51c244fb4f74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260130 02:58:12.685753 19840 tablet_bootstrap.cc:492] T 345eaaa5625c48d099280b7fbb2e514b P 78c2c46c7520492db35d51c244fb4f74: Bootstrap starting.
I20260130 02:58:12.689332 20538 consensus_queue.cc:1048] T 0e8018c094024c4788c484e8f5466e00 P 78c2c46c7520492db35d51c244fb4f74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260130 02:58:12.671415 19625 raft_consensus.cc:359] T 7caf7cbad5054396b5617094e4d536a0 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:12.691361 19625 raft_consensus.cc:385] T 7caf7cbad5054396b5617094e4d536a0 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:12.691632 19625 raft_consensus.cc:740] T 7caf7cbad5054396b5617094e4d536a0 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d870164263bb44b6b780a557daea3656, State: Initialized, Role: FOLLOWER
I20260130 02:58:12.692230 19625 consensus_queue.cc:260] T 7caf7cbad5054396b5617094e4d536a0 P d870164263bb44b6b780a557daea3656 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:12.693754 19625 ts_tablet_manager.cc:1434] T 7caf7cbad5054396b5617094e4d536a0 P d870164263bb44b6b780a557daea3656: Time spent starting tablet: real 0.024s user 0.004s sys 0.000s
I20260130 02:58:12.699558 19625 tablet_bootstrap.cc:492] T 16691c9315e94253bb9b3fb5058e5636 P d870164263bb44b6b780a557daea3656: Bootstrap starting.
I20260130 02:58:12.700022 20498 leader_election.cc:290] T dc09a0f0cecf49738bab04596944c0f3 P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 election: Requested vote from peers 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767), d870164263bb44b6b780a557daea3656 (127.18.100.67:39571)
I20260130 02:58:12.700915 19840 tablet_bootstrap.cc:654] T 345eaaa5625c48d099280b7fbb2e514b P 78c2c46c7520492db35d51c244fb4f74: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:12.702517 19496 tablet_service.cc:1505] Processing CreateTablet for tablet e34d5da0ad864b2eaea0d575c982e447 (DEFAULT_TABLE table=default.Table_aa4e5fd22945475da1dad48e1508595e [id=5b41834c6d8b4d89b773c2ea4c4ca7e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:12.708636 19496 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e34d5da0ad864b2eaea0d575c982e447. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:12.718616 19840 tablet_bootstrap.cc:492] T 345eaaa5625c48d099280b7fbb2e514b P 78c2c46c7520492db35d51c244fb4f74: No bootstrap required, opened a new log
I20260130 02:58:12.719005 19840 ts_tablet_manager.cc:1403] T 345eaaa5625c48d099280b7fbb2e514b P 78c2c46c7520492db35d51c244fb4f74: Time spent bootstrapping tablet: real 0.033s user 0.011s sys 0.001s
I20260130 02:58:12.718962 19205 tablet_service.cc:1505] Processing CreateTablet for tablet e34d5da0ad864b2eaea0d575c982e447 (DEFAULT_TABLE table=default.Table_aa4e5fd22945475da1dad48e1508595e [id=5b41834c6d8b4d89b773c2ea4c4ca7e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:12.719960 19205 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e34d5da0ad864b2eaea0d575c982e447. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:12.721235 19840 raft_consensus.cc:359] T 345eaaa5625c48d099280b7fbb2e514b P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:12.721804 19840 raft_consensus.cc:385] T 345eaaa5625c48d099280b7fbb2e514b P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:12.722045 19840 raft_consensus.cc:740] T 345eaaa5625c48d099280b7fbb2e514b P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78c2c46c7520492db35d51c244fb4f74, State: Initialized, Role: FOLLOWER
I20260130 02:58:12.722612 19840 consensus_queue.cc:260] T 345eaaa5625c48d099280b7fbb2e514b P 78c2c46c7520492db35d51c244fb4f74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:12.724463 19840 ts_tablet_manager.cc:1434] T 345eaaa5625c48d099280b7fbb2e514b P 78c2c46c7520492db35d51c244fb4f74: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260130 02:58:12.728402 19840 tablet_bootstrap.cc:492] T 5fbe92a2227d44d88b4531a7f8aa0e08 P 78c2c46c7520492db35d51c244fb4f74: Bootstrap starting.
I20260130 02:58:12.730696 20538 raft_consensus.cc:493] T 4907294442c145c6b03eb1de057c66a8 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:12.731191 20538 raft_consensus.cc:515] T 4907294442c145c6b03eb1de057c66a8 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:12.733567 19840 tablet_bootstrap.cc:654] T 5fbe92a2227d44d88b4531a7f8aa0e08 P 78c2c46c7520492db35d51c244fb4f74: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:12.735013 19521 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4907294442c145c6b03eb1de057c66a8" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
W20260130 02:58:12.736649 19313 leader_election.cc:343] T 4907294442c145c6b03eb1de057c66a8 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:12.738297 20538 leader_election.cc:290] T 4907294442c145c6b03eb1de057c66a8 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), d870164263bb44b6b780a557daea3656 (127.18.100.67:39571)
I20260130 02:58:12.738505 19222 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4907294442c145c6b03eb1de057c66a8" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152" is_pre_election: true
I20260130 02:58:12.739132 19222 raft_consensus.cc:2468] T 4907294442c145c6b03eb1de057c66a8 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78c2c46c7520492db35d51c244fb4f74 in term 0.
W20260130 02:58:12.741114 19313 consensus_peers.cc:597] T 16691c9315e94253bb9b3fb5058e5636 P 78c2c46c7520492db35d51c244fb4f74 -> Peer d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): Couldn't send request to peer d870164263bb44b6b780a557daea3656. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20260130 02:58:12.746567 19314 leader_election.cc:304] T 4907294442c145c6b03eb1de057c66a8 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 371a36d219164974abdf848841528152, 78c2c46c7520492db35d51c244fb4f74; no voters: d870164263bb44b6b780a557daea3656
I20260130 02:58:12.747814 20538 raft_consensus.cc:2804] T 4907294442c145c6b03eb1de057c66a8 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260130 02:58:12.748145 20538 raft_consensus.cc:493] T 4907294442c145c6b03eb1de057c66a8 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260130 02:58:12.748445 20538 raft_consensus.cc:3060] T 4907294442c145c6b03eb1de057c66a8 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:12.752178 19840 tablet_bootstrap.cc:492] T 5fbe92a2227d44d88b4531a7f8aa0e08 P 78c2c46c7520492db35d51c244fb4f74: No bootstrap required, opened a new log
I20260130 02:58:12.752570 19840 ts_tablet_manager.cc:1403] T 5fbe92a2227d44d88b4531a7f8aa0e08 P 78c2c46c7520492db35d51c244fb4f74: Time spent bootstrapping tablet: real 0.027s user 0.000s sys 0.008s
I20260130 02:58:12.765794 19840 raft_consensus.cc:359] T 5fbe92a2227d44d88b4531a7f8aa0e08 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:12.766464 19840 raft_consensus.cc:385] T 5fbe92a2227d44d88b4531a7f8aa0e08 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:12.766750 19840 raft_consensus.cc:740] T 5fbe92a2227d44d88b4531a7f8aa0e08 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78c2c46c7520492db35d51c244fb4f74, State: Initialized, Role: FOLLOWER
I20260130 02:58:12.767365 19840 consensus_queue.cc:260] T 5fbe92a2227d44d88b4531a7f8aa0e08 P 78c2c46c7520492db35d51c244fb4f74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:12.769289 19840 ts_tablet_manager.cc:1434] T 5fbe92a2227d44d88b4531a7f8aa0e08 P 78c2c46c7520492db35d51c244fb4f74: Time spent starting tablet: real 0.016s user 0.005s sys 0.001s
I20260130 02:58:12.779044 19350 tablet_service.cc:1505] Processing CreateTablet for tablet e34d5da0ad864b2eaea0d575c982e447 (DEFAULT_TABLE table=default.Table_aa4e5fd22945475da1dad48e1508595e [id=5b41834c6d8b4d89b773c2ea4c4ca7e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:12.779945 19840 tablet_bootstrap.cc:492] T 274530359b174ff79f046c2d4a6afa76 P 78c2c46c7520492db35d51c244fb4f74: Bootstrap starting.
I20260130 02:58:12.780179 19350 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e34d5da0ad864b2eaea0d575c982e447. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:12.785251 19840 tablet_bootstrap.cc:654] T 274530359b174ff79f046c2d4a6afa76 P 78c2c46c7520492db35d51c244fb4f74: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:12.802253 19625 tablet_bootstrap.cc:654] T 16691c9315e94253bb9b3fb5058e5636 P d870164263bb44b6b780a557daea3656: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:12.809350 20479 consensus_queue.cc:1048] T 753344737b5941d9a93b6779e72f63d4 P 78c2c46c7520492db35d51c244fb4f74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260130 02:58:12.810415 19502 tablet_service.cc:1505] Processing CreateTablet for tablet 2b0edaad6877460cb7982efc62b00b66 (DEFAULT_TABLE table=default.Table_68be882f76b74def9608a4147dd883d0 [id=64a8e0e079f34806aa68795f3d548c5e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:12.812218 19198 tablet_service.cc:1505] Processing CreateTablet for tablet 2b0edaad6877460cb7982efc62b00b66 (DEFAULT_TABLE table=default.Table_68be882f76b74def9608a4147dd883d0 [id=64a8e0e079f34806aa68795f3d548c5e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:12.812868 19351 tablet_service.cc:1505] Processing CreateTablet for tablet 2b0edaad6877460cb7982efc62b00b66 (DEFAULT_TABLE table=default.Table_68be882f76b74def9608a4147dd883d0 [id=64a8e0e079f34806aa68795f3d548c5e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:12.813184 19198 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2b0edaad6877460cb7982efc62b00b66. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:12.813870 19351 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2b0edaad6877460cb7982efc62b00b66. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:12.825402 19525 raft_consensus.cc:2468] T 3389db58493448aea3cf00aca411047c P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78c2c46c7520492db35d51c244fb4f74 in term 1.
I20260130 02:58:12.826714 19313 leader_election.cc:304] T 3389db58493448aea3cf00aca411047c P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 78c2c46c7520492db35d51c244fb4f74, d870164263bb44b6b780a557daea3656; no voters: 371a36d219164974abdf848841528152
I20260130 02:58:12.827448 20544 raft_consensus.cc:2804] T 3389db58493448aea3cf00aca411047c P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Leader election won for term 1
I20260130 02:58:12.827986 20544 raft_consensus.cc:697] T 3389db58493448aea3cf00aca411047c P 78c2c46c7520492db35d51c244fb4f74 [term 1 LEADER]: Becoming Leader. State: Replica: 78c2c46c7520492db35d51c244fb4f74, State: Running, Role: LEADER
I20260130 02:58:12.706250 18871 catalog_manager.cc:5746] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021) does not have the latest schema for tablet a8a5038e9def44eb8b3b1e4aeae25652 (table default.Table_dee034aec0b24982b8dc2a3fa1b326f1 [id=66033e75b3814a8584311cf7e4a07ad3]). Expected version 1 got 0
I20260130 02:58:12.828800 20544 consensus_queue.cc:237] T 3389db58493448aea3cf00aca411047c P 78c2c46c7520492db35d51c244fb4f74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:12.841373 19625 tablet_bootstrap.cc:492] T 16691c9315e94253bb9b3fb5058e5636 P d870164263bb44b6b780a557daea3656: No bootstrap required, opened a new log
I20260130 02:58:12.841790 19625 ts_tablet_manager.cc:1403] T 16691c9315e94253bb9b3fb5058e5636 P d870164263bb44b6b780a557daea3656: Time spent bootstrapping tablet: real 0.142s user 0.011s sys 0.001s
I20260130 02:58:12.844204 19625 raft_consensus.cc:359] T 16691c9315e94253bb9b3fb5058e5636 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:12.844796 19625 raft_consensus.cc:385] T 16691c9315e94253bb9b3fb5058e5636 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:12.845453 19625 raft_consensus.cc:740] T 16691c9315e94253bb9b3fb5058e5636 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d870164263bb44b6b780a557daea3656, State: Initialized, Role: FOLLOWER
I20260130 02:58:12.846029 19625 consensus_queue.cc:260] T 16691c9315e94253bb9b3fb5058e5636 P d870164263bb44b6b780a557daea3656 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:12.849890 19625 ts_tablet_manager.cc:1434] T 16691c9315e94253bb9b3fb5058e5636 P d870164263bb44b6b780a557daea3656: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20260130 02:58:12.854740 19502 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2b0edaad6877460cb7982efc62b00b66. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:12.859531 19840 tablet_bootstrap.cc:492] T 274530359b174ff79f046c2d4a6afa76 P 78c2c46c7520492db35d51c244fb4f74: No bootstrap required, opened a new log
I20260130 02:58:12.859572 19625 tablet_bootstrap.cc:492] T 8e57ae31ab3941f789ac69627384e3b0 P d870164263bb44b6b780a557daea3656: Bootstrap starting.
I20260130 02:58:12.859939 19840 ts_tablet_manager.cc:1403] T 274530359b174ff79f046c2d4a6afa76 P 78c2c46c7520492db35d51c244fb4f74: Time spent bootstrapping tablet: real 0.090s user 0.004s sys 0.004s
I20260130 02:58:12.863621 20530 raft_consensus.cc:493] T 8bd3e1c06b89426da57e2b259e7983af P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:12.864033 19840 raft_consensus.cc:359] T 274530359b174ff79f046c2d4a6afa76 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:12.864032 20530 raft_consensus.cc:515] T 8bd3e1c06b89426da57e2b259e7983af P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:12.864615 19840 raft_consensus.cc:385] T 274530359b174ff79f046c2d4a6afa76 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:12.864842 19840 raft_consensus.cc:740] T 274530359b174ff79f046c2d4a6afa76 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78c2c46c7520492db35d51c244fb4f74, State: Initialized, Role: FOLLOWER
I20260130 02:58:12.865483 19840 consensus_queue.cc:260] T 274530359b174ff79f046c2d4a6afa76 P 78c2c46c7520492db35d51c244fb4f74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:12.865801 20530 leader_election.cc:290] T 8bd3e1c06b89426da57e2b259e7983af P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767)
W20260130 02:58:12.868192 19463 leader_election.cc:343] T 8bd3e1c06b89426da57e2b259e7983af P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Not found: Tablet not found: 8bd3e1c06b89426da57e2b259e7983af
I20260130 02:58:12.871206 20542 consensus_queue.cc:1048] T 16691c9315e94253bb9b3fb5058e5636 P 78c2c46c7520492db35d51c244fb4f74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260130 02:58:12.877106 19462 leader_election.cc:343] T 8bd3e1c06b89426da57e2b259e7983af P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767): Not found: Tablet not found: 8bd3e1c06b89426da57e2b259e7983af
I20260130 02:58:12.877472 19462 leader_election.cc:304] T 8bd3e1c06b89426da57e2b259e7983af P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d870164263bb44b6b780a557daea3656; no voters: 371a36d219164974abdf848841528152, 78c2c46c7520492db35d51c244fb4f74
I20260130 02:58:12.878324 20530 raft_consensus.cc:2749] T 8bd3e1c06b89426da57e2b259e7983af P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260130 02:58:12.879029 19840 ts_tablet_manager.cc:1434] T 274530359b174ff79f046c2d4a6afa76 P 78c2c46c7520492db35d51c244fb4f74: Time spent starting tablet: real 0.019s user 0.006s sys 0.000s
I20260130 02:58:12.881800 20530 consensus_queue.cc:1048] T 3b02480d9c6f4328af83f7940716d698 P d870164263bb44b6b780a557daea3656 [LEADER]: Connected to new peer: Peer: permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260130 02:58:12.884377 19840 tablet_bootstrap.cc:492] T 03025d13f0e545cfa6288677a2715343 P 78c2c46c7520492db35d51c244fb4f74: Bootstrap starting.
I20260130 02:58:12.886147 19625 tablet_bootstrap.cc:654] T 8e57ae31ab3941f789ac69627384e3b0 P d870164263bb44b6b780a557daea3656: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:12.888401 18871 catalog_manager.cc:5746] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021) does not have the latest schema for tablet 54be22352e404ff9b798a075693f7f21 (table default.Table_3a858a3cb886427cb8268f42c953ad4c [id=17150cb73e374ff9a1e92c43e0c3f2c1]). Expected version 1 got 0
I20260130 02:58:12.894243 20537 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260130 02:58:12.896505 19807 tablet_bootstrap.cc:700] T 3389db58493448aea3cf00aca411047c P 371a36d219164974abdf848841528152: Time spent opening tablet: real 0.365s user 0.001s sys 0.001s
I20260130 02:58:12.897202 19807 tablet_bootstrap.cc:654] T 3389db58493448aea3cf00aca411047c P 371a36d219164974abdf848841528152: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:12.903936 20474 raft_consensus.cc:515] T 4c139aa8cd184f488cec4e05ef7dcada P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:12.907374 20491 raft_consensus.cc:493] T 7caf7cbad5054396b5617094e4d536a0 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:12.907838 20491 raft_consensus.cc:515] T 7caf7cbad5054396b5617094e4d536a0 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:12.908526 19840 tablet_bootstrap.cc:654] T 03025d13f0e545cfa6288677a2715343 P 78c2c46c7520492db35d51c244fb4f74: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:12.909720 20491 leader_election.cc:290] T 7caf7cbad5054396b5617094e4d536a0 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767)
I20260130 02:58:12.912786 19376 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7caf7cbad5054396b5617094e4d536a0" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74" is_pre_election: true
I20260130 02:58:12.913331 19376 raft_consensus.cc:2468] T 7caf7cbad5054396b5617094e4d536a0 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d870164263bb44b6b780a557daea3656 in term 0.
I20260130 02:58:12.916049 19462 leader_election.cc:304] T 7caf7cbad5054396b5617094e4d536a0 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78c2c46c7520492db35d51c244fb4f74, d870164263bb44b6b780a557daea3656; no voters:
I20260130 02:58:12.916289 19223 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7caf7cbad5054396b5617094e4d536a0" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152" is_pre_election: true
I20260130 02:58:12.916780 20530 raft_consensus.cc:2804] T 7caf7cbad5054396b5617094e4d536a0 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260130 02:58:12.917132 20530 raft_consensus.cc:493] T 7caf7cbad5054396b5617094e4d536a0 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260130 02:58:12.917490 20530 raft_consensus.cc:3060] T 7caf7cbad5054396b5617094e4d536a0 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Advancing to term 1
W20260130 02:58:12.917796 19463 leader_election.cc:343] T 7caf7cbad5054396b5617094e4d536a0 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:12.924590 20530 raft_consensus.cc:515] T 7caf7cbad5054396b5617094e4d536a0 P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:12.940332 19840 tablet_bootstrap.cc:492] T 03025d13f0e545cfa6288677a2715343 P 78c2c46c7520492db35d51c244fb4f74: No bootstrap required, opened a new log
I20260130 02:58:12.941071 19376 raft_consensus.cc:3060] T 3b02480d9c6f4328af83f7940716d698 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:12.947963 19840 ts_tablet_manager.cc:1403] T 03025d13f0e545cfa6288677a2715343 P 78c2c46c7520492db35d51c244fb4f74: Time spent bootstrapping tablet: real 0.064s user 0.010s sys 0.000s
I20260130 02:58:12.950470 19840 raft_consensus.cc:359] T 03025d13f0e545cfa6288677a2715343 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:12.951260 19840 raft_consensus.cc:385] T 03025d13f0e545cfa6288677a2715343 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:12.951560 19840 raft_consensus.cc:740] T 03025d13f0e545cfa6288677a2715343 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78c2c46c7520492db35d51c244fb4f74, State: Initialized, Role: FOLLOWER
I20260130 02:58:12.952247 19840 consensus_queue.cc:260] T 03025d13f0e545cfa6288677a2715343 P 78c2c46c7520492db35d51c244fb4f74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:12.954824 19840 ts_tablet_manager.cc:1434] T 03025d13f0e545cfa6288677a2715343 P 78c2c46c7520492db35d51c244fb4f74: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260130 02:58:12.956604 19840 tablet_bootstrap.cc:492] T f41f4cff7dd4489f948700a45e9a7057 P 78c2c46c7520492db35d51c244fb4f74: Bootstrap starting.
I20260130 02:58:12.962869 19872 fs_manager.cc:730] Time spent opening block manager: real 0.338s user 0.019s sys 0.041s
I20260130 02:58:12.963379 19872 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/master-2/data,/tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/master-2/wal
uuid: "3749baa287ed4f249329a74071781edd"
format_stamp: "Formatted at 2026-01-30 02:57:44 on dist-test-slave-mn07"
I20260130 02:58:12.967597 19872 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260130 02:58:12.975310 20559 tablet.cc:1722] T 127a84346b9f473d9e7f739b1e65feb8 P 371a36d219164974abdf848841528152: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 5
I20260130 02:58:12.983217 19205 tablet_service.cc:1505] Processing CreateTablet for tablet ab6afa40408a4f458495abf20290e55b (DEFAULT_TABLE table=default.Table_748cb9c252084fb48ceac74812b82a64 [id=237bc2670f354027a817a6203f7e662d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:12.889676 18871 catalog_manager.cc:5746] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021) does not have the latest schema for tablet ee7c8ff92ca9465fa18be6eae63d78ca (table default.Table_dd22e9dc63f84d8688c0071f427f0b25 [id=77979d498aa040828de415129cb15a06]). Expected version 1 got 0
I20260130 02:58:12.984179 19205 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ab6afa40408a4f458495abf20290e55b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:12.986110 19807 tablet_bootstrap.cc:492] T 3389db58493448aea3cf00aca411047c P 371a36d219164974abdf848841528152: No bootstrap required, opened a new log
I20260130 02:58:12.986487 19807 ts_tablet_manager.cc:1403] T 3389db58493448aea3cf00aca411047c P 371a36d219164974abdf848841528152: Time spent bootstrapping tablet: real 0.495s user 0.004s sys 0.004s
I20260130 02:58:12.986850 19840 tablet_bootstrap.cc:654] T f41f4cff7dd4489f948700a45e9a7057 P 78c2c46c7520492db35d51c244fb4f74: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:12.948554 20491 consensus_queue.cc:1048] T 3b02480d9c6f4328af83f7940716d698 P d870164263bb44b6b780a557daea3656 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260130 02:58:12.961593 19496 tablet_service.cc:1505] Processing CreateTablet for tablet ab6afa40408a4f458495abf20290e55b (DEFAULT_TABLE table=default.Table_748cb9c252084fb48ceac74812b82a64 [id=237bc2670f354027a817a6203f7e662d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:12.999387 19496 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ab6afa40408a4f458495abf20290e55b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:13.004681 19508 tablet_service.cc:1505] Processing CreateTablet for tablet eb7a755ab62b4552bb8588264c5f7b75 (DEFAULT_TABLE table=default.Table_7e6ab9964f4c4e0f906d875a937a781d [id=bcf41af45d284bd89800977d14194812]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:13.005649 19508 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eb7a755ab62b4552bb8588264c5f7b75. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:13.008292 19350 tablet_service.cc:1505] Processing CreateTablet for tablet eb7a755ab62b4552bb8588264c5f7b75 (DEFAULT_TABLE table=default.Table_7e6ab9964f4c4e0f906d875a937a781d [id=bcf41af45d284bd89800977d14194812]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:13.009971 19350 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eb7a755ab62b4552bb8588264c5f7b75. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:13.011250 19198 tablet_service.cc:1505] Processing CreateTablet for tablet eb7a755ab62b4552bb8588264c5f7b75 (DEFAULT_TABLE table=default.Table_7e6ab9964f4c4e0f906d875a937a781d [id=bcf41af45d284bd89800977d14194812]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:13.016532 19840 tablet_bootstrap.cc:492] T f41f4cff7dd4489f948700a45e9a7057 P 78c2c46c7520492db35d51c244fb4f74: No bootstrap required, opened a new log
I20260130 02:58:13.016999 19840 ts_tablet_manager.cc:1403] T f41f4cff7dd4489f948700a45e9a7057 P 78c2c46c7520492db35d51c244fb4f74: Time spent bootstrapping tablet: real 0.061s user 0.009s sys 0.000s
I20260130 02:58:12.965883 20538 raft_consensus.cc:515] T 4907294442c145c6b03eb1de057c66a8 P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:13.019258 19840 raft_consensus.cc:359] T f41f4cff7dd4489f948700a45e9a7057 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:13.019920 19840 raft_consensus.cc:385] T f41f4cff7dd4489f948700a45e9a7057 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:13.019677 19225 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4907294442c145c6b03eb1de057c66a8" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152"
I20260130 02:58:13.020206 19840 raft_consensus.cc:740] T f41f4cff7dd4489f948700a45e9a7057 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78c2c46c7520492db35d51c244fb4f74, State: Initialized, Role: FOLLOWER
I20260130 02:58:13.020429 19225 raft_consensus.cc:3060] T 4907294442c145c6b03eb1de057c66a8 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:13.021517 19840 consensus_queue.cc:260] T f41f4cff7dd4489f948700a45e9a7057 P 78c2c46c7520492db35d51c244fb4f74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:13.023741 19840 ts_tablet_manager.cc:1434] T f41f4cff7dd4489f948700a45e9a7057 P 78c2c46c7520492db35d51c244fb4f74: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260130 02:58:13.024560 19840 tablet_bootstrap.cc:492] T b1e8b78eff0d4a0b92059adb2de4a3ce P 78c2c46c7520492db35d51c244fb4f74: Bootstrap starting.
I20260130 02:58:12.984266 18871 catalog_manager.cc:5746] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021) does not have the latest schema for tablet 16691c9315e94253bb9b3fb5058e5636 (table default.Table_8f311c5878744ef692b496ecfdfc3ede [id=4f499357303d464aafc17679ffcaaa07]). Expected version 1 got 0
I20260130 02:58:13.025326 18871 catalog_manager.cc:5746] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021) does not have the latest schema for tablet 127a84346b9f473d9e7f739b1e65feb8 (table default.Table_4f2d6e4d9c8e465087d65efc9c2de442 [id=1515db389f284e2f9ed1a3cd0061d0f1]). Expected version 5 got 0
I20260130 02:58:13.027608 18874 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:50284 (request call id 227) took 1711 ms. Trace:
I20260130 02:58:13.029503 19840 tablet_bootstrap.cc:654] T b1e8b78eff0d4a0b92059adb2de4a3ce P 78c2c46c7520492db35d51c244fb4f74: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:13.027920 18874 rpcz_store.cc:276] 0130 02:58:11.316148 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:58:11.501559 (+185411us) service_pool.cc:225] Handling call
0130 02:58:11.502508 (+ 949us) catalog_manager.cc:2393] Looking up, locking, and authorizing table
0130 02:58:11.503182 (+ 674us) catalog_manager.cc:3708] Apply alter schema
0130 02:58:11.512613 (+ 9431us) catalog_manager.cc:3740] Acquired catalog manager lock
0130 02:58:11.512813 (+ 200us) catalog_manager.cc:3898] Updating metadata on disk
0130 02:58:11.642544 (+129731us) catalog_manager.cc:3929] Committing alterations to in-memory state
0130 02:58:13.027545 (+1385001us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0130 02:58:11.513942 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0130 02:58:11.514317 (+ 375us) write_op.cc:432] Acquiring schema lock in shared mode
0130 02:58:11.514339 (+ 22us) write_op.cc:435] Acquired schema lock
0130 02:58:11.514350 (+ 11us) tablet.cc:660] Decoding operations
0130 02:58:11.514604 (+ 254us) write_op.cc:620] Acquiring the partition lock for write op
0130 02:58:11.514644 (+ 40us) write_op.cc:641] Partition lock acquired for write op
0130 02:58:11.514665 (+ 21us) tablet.cc:683] Acquiring locks for 1 operations
0130 02:58:11.514797 (+ 132us) tablet.cc:699] Row locks acquired
0130 02:58:11.514809 (+ 12us) write_op.cc:260] PREPARE: finished
0130 02:58:11.514905 (+ 96us) write_op.cc:270] Start()
0130 02:58:11.514982 (+ 77us) write_op.cc:276] Timestamp: P: 1769741891514822 usec, L: 0
0130 02:58:11.514996 (+ 14us) op_driver.cc:348] REPLICATION: starting
0130 02:58:11.515422 (+ 426us) log.cc:844] Serialized 405 byte log entry
0130 02:58:11.640049 (+124627us) op_driver.cc:464] REPLICATION: finished
0130 02:58:11.640351 (+ 302us) write_op.cc:301] APPLY: starting
0130 02:58:11.640504 (+ 153us) tablet.cc:1365] starting BulkCheckPresence
0130 02:58:11.640594 (+ 90us) tablet.cc:1368] finished BulkCheckPresence
0130 02:58:11.640685 (+ 91us) tablet.cc:1370] starting ApplyRowOperation cycle
0130 02:58:11.640879 (+ 194us) tablet.cc:1381] finished ApplyRowOperation cycle
0130 02:58:11.641006 (+ 127us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0130 02:58:11.641048 (+ 42us) write_op.cc:312] APPLY: finished
0130 02:58:11.641228 (+ 180us) log.cc:844] Serialized 23 byte log entry
0130 02:58:11.641832 (+ 604us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:58:11.641895 (+ 63us) write_op.cc:454] Released schema lock
0130 02:58:11.641925 (+ 30us) write_op.cc:341] FINISH: Updating metrics
0130 02:58:11.642126 (+ 201us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:58:11.642149 (+ 23us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":164,"apply.run_cpu_time_us":1225,"apply.run_wall_time_us":1738,"num_ops":1,"prepare.queue_time_us":86,"prepare.run_cpu_time_us":1727,"prepare.run_wall_time_us":2743,"raft.queue_time_us":71,"raft.run_cpu_time_us":11,"raft.run_wall_time_us":12,"replication_time_us":124957}]]}
I20260130 02:58:13.066057 19222 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4c139aa8cd184f488cec4e05ef7dcada" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152"
I20260130 02:58:13.066686 19222 raft_consensus.cc:3060] T 4c139aa8cd184f488cec4e05ef7dcada P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:13.070242 19198 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eb7a755ab62b4552bb8588264c5f7b75. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:13.076757 19840 tablet_bootstrap.cc:492] T b1e8b78eff0d4a0b92059adb2de4a3ce P 78c2c46c7520492db35d51c244fb4f74: No bootstrap required, opened a new log
I20260130 02:58:13.077159 19840 ts_tablet_manager.cc:1403] T b1e8b78eff0d4a0b92059adb2de4a3ce P 78c2c46c7520492db35d51c244fb4f74: Time spent bootstrapping tablet: real 0.053s user 0.006s sys 0.002s
I20260130 02:58:13.079298 19840 raft_consensus.cc:359] T b1e8b78eff0d4a0b92059adb2de4a3ce P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:13.079897 19840 raft_consensus.cc:385] T b1e8b78eff0d4a0b92059adb2de4a3ce P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:13.080128 19840 raft_consensus.cc:740] T b1e8b78eff0d4a0b92059adb2de4a3ce P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78c2c46c7520492db35d51c244fb4f74, State: Initialized, Role: FOLLOWER
I20260130 02:58:13.088301 20559 tablet.cc:1722] T ee7c8ff92ca9465fa18be6eae63d78ca P 371a36d219164974abdf848841528152: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260130 02:58:13.088656 19222 raft_consensus.cc:2468] T 4c139aa8cd184f488cec4e05ef7dcada P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78c2c46c7520492db35d51c244fb4f74 in term 1.
I20260130 02:58:13.089732 19314 leader_election.cc:304] T 4c139aa8cd184f488cec4e05ef7dcada P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 371a36d219164974abdf848841528152, 78c2c46c7520492db35d51c244fb4f74; no voters:
I20260130 02:58:13.090422 20479 raft_consensus.cc:2804] T 4c139aa8cd184f488cec4e05ef7dcada P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Leader election won for term 1
I20260130 02:58:13.090866 20479 raft_consensus.cc:697] T 4c139aa8cd184f488cec4e05ef7dcada P 78c2c46c7520492db35d51c244fb4f74 [term 1 LEADER]: Becoming Leader. State: Replica: 78c2c46c7520492db35d51c244fb4f74, State: Running, Role: LEADER
I20260130 02:58:13.091575 20479 consensus_queue.cc:237] T 4c139aa8cd184f488cec4e05ef7dcada P 78c2c46c7520492db35d51c244fb4f74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:13.094338 20566 tablet.cc:1722] T 127a84346b9f473d9e7f739b1e65feb8 P d870164263bb44b6b780a557daea3656: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 5
I20260130 02:58:13.097899 19524 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4c139aa8cd184f488cec4e05ef7dcada" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656"
W20260130 02:58:13.101425 19313 leader_election.cc:343] T 4c139aa8cd184f488cec4e05ef7dcada P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:13.105880 19840 consensus_queue.cc:260] T b1e8b78eff0d4a0b92059adb2de4a3ce P 78c2c46c7520492db35d51c244fb4f74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
W20260130 02:58:13.117408 20549 log.cc:927] Time spent T 753344737b5941d9a93b6779e72f63d4 P d870164263bb44b6b780a557daea3656: Append to log took a long time: real 0.150s user 0.000s sys 0.000s
I20260130 02:58:13.108016 19840 ts_tablet_manager.cc:1434] T b1e8b78eff0d4a0b92059adb2de4a3ce P 78c2c46c7520492db35d51c244fb4f74: Time spent starting tablet: real 0.031s user 0.006s sys 0.000s
I20260130 02:58:13.118742 19807 raft_consensus.cc:359] T 3389db58493448aea3cf00aca411047c P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:13.119378 19840 tablet_bootstrap.cc:492] T a90712651a604b83ba187f893ab17711 P 78c2c46c7520492db35d51c244fb4f74: Bootstrap starting.
I20260130 02:58:13.119362 19807 raft_consensus.cc:385] T 3389db58493448aea3cf00aca411047c P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:13.122045 18869 catalog_manager.cc:5941] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4: Sending DeleteTablet for 3 replicas of tablet 3b148691090e4ee581d46f9845a6e49a
I20260130 02:58:13.123451 18869 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:50284 (request call id 217) took 3957 ms. Trace:
I20260130 02:58:13.124449 19840 tablet_bootstrap.cc:654] T a90712651a604b83ba187f893ab17711 P 78c2c46c7520492db35d51c244fb4f74: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:13.124805 20544 raft_consensus.cc:493] T ba91a508b1cb4430a4d8ca26946aecc8 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:13.125239 20544 raft_consensus.cc:515] T ba91a508b1cb4430a4d8ca26946aecc8 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:13.123747 18869 rpcz_store.cc:276] 0130 02:58:09.166512 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:58:10.656265 (+1489753us) service_pool.cc:225] Handling call
0130 02:58:10.657440 (+ 1175us) catalog_manager.cc:2393] Looking up, locking, and authorizing table
0130 02:58:10.658110 (+ 670us) catalog_manager.cc:2815] Modifying in-memory table state
0130 02:58:10.658249 (+ 139us) catalog_manager.cc:2824] Locking tablets
0130 02:58:11.131524 (+473275us) catalog_manager.cc:2838] Removing table and tablets from system table
0130 02:58:11.176702 (+ 45178us) catalog_manager.cc:2856] Removing table from by-name map
0130 02:58:11.176945 (+ 243us) catalog_manager.cc:2872] Committing in-memory state
0130 02:58:13.121933 (+1944988us) catalog_manager.cc:2876] Aborting table tasks
0130 02:58:13.123395 (+ 1462us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0130 02:58:11.135838 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0130 02:58:11.136325 (+ 487us) write_op.cc:432] Acquiring schema lock in shared mode
0130 02:58:11.136352 (+ 27us) write_op.cc:435] Acquired schema lock
0130 02:58:11.136364 (+ 12us) tablet.cc:660] Decoding operations
0130 02:58:11.136723 (+ 359us) write_op.cc:620] Acquiring the partition lock for write op
0130 02:58:11.136767 (+ 44us) write_op.cc:641] Partition lock acquired for write op
0130 02:58:11.136798 (+ 31us) tablet.cc:683] Acquiring locks for 2 operations
0130 02:58:11.137082 (+ 284us) tablet.cc:699] Row locks acquired
0130 02:58:11.137094 (+ 12us) write_op.cc:260] PREPARE: finished
0130 02:58:11.137225 (+ 131us) write_op.cc:270] Start()
0130 02:58:11.137322 (+ 97us) write_op.cc:276] Timestamp: P: 1769741891137176 usec, L: 0
0130 02:58:11.137339 (+ 17us) op_driver.cc:348] REPLICATION: starting
0130 02:58:11.137878 (+ 539us) log.cc:844] Serialized 796 byte log entry
0130 02:58:11.169784 (+ 31906us) op_driver.cc:464] REPLICATION: finished
0130 02:58:11.173899 (+ 4115us) write_op.cc:301] APPLY: starting
0130 02:58:11.173943 (+ 44us) tablet.cc:1365] starting BulkCheckPresence
0130 02:58:11.174033 (+ 90us) tablet.cc:1368] finished BulkCheckPresence
0130 02:58:11.174134 (+ 101us) tablet.cc:1370] starting ApplyRowOperation cycle
0130 02:58:11.174383 (+ 249us) tablet.cc:1381] finished ApplyRowOperation cycle
0130 02:58:11.174497 (+ 114us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0130 02:58:11.174535 (+ 38us) write_op.cc:312] APPLY: finished
0130 02:58:11.175529 (+ 994us) log.cc:844] Serialized 29 byte log entry
0130 02:58:11.175663 (+ 134us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:58:11.175725 (+ 62us) write_op.cc:454] Released schema lock
0130 02:58:11.175757 (+ 32us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":3996,"apply.run_cpu_time_us":1256,"apply.run_wall_time_us":5599,"num_ops":2,"prepare.queue_time_us":2399,"prepare.run_cpu_time_us":2279,"prepare.run_wall_time_us":3334,"replication_time_us":32375,"thread_start_us":95903,"threads_started":1}]]}
I20260130 02:58:13.127110 20544 leader_election.cc:290] T ba91a508b1cb4430a4d8ca26946aecc8 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), d870164263bb44b6b780a557daea3656 (127.18.100.67:39571)
I20260130 02:58:13.123009 20530 leader_election.cc:290] T 7caf7cbad5054396b5617094e4d536a0 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 election: Requested vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767)
I20260130 02:58:13.131330 19355 tablet_service.cc:1552] Processing DeleteTablet for tablet 3b148691090e4ee581d46f9845a6e49a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:58:10 UTC) from {username='slave'} at 127.0.0.1:60110
I20260130 02:58:13.134135 19528 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ba91a508b1cb4430a4d8ca26946aecc8" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
I20260130 02:58:13.135746 19378 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7caf7cbad5054396b5617094e4d536a0" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74"
I20260130 02:58:13.136437 19378 raft_consensus.cc:3060] T 7caf7cbad5054396b5617094e4d536a0 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:13.139006 19496 tablet_service.cc:1552] Processing DeleteTablet for tablet 3b148691090e4ee581d46f9845a6e49a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:58:10 UTC) from {username='slave'} at 127.0.0.1:59068
W20260130 02:58:13.141332 19313 leader_election.cc:343] T ba91a508b1cb4430a4d8ca26946aecc8 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:13.159122 19378 raft_consensus.cc:2468] T 7caf7cbad5054396b5617094e4d536a0 P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d870164263bb44b6b780a557daea3656 in term 1.
I20260130 02:58:13.170094 20571 tablet_replica.cc:333] T 3b148691090e4ee581d46f9845a6e49a P d870164263bb44b6b780a557daea3656: stopping tablet replica
I20260130 02:58:13.163911 19840 tablet_bootstrap.cc:492] T a90712651a604b83ba187f893ab17711 P 78c2c46c7520492db35d51c244fb4f74: No bootstrap required, opened a new log
I20260130 02:58:13.180321 19840 ts_tablet_manager.cc:1403] T a90712651a604b83ba187f893ab17711 P 78c2c46c7520492db35d51c244fb4f74: Time spent bootstrapping tablet: real 0.061s user 0.009s sys 0.004s
I20260130 02:58:13.180850 19462 leader_election.cc:304] T 7caf7cbad5054396b5617094e4d536a0 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78c2c46c7520492db35d51c244fb4f74, d870164263bb44b6b780a557daea3656; no voters:
I20260130 02:58:13.182812 20562 raft_consensus.cc:2804] T 7caf7cbad5054396b5617094e4d536a0 P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Leader election won for term 1
I20260130 02:58:13.182602 19840 raft_consensus.cc:359] T a90712651a604b83ba187f893ab17711 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:13.183228 19840 raft_consensus.cc:385] T a90712651a604b83ba187f893ab17711 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:13.183480 19840 raft_consensus.cc:740] T a90712651a604b83ba187f893ab17711 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78c2c46c7520492db35d51c244fb4f74, State: Initialized, Role: FOLLOWER
I20260130 02:58:13.183626 20562 raft_consensus.cc:697] T 7caf7cbad5054396b5617094e4d536a0 P d870164263bb44b6b780a557daea3656 [term 1 LEADER]: Becoming Leader. State: Replica: d870164263bb44b6b780a557daea3656, State: Running, Role: LEADER
I20260130 02:58:13.184010 19840 consensus_queue.cc:260] T a90712651a604b83ba187f893ab17711 P 78c2c46c7520492db35d51c244fb4f74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:13.186313 19840 ts_tablet_manager.cc:1434] T a90712651a604b83ba187f893ab17711 P 78c2c46c7520492db35d51c244fb4f74: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260130 02:58:13.187193 19840 tablet_bootstrap.cc:492] T 7783db3568f4434e91e9e456799cc312 P 78c2c46c7520492db35d51c244fb4f74: Bootstrap starting.
I20260130 02:58:13.187491 20568 tablet.cc:1722] T ee7c8ff92ca9465fa18be6eae63d78ca P 78c2c46c7520492db35d51c244fb4f74: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260130 02:58:13.187577 20562 consensus_queue.cc:237] T 7caf7cbad5054396b5617094e4d536a0 P d870164263bb44b6b780a557daea3656 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:13.195114 19840 tablet_bootstrap.cc:654] T 7783db3568f4434e91e9e456799cc312 P 78c2c46c7520492db35d51c244fb4f74: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:13.202039 18872 catalog_manager.cc:5941] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4: Sending DeleteTablet for 3 replicas of tablet e364f4e70e5749e083e14e7fa7416ff0
I20260130 02:58:13.203032 19625 tablet_bootstrap.cc:492] T 8e57ae31ab3941f789ac69627384e3b0 P d870164263bb44b6b780a557daea3656: No bootstrap required, opened a new log
I20260130 02:58:13.204962 19625 ts_tablet_manager.cc:1403] T 8e57ae31ab3941f789ac69627384e3b0 P d870164263bb44b6b780a557daea3656: Time spent bootstrapping tablet: real 0.346s user 0.010s sys 0.000s
I20260130 02:58:13.220396 19625 raft_consensus.cc:359] T 8e57ae31ab3941f789ac69627384e3b0 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:13.223733 19625 raft_consensus.cc:385] T 8e57ae31ab3941f789ac69627384e3b0 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:13.227648 18872 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:50284 (request call id 216) took 4233 ms. Trace:
I20260130 02:58:13.229341 19625 raft_consensus.cc:740] T 8e57ae31ab3941f789ac69627384e3b0 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d870164263bb44b6b780a557daea3656, State: Initialized, Role: FOLLOWER
I20260130 02:58:13.231334 19496 tablet_service.cc:1552] Processing DeleteTablet for tablet e364f4e70e5749e083e14e7fa7416ff0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:58:10 UTC) from {username='slave'} at 127.0.0.1:59068
I20260130 02:58:13.227963 18872 rpcz_store.cc:276] 0130 02:58:08.994203 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:58:10.545275 (+1551072us) service_pool.cc:225] Handling call
0130 02:58:10.583610 (+ 38335us) catalog_manager.cc:2393] Looking up, locking, and authorizing table
0130 02:58:10.605716 (+ 22106us) catalog_manager.cc:2815] Modifying in-memory table state
0130 02:58:10.605919 (+ 203us) catalog_manager.cc:2824] Locking tablets
0130 02:58:10.606543 (+ 624us) catalog_manager.cc:2838] Removing table and tablets from system table
0130 02:58:11.036009 (+429466us) catalog_manager.cc:2856] Removing table from by-name map
0130 02:58:11.036205 (+ 196us) catalog_manager.cc:2872] Committing in-memory state
0130 02:58:13.201920 (+2165715us) catalog_manager.cc:2876] Aborting table tasks
0130 02:58:13.227581 (+ 25661us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0130 02:58:10.740333 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0130 02:58:10.740817 (+ 484us) write_op.cc:432] Acquiring schema lock in shared mode
0130 02:58:10.740847 (+ 30us) write_op.cc:435] Acquired schema lock
0130 02:58:10.740857 (+ 10us) tablet.cc:660] Decoding operations
0130 02:58:10.741232 (+ 375us) write_op.cc:620] Acquiring the partition lock for write op
0130 02:58:10.741276 (+ 44us) write_op.cc:641] Partition lock acquired for write op
0130 02:58:10.741300 (+ 24us) tablet.cc:683] Acquiring locks for 2 operations
0130 02:58:10.741592 (+ 292us) tablet.cc:699] Row locks acquired
0130 02:58:10.741609 (+ 17us) write_op.cc:260] PREPARE: finished
0130 02:58:10.741736 (+ 127us) write_op.cc:270] Start()
0130 02:58:10.741826 (+ 90us) write_op.cc:276] Timestamp: P: 1769741890741675 usec, L: 0
0130 02:58:10.741845 (+ 19us) op_driver.cc:348] REPLICATION: starting
0130 02:58:10.742416 (+ 571us) log.cc:844] Serialized 771 byte log entry
0130 02:58:10.973172 (+230756us) op_driver.cc:464] REPLICATION: finished
0130 02:58:11.022293 (+ 49121us) write_op.cc:301] APPLY: starting
0130 02:58:11.022347 (+ 54us) tablet.cc:1365] starting BulkCheckPresence
0130 02:58:11.022439 (+ 92us) tablet.cc:1368] finished BulkCheckPresence
0130 02:58:11.022538 (+ 99us) tablet.cc:1370] starting ApplyRowOperation cycle
0130 02:58:11.034607 (+ 12069us) tablet.cc:1381] finished ApplyRowOperation cycle
0130 02:58:11.034758 (+ 151us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0130 02:58:11.034796 (+ 38us) write_op.cc:312] APPLY: finished
0130 02:58:11.034969 (+ 173us) log.cc:844] Serialized 29 byte log entry
0130 02:58:11.035707 (+ 738us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:58:11.035773 (+ 66us) write_op.cc:454] Released schema lock
0130 02:58:11.035804 (+ 31us) write_op.cc:341] FINISH: Updating metrics
0130 02:58:11.036404 (+ 600us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:58:11.036429 (+ 25us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":48928,"apply.run_cpu_time_us":1506,"apply.run_wall_time_us":14084,"num_ops":2,"prepare.queue_time_us":53335,"prepare.run_cpu_time_us":2303,"prepare.run_wall_time_us":2304,"replication_time_us":231226,"thread_start_us":51196,"threads_started":1}]]}
I20260130 02:58:13.233261 19356 tablet_service.cc:1552] Processing DeleteTablet for tablet e364f4e70e5749e083e14e7fa7416ff0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:58:10 UTC) from {username='slave'} at 127.0.0.1:60110
I20260130 02:58:13.250348 19840 tablet_bootstrap.cc:492] T 7783db3568f4434e91e9e456799cc312 P 78c2c46c7520492db35d51c244fb4f74: No bootstrap required, opened a new log
I20260130 02:58:13.250795 19840 ts_tablet_manager.cc:1403] T 7783db3568f4434e91e9e456799cc312 P 78c2c46c7520492db35d51c244fb4f74: Time spent bootstrapping tablet: real 0.064s user 0.013s sys 0.000s
I20260130 02:58:13.252996 19840 raft_consensus.cc:359] T 7783db3568f4434e91e9e456799cc312 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:13.253614 19840 raft_consensus.cc:385] T 7783db3568f4434e91e9e456799cc312 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:13.258705 19840 raft_consensus.cc:740] T 7783db3568f4434e91e9e456799cc312 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78c2c46c7520492db35d51c244fb4f74, State: Initialized, Role: FOLLOWER
I20260130 02:58:13.259378 19840 consensus_queue.cc:260] T 7783db3568f4434e91e9e456799cc312 P 78c2c46c7520492db35d51c244fb4f74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:13.266299 19840 ts_tablet_manager.cc:1434] T 7783db3568f4434e91e9e456799cc312 P 78c2c46c7520492db35d51c244fb4f74: Time spent starting tablet: real 0.015s user 0.005s sys 0.000s
I20260130 02:58:13.273145 19840 tablet_bootstrap.cc:492] T 6e12a8eeb3204325a9a8b7779f72ab29 P 78c2c46c7520492db35d51c244fb4f74: Bootstrap starting.
I20260130 02:58:13.277838 20479 raft_consensus.cc:493] T a90712651a604b83ba187f893ab17711 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:13.278429 20479 raft_consensus.cc:515] T a90712651a604b83ba187f893ab17711 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:13.281208 20479 leader_election.cc:290] T a90712651a604b83ba187f893ab17711 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), d870164263bb44b6b780a557daea3656 (127.18.100.67:39571)
I20260130 02:58:13.282521 19526 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a90712651a604b83ba187f893ab17711" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
I20260130 02:58:13.288015 20538 leader_election.cc:290] T 4907294442c145c6b03eb1de057c66a8 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 election: Requested vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), d870164263bb44b6b780a557daea3656 (127.18.100.67:39571)
I20260130 02:58:13.288630 19524 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4907294442c145c6b03eb1de057c66a8" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656"
I20260130 02:58:13.291903 19840 tablet_bootstrap.cc:654] T 6e12a8eeb3204325a9a8b7779f72ab29 P 78c2c46c7520492db35d51c244fb4f74: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:13.297304 20570 tablet_replica.cc:333] T 3b148691090e4ee581d46f9845a6e49a P 78c2c46c7520492db35d51c244fb4f74: stopping tablet replica
I20260130 02:58:13.308348 19840 tablet_bootstrap.cc:492] T 6e12a8eeb3204325a9a8b7779f72ab29 P 78c2c46c7520492db35d51c244fb4f74: No bootstrap required, opened a new log
I20260130 02:58:13.308830 19840 ts_tablet_manager.cc:1403] T 6e12a8eeb3204325a9a8b7779f72ab29 P 78c2c46c7520492db35d51c244fb4f74: Time spent bootstrapping tablet: real 0.036s user 0.008s sys 0.000s
W20260130 02:58:13.311404 18861 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet 49efcfced0b74c48962393149ac2a500 (table default.Table_2e81f0a0cbc449a397141b717db17025 [id=a9c919f57c0c437cb35a6921103e45a6], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 49efcfced0b74c48962393149ac2a500 consensus state has no leader
I20260130 02:58:13.311359 19840 raft_consensus.cc:359] T 6e12a8eeb3204325a9a8b7779f72ab29 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:13.311960 19840 raft_consensus.cc:385] T 6e12a8eeb3204325a9a8b7779f72ab29 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:13.312217 19840 raft_consensus.cc:740] T 6e12a8eeb3204325a9a8b7779f72ab29 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78c2c46c7520492db35d51c244fb4f74, State: Initialized, Role: FOLLOWER
I20260130 02:58:13.320711 19840 consensus_queue.cc:260] T 6e12a8eeb3204325a9a8b7779f72ab29 P 78c2c46c7520492db35d51c244fb4f74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:13.066380 20474 leader_election.cc:290] T 4c139aa8cd184f488cec4e05ef7dcada P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 election: Requested vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), d870164263bb44b6b780a557daea3656 (127.18.100.67:39571)
I20260130 02:58:13.322263 19840 ts_tablet_manager.cc:1434] T 6e12a8eeb3204325a9a8b7779f72ab29 P 78c2c46c7520492db35d51c244fb4f74: Time spent starting tablet: real 0.013s user 0.004s sys 0.000s
I20260130 02:58:13.323172 19840 tablet_bootstrap.cc:492] T 78e9bc3e55fe4e4a98e8eb01da59d365 P 78c2c46c7520492db35d51c244fb4f74: Bootstrap starting.
I20260130 02:58:13.325407 19625 consensus_queue.cc:260] T 8e57ae31ab3941f789ac69627384e3b0 P d870164263bb44b6b780a557daea3656 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:13.334618 19625 ts_tablet_manager.cc:1434] T 8e57ae31ab3941f789ac69627384e3b0 P d870164263bb44b6b780a557daea3656: Time spent starting tablet: real 0.129s user 0.008s sys 0.000s
I20260130 02:58:13.340420 19351 tablet_service.cc:1505] Processing CreateTablet for tablet ab6afa40408a4f458495abf20290e55b (DEFAULT_TABLE table=default.Table_748cb9c252084fb48ceac74812b82a64 [id=237bc2670f354027a817a6203f7e662d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:13.336092 19625 tablet_bootstrap.cc:492] T 43cec19c51ec4f05b63800c07f9534ef P d870164263bb44b6b780a557daea3656: Bootstrap starting.
W20260130 02:58:13.354382 19313 leader_election.cc:343] T 4907294442c145c6b03eb1de057c66a8 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:13.354485 19502 tablet_service.cc:1552] Processing DeleteTablet for tablet 3b4908bd1a634ca0907234dd3c39e1c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:58:08 UTC) from {username='slave'} at 127.0.0.1:59068
W20260130 02:58:13.355458 19313 leader_election.cc:343] T a90712651a604b83ba187f893ab17711 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:13.358785 19496 tablet_service.cc:1552] Processing DeleteTablet for tablet 1bbc55bbf2254d8bb6b721d7e8f69d85 with delete_type TABLET_DATA_DELETED (replaced by tablet 657e9b0d8c05413f904129dce745710b) from {username='slave'} at 127.0.0.1:59068
I20260130 02:58:13.361325 19496 tablet_service.cc:1552] Processing DeleteTablet for tablet f54addb1a4304ce69c711ae0cc3b452c with delete_type TABLET_DATA_DELETED (replaced by tablet 7783db3568f4434e91e9e456799cc312) from {username='slave'} at 127.0.0.1:59068
I20260130 02:58:13.362236 20538 raft_consensus.cc:493] T 5fbe92a2227d44d88b4531a7f8aa0e08 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:13.362645 19496 tablet_service.cc:1552] Processing DeleteTablet for tablet 6573198830ad45628e71952a7d9c00a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:58:03 UTC) from {username='slave'} at 127.0.0.1:59068
I20260130 02:58:13.362682 20538 raft_consensus.cc:515] T 5fbe92a2227d44d88b4531a7f8aa0e08 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:13.363377 19840 tablet_bootstrap.cc:654] T 78e9bc3e55fe4e4a98e8eb01da59d365 P 78c2c46c7520492db35d51c244fb4f74: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:13.365355 20538 leader_election.cc:290] T 5fbe92a2227d44d88b4531a7f8aa0e08 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), d870164263bb44b6b780a557daea3656 (127.18.100.67:39571)
I20260130 02:58:13.366042 20566 tablet.cc:1722] T ee7c8ff92ca9465fa18be6eae63d78ca P d870164263bb44b6b780a557daea3656: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260130 02:58:13.366927 19523 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5fbe92a2227d44d88b4531a7f8aa0e08" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
W20260130 02:58:13.368016 19313 leader_election.cc:343] T 5fbe92a2227d44d88b4531a7f8aa0e08 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:13.368038 19496 tablet_service.cc:1552] Processing DeleteTablet for tablet b2ac91da01954c2390b6b7dcc06c394f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:58:04 UTC) from {username='slave'} at 127.0.0.1:59068
I20260130 02:58:13.370483 19502 tablet_service.cc:1552] Processing DeleteTablet for tablet 8bd3e1c06b89426da57e2b259e7983af with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:58:08 UTC) from {username='slave'} at 127.0.0.1:59068
I20260130 02:58:13.371927 19351 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ab6afa40408a4f458495abf20290e55b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:13.375219 20570 raft_consensus.cc:2243] T 3b148691090e4ee581d46f9845a6e49a P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260130 02:58:13.375860 20570 raft_consensus.cc:2272] T 3b148691090e4ee581d46f9845a6e49a P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260130 02:58:13.378095 20570 ts_tablet_manager.cc:1916] T 3b148691090e4ee581d46f9845a6e49a P 78c2c46c7520492db35d51c244fb4f74: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260130 02:58:13.129638 19223 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ba91a508b1cb4430a4d8ca26946aecc8" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152" is_pre_election: true
I20260130 02:58:13.134079 19197 tablet_service.cc:1552] Processing DeleteTablet for tablet f54addb1a4304ce69c711ae0cc3b452c with delete_type TABLET_DATA_DELETED (replaced by tablet 7783db3568f4434e91e9e456799cc312) from {username='slave'} at 127.0.0.1:42870
I20260130 02:58:13.134541 19199 tablet_service.cc:1552] Processing DeleteTablet for tablet 6573198830ad45628e71952a7d9c00a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:58:03 UTC) from {username='slave'} at 127.0.0.1:42870
I20260130 02:58:13.134958 19203 tablet_service.cc:1552] Processing DeleteTablet for tablet b2ac91da01954c2390b6b7dcc06c394f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:58:04 UTC) from {username='slave'} at 127.0.0.1:42870
I20260130 02:58:13.135079 19201 tablet_service.cc:1552] Processing DeleteTablet for tablet 730bd1d7d15b4cbe945b3ef03b2455f7 with delete_type TABLET_DATA_DELETED (replaced by tablet 3b02480d9c6f4328af83f7940716d698) from {username='slave'} at 127.0.0.1:42870
I20260130 02:58:13.136286 19200 tablet_service.cc:1552] Processing DeleteTablet for tablet 7d96e93d95f14af79cfb268626559b1e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:58:04 UTC) from {username='slave'} at 127.0.0.1:42870
I20260130 02:58:13.156082 19205 tablet_service.cc:1552] Processing DeleteTablet for tablet 3b148691090e4ee581d46f9845a6e49a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:58:10 UTC) from {username='slave'} at 127.0.0.1:42870
I20260130 02:58:13.161231 19222 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7caf7cbad5054396b5617094e4d536a0" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152"
I20260130 02:58:13.237165 19204 tablet_service.cc:1552] Processing DeleteTablet for tablet e364f4e70e5749e083e14e7fa7416ff0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:58:10 UTC) from {username='slave'} at 127.0.0.1:42870
I20260130 02:58:13.281083 19224 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a90712651a604b83ba187f893ab17711" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152" is_pre_election: true
W20260130 02:58:13.385169 19463 leader_election.cc:343] T 7caf7cbad5054396b5617094e4d536a0 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:13.387106 19204 tablet_service.cc:1505] Processing CreateTablet for tablet 0855d88806914a36b4bf6b49036f0ad0 (DEFAULT_TABLE table=default.Table_6773a0ad5d6a429086a5818d731f3f4a [id=90b72499c7174fd580914d7fe3714891]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:13.388173 19204 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0855d88806914a36b4bf6b49036f0ad0. 1 dirs total, 0 dirs full, 0 dirs failed
W20260130 02:58:13.398023 18861 catalog_manager.cc:4952] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021): delete failed for tablet f54addb1a4304ce69c711ae0cc3b452c because the tablet was not found. No further retry: Not found: Tablet not found: f54addb1a4304ce69c711ae0cc3b452c
W20260130 02:58:13.399571 18861 catalog_manager.cc:4952] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021): delete failed for tablet b2ac91da01954c2390b6b7dcc06c394f because the tablet was not found. No further retry: Not found: Tablet not found: b2ac91da01954c2390b6b7dcc06c394f
W20260130 02:58:13.400372 18861 catalog_manager.cc:4952] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021): delete failed for tablet 730bd1d7d15b4cbe945b3ef03b2455f7 because the tablet was not found. No further retry: Not found: Tablet not found: 730bd1d7d15b4cbe945b3ef03b2455f7
I20260130 02:58:13.402698 20581 tablet_replica.cc:333] T 7d96e93d95f14af79cfb268626559b1e P 371a36d219164974abdf848841528152: stopping tablet replica
I20260130 02:58:13.405920 20581 raft_consensus.cc:2243] T 7d96e93d95f14af79cfb268626559b1e P 371a36d219164974abdf848841528152 [term 1 LEADER]: Raft consensus shutting down.
I20260130 02:58:13.344884 19225 raft_consensus.cc:2468] T 4907294442c145c6b03eb1de057c66a8 P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78c2c46c7520492db35d51c244fb4f74 in term 1.
I20260130 02:58:13.354475 19202 tablet_service.cc:1505] Processing CreateTablet for tablet 1fc8f09d6329407c8078542b4123ef4d (DEFAULT_TABLE table=default.Table_7d1c27ab3515406ba5b5333c90df093e [id=b6fabeeae79941dc8094d0f2b51704f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:13.408434 20581 raft_consensus.cc:2272] T 7d96e93d95f14af79cfb268626559b1e P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260130 02:58:13.409339 19202 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1fc8f09d6329407c8078542b4123ef4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:13.410300 19496 tablet_service.cc:1505] Processing CreateTablet for tablet 0855d88806914a36b4bf6b49036f0ad0 (DEFAULT_TABLE table=default.Table_6773a0ad5d6a429086a5818d731f3f4a [id=90b72499c7174fd580914d7fe3714891]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:13.365432 19221 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5fbe92a2227d44d88b4531a7f8aa0e08" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152" is_pre_election: true
I20260130 02:58:13.411577 19502 tablet_service.cc:1505] Processing CreateTablet for tablet 1fc8f09d6329407c8078542b4123ef4d (DEFAULT_TABLE table=default.Table_7d1c27ab3515406ba5b5333c90df093e [id=b6fabeeae79941dc8094d0f2b51704f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:13.410418 20581 ts_tablet_manager.cc:1916] T 7d96e93d95f14af79cfb268626559b1e P 371a36d219164974abdf848841528152: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260130 02:58:13.320960 19159 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.100.66:40767, user_credentials={real_user=slave}} blocked reactor thread for 258636us
I20260130 02:58:13.431737 19807 raft_consensus.cc:740] T 3389db58493448aea3cf00aca411047c P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 371a36d219164974abdf848841528152, State: Initialized, Role: FOLLOWER
I20260130 02:58:13.432502 19807 consensus_queue.cc:260] T 3389db58493448aea3cf00aca411047c P 371a36d219164974abdf848841528152 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
W20260130 02:58:13.438186 18873 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.18.100.65:51021 (request call id 32) took 3766 ms (client timeout 1000 ms). Trace:
W20260130 02:58:13.438647 18873 rpcz_store.cc:269] 0130 02:58:09.671633 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:58:09.719120 (+ 47487us) service_pool.cc:225] Handling call
0130 02:58:13.438079 (+3718959us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":9476,"reported_tablets":36}
I20260130 02:58:13.442453 19625 tablet_bootstrap.cc:654] T 43cec19c51ec4f05b63800c07f9534ef P d870164263bb44b6b780a557daea3656: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:13.454708 20562 raft_consensus.cc:493] T f54addb1a4304ce69c711ae0cc3b452c P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 78c2c46c7520492db35d51c244fb4f74)
I20260130 02:58:13.455271 20562 raft_consensus.cc:515] T f54addb1a4304ce69c711ae0cc3b452c P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:13.457056 20562 leader_election.cc:290] T f54addb1a4304ce69c711ae0cc3b452c P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767)
W20260130 02:58:13.459688 19463 leader_election.cc:343] T f54addb1a4304ce69c711ae0cc3b452c P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Not found: Tablet not found: f54addb1a4304ce69c711ae0cc3b452c
W20260130 02:58:13.472927 18860 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.100.124:34629, user_credentials={real_user=slave}} blocked reactor thread for 60676.4us
I20260130 02:58:13.483731 19508 tablet_service.cc:1552] Processing DeleteTablet for tablet 730bd1d7d15b4cbe945b3ef03b2455f7 with delete_type TABLET_DATA_DELETED (replaced by tablet 3b02480d9c6f4328af83f7940716d698) from {username='slave'} at 127.0.0.1:59068
I20260130 02:58:13.485129 19508 tablet_service.cc:1552] Processing DeleteTablet for tablet 3cf5154d8cc44f2bb12978202699c193 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:58:06 UTC) from {username='slave'} at 127.0.0.1:59068
I20260130 02:58:13.501637 19508 tablet_service.cc:1552] Processing DeleteTablet for tablet 7d96e93d95f14af79cfb268626559b1e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:58:04 UTC) from {username='slave'} at 127.0.0.1:59068
W20260130 02:58:13.503139 18867 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.18.100.67:47213 (request call id 27) took 2696 ms (client timeout 1000 ms). Trace:
W20260130 02:58:13.503540 18867 rpcz_store.cc:269] 0130 02:58:10.807183 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:58:10.834660 (+ 27477us) service_pool.cc:225] Handling call
0130 02:58:13.503065 (+2668405us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":115306,"reported_tablets":31}
I20260130 02:58:13.504907 18867 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50284:
name: "default.Table_e894e5560f194692bac730600c6bf5f4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260130 02:58:13.506484 18867 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e894e5560f194692bac730600c6bf5f4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260130 02:58:13.511220 18875 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.18.100.67:47213 (request call id 26) took 3734 ms (client timeout 1000 ms). Trace:
W20260130 02:58:13.512039 18875 rpcz_store.cc:269] 0130 02:58:09.777092 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:58:09.835985 (+ 58893us) service_pool.cc:225] Handling call
0130 02:58:13.511058 (+3675073us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1265,"reported_tablets":30}
I20260130 02:58:13.513720 18875 master_service.cc:954] ReplaceTablet: received request to replace tablet d55291d7cea64a0d9a345ce712fbfa80 from {username='slave'} at 127.0.0.1:50284
I20260130 02:58:13.487360 20581 ts_tablet_manager.cc:1929] T 7d96e93d95f14af79cfb268626559b1e P 371a36d219164974abdf848841528152: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260130 02:58:13.515190 20581 log.cc:1199] T 7d96e93d95f14af79cfb268626559b1e P 371a36d219164974abdf848841528152: Deleting WAL directory at /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-0/wal/wals/7d96e93d95f14af79cfb268626559b1e
I20260130 02:58:13.516031 20581 ts_tablet_manager.cc:1950] T 7d96e93d95f14af79cfb268626559b1e P 371a36d219164974abdf848841528152: Deleting consensus metadata
I20260130 02:58:13.518244 20497 raft_consensus.cc:493] T 3389db58493448aea3cf00aca411047c P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:13.518704 20497 raft_consensus.cc:515] T 3389db58493448aea3cf00aca411047c P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:13.519426 20581 tablet_replica.cc:333] T e364f4e70e5749e083e14e7fa7416ff0 P 371a36d219164974abdf848841528152: stopping tablet replica
I20260130 02:58:13.520224 20581 raft_consensus.cc:2243] T e364f4e70e5749e083e14e7fa7416ff0 P 371a36d219164974abdf848841528152 [term 1 LEADER]: Raft consensus shutting down.
I20260130 02:58:13.520495 20497 leader_election.cc:290] T 3389db58493448aea3cf00aca411047c P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767), d870164263bb44b6b780a557daea3656 (127.18.100.67:39571)
I20260130 02:58:13.520795 20581 raft_consensus.cc:2272] T e364f4e70e5749e083e14e7fa7416ff0 P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260130 02:58:13.434962 19807 ts_tablet_manager.cc:1434] T 3389db58493448aea3cf00aca411047c P 371a36d219164974abdf848841528152: Time spent starting tablet: real 0.448s user 0.006s sys 0.000s
W20260130 02:58:13.530208 19807 ts_tablet_manager.cc:1467] T 3389db58493448aea3cf00aca411047c P 371a36d219164974abdf848841528152: Tablet startup took 543ms
I20260130 02:58:13.371721 19355 tablet_service.cc:1505] Processing CreateTablet for tablet 1fc8f09d6329407c8078542b4123ef4d (DEFAULT_TABLE table=default.Table_7d1c27ab3515406ba5b5333c90df093e [id=b6fabeeae79941dc8094d0f2b51704f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:13.531260 19355 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1fc8f09d6329407c8078542b4123ef4d. 1 dirs total, 0 dirs full, 0 dirs failed
W20260130 02:58:13.530505 19807 ts_tablet_manager.cc:1469] T 3389db58493448aea3cf00aca411047c P 371a36d219164974abdf848841528152: Trace:
0130 02:58:05.176307 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:58:05.178479 (+ 2172us) service_pool.cc:225] Handling call
0130 02:58:05.196199 (+ 17720us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0130 02:58:05.196346 (+ 147us) ts_tablet_manager.cc:682] Creating new metadata...
0130 02:58:05.236422 (+ 40076us) tablet_metadata.cc:641] Metadata flushed
0130 02:58:05.299857 (+ 63435us) tablet_replica.cc:206] Creating consensus instance
0130 02:58:05.300258 (+ 401us) inbound_call.cc:173] Queueing success response
0130 02:58:12.491786 (+7191528us) ts_tablet_manager.cc:1379] Bootstrapping tablet
0130 02:58:12.986785 (+494999us) ts_tablet_manager.cc:1435] Starting tablet replica
0130 02:58:12.987255 (+ 470us) tablet_replica.cc:261] Starting instrumentation
0130 02:58:13.068879 (+ 81624us) tablet_replica.cc:280] Starting consensus
Metrics: {"raft.queue_time_us":99,"raft.run_cpu_time_us":163,"raft.run_wall_time_us":165,"tablet-open.queue_time_us":7191617}
I20260130 02:58:13.531723 18861 catalog_manager.cc:4985] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021): tablet 7d96e93d95f14af79cfb268626559b1e (table default.Table_6eb0a92b1926487cbf0b978d26d6a7b1 [id=de7a420d40864f55b156e149308ba12f]) successfully deleted
W20260130 02:58:13.532378 18861 catalog_manager.cc:4952] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021): delete failed for tablet 6573198830ad45628e71952a7d9c00a0 because the tablet was not found. No further retry: Not found: Tablet not found: 6573198830ad45628e71952a7d9c00a0
W20260130 02:58:13.383059 19314 leader_election.cc:343] T ba91a508b1cb4430a4d8ca26946aecc8 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:13.532944 19314 leader_election.cc:304] T ba91a508b1cb4430a4d8ca26946aecc8 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78c2c46c7520492db35d51c244fb4f74; no voters: 371a36d219164974abdf848841528152, d870164263bb44b6b780a557daea3656
I20260130 02:58:13.533613 20474 raft_consensus.cc:2749] T ba91a508b1cb4430a4d8ca26946aecc8 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260130 02:58:13.533704 19314 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.100.65:42021, user_credentials={real_user=slave}} blocked reactor thread for 150683us
I20260130 02:58:13.534286 20581 ts_tablet_manager.cc:1916] T e364f4e70e5749e083e14e7fa7416ff0 P 371a36d219164974abdf848841528152: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260130 02:58:13.535524 19314 leader_election.cc:343] T a90712651a604b83ba187f893ab17711 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:13.535851 19314 leader_election.cc:304] T a90712651a604b83ba187f893ab17711 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78c2c46c7520492db35d51c244fb4f74; no voters: 371a36d219164974abdf848841528152, d870164263bb44b6b780a557daea3656
W20260130 02:58:13.442255 19313 consensus_peers.cc:597] T 9fe0448bcc4a4346b3c67fc3f626ed3c P 78c2c46c7520492db35d51c244fb4f74 -> Peer d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): Couldn't send request to peer d870164263bb44b6b780a557daea3656. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20260130 02:58:13.537102 19313 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.100.67:39571, user_credentials={real_user=slave}} blocked reactor thread for 94953.9us
I20260130 02:58:13.540264 19523 raft_consensus.cc:3060] T 8e57ae31ab3941f789ac69627384e3b0 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:13.547569 20544 raft_consensus.cc:2749] T a90712651a604b83ba187f893ab17711 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260130 02:58:13.555658 19351 tablet_service.cc:1552] Processing DeleteTablet for tablet f54addb1a4304ce69c711ae0cc3b452c with delete_type TABLET_DATA_DELETED (replaced by tablet 7783db3568f4434e91e9e456799cc312) from {username='slave'} at 127.0.0.1:60110
I20260130 02:58:13.556337 19356 tablet_service.cc:1552] Processing DeleteTablet for tablet 7b80b22bd44c4233837c951aa97e6996 with delete_type TABLET_DATA_DELETED (replaced by tablet 1fc8f09d6329407c8078542b4123ef4d) from {username='slave'} at 127.0.0.1:60110
I20260130 02:58:13.556856 19354 tablet_service.cc:1552] Processing DeleteTablet for tablet b2ac91da01954c2390b6b7dcc06c394f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:58:04 UTC) from {username='slave'} at 127.0.0.1:60110
W20260130 02:58:13.557780 18866 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.18.100.66:59691 (request call id 29) took 3348 ms (client timeout 1000 ms). Trace:
I20260130 02:58:13.559326 19351 tablet_service.cc:1552] Processing DeleteTablet for tablet 7d96e93d95f14af79cfb268626559b1e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:58:04 UTC) from {username='slave'} at 127.0.0.1:60110
W20260130 02:58:13.558072 18866 rpcz_store.cc:269] 0130 02:58:10.209816 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:58:10.400941 (+191125us) service_pool.cc:225] Handling call
0130 02:58:13.557716 (+3156775us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0130 02:58:11.792053 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0130 02:58:11.792410 (+ 357us) write_op.cc:432] Acquiring schema lock in shared mode
0130 02:58:11.792433 (+ 23us) write_op.cc:435] Acquired schema lock
0130 02:58:11.792445 (+ 12us) tablet.cc:660] Decoding operations
0130 02:58:11.792688 (+ 243us) write_op.cc:620] Acquiring the partition lock for write op
0130 02:58:11.792727 (+ 39us) write_op.cc:641] Partition lock acquired for write op
0130 02:58:11.792748 (+ 21us) tablet.cc:683] Acquiring locks for 1 operations
0130 02:58:11.792889 (+ 141us) tablet.cc:699] Row locks acquired
0130 02:58:11.792903 (+ 14us) write_op.cc:260] PREPARE: finished
0130 02:58:11.793019 (+ 116us) write_op.cc:270] Start()
0130 02:58:11.793100 (+ 81us) write_op.cc:276] Timestamp: P: 1769741891792959 usec, L: 0
0130 02:58:11.793115 (+ 15us) op_driver.cc:348] REPLICATION: starting
0130 02:58:11.793580 (+ 465us) log.cc:844] Serialized 533 byte log entry
0130 02:58:11.814171 (+ 20591us) op_driver.cc:464] REPLICATION: finished
0130 02:58:11.814454 (+ 283us) write_op.cc:301] APPLY: starting
0130 02:58:11.814509 (+ 55us) tablet.cc:1365] starting BulkCheckPresence
0130 02:58:11.814614 (+ 105us) tablet.cc:1368] finished BulkCheckPresence
0130 02:58:11.814725 (+ 111us) tablet.cc:1370] starting ApplyRowOperation cycle
0130 02:58:11.814895 (+ 170us) tablet.cc:1381] finished ApplyRowOperation cycle
0130 02:58:11.815023 (+ 128us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0130 02:58:11.815063 (+ 40us) write_op.cc:312] APPLY: finished
0130 02:58:11.815236 (+ 173us) log.cc:844] Serialized 23 byte log entry
0130 02:58:11.815369 (+ 133us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:58:11.815432 (+ 63us) write_op.cc:454] Released schema lock
0130 02:58:11.815463 (+ 31us) write_op.cc:341] FINISH: Updating metrics
0130 02:58:11.815676 (+ 213us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:58:11.815697 (+ 21us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":106150,"reported_tablets":36,"child_traces":[["op",{"apply.queue_time_us":149,"apply.run_cpu_time_us":1187,"apply.run_wall_time_us":1186,"num_ops":1,"prepare.queue_time_us":69690,"prepare.run_cpu_time_us":1775,"prepare.run_wall_time_us":1775,"raft.queue_time_us":937,"raft.run_cpu_time_us":753,"raft.run_wall_time_us":983,"replication_time_us":20963,"spinlock_wait_cycles":795008}]]}
I20260130 02:58:13.563133 20538 raft_consensus.cc:493] T 274530359b174ff79f046c2d4a6afa76 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:13.563232 20479 raft_consensus.cc:493] T 03025d13f0e545cfa6288677a2715343 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:13.563612 20479 raft_consensus.cc:515] T 03025d13f0e545cfa6288677a2715343 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:13.565374 20479 leader_election.cc:290] T 03025d13f0e545cfa6288677a2715343 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), d870164263bb44b6b780a557daea3656 (127.18.100.67:39571)
I20260130 02:58:13.475426 19350 tablet_service.cc:1505] Processing CreateTablet for tablet 0855d88806914a36b4bf6b49036f0ad0 (DEFAULT_TABLE table=default.Table_6773a0ad5d6a429086a5818d731f3f4a [id=90b72499c7174fd580914d7fe3714891]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:13.566529 19350 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0855d88806914a36b4bf6b49036f0ad0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:13.567044 19525 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "03025d13f0e545cfa6288677a2715343" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
W20260130 02:58:13.568506 19313 leader_election.cc:343] T 03025d13f0e545cfa6288677a2715343 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:13.548722 19314 leader_election.cc:304] T 4907294442c145c6b03eb1de057c66a8 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 371a36d219164974abdf848841528152, 78c2c46c7520492db35d51c244fb4f74; no voters: d870164263bb44b6b780a557daea3656
W20260130 02:58:13.571339 19314 leader_election.cc:343] T 5fbe92a2227d44d88b4531a7f8aa0e08 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:13.571671 19314 leader_election.cc:304] T 5fbe92a2227d44d88b4531a7f8aa0e08 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78c2c46c7520492db35d51c244fb4f74; no voters: 371a36d219164974abdf848841528152, d870164263bb44b6b780a557daea3656
I20260130 02:58:13.572360 20474 raft_consensus.cc:2804] T 4907294442c145c6b03eb1de057c66a8 P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Leader election won for term 1
I20260130 02:58:13.572795 20474 raft_consensus.cc:697] T 4907294442c145c6b03eb1de057c66a8 P 78c2c46c7520492db35d51c244fb4f74 [term 1 LEADER]: Becoming Leader. State: Replica: 78c2c46c7520492db35d51c244fb4f74, State: Running, Role: LEADER
I20260130 02:58:13.573185 19351 tablet_service.cc:1552] Processing DeleteTablet for tablet 6573198830ad45628e71952a7d9c00a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:58:03 UTC) from {username='slave'} at 127.0.0.1:60110
W20260130 02:58:13.451864 20568 tablet_replica.cc:1410] Time spent flushing the Log queue took a long time: real 0.263s user 0.001s sys 0.000s
I20260130 02:58:13.573509 20474 consensus_queue.cc:237] T 4907294442c145c6b03eb1de057c66a8 P 78c2c46c7520492db35d51c244fb4f74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:13.575629 20544 raft_consensus.cc:2749] T 5fbe92a2227d44d88b4531a7f8aa0e08 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260130 02:58:13.576524 20562 raft_consensus.cc:493] T 345eaaa5625c48d099280b7fbb2e514b P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:13.576943 20562 raft_consensus.cc:515] T 345eaaa5625c48d099280b7fbb2e514b P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:13.577346 20581 ts_tablet_manager.cc:1929] T e364f4e70e5749e083e14e7fa7416ff0 P 371a36d219164974abdf848841528152: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260130 02:58:13.577674 20581 log.cc:1199] T e364f4e70e5749e083e14e7fa7416ff0 P 371a36d219164974abdf848841528152: Deleting WAL directory at /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-0/wal/wals/e364f4e70e5749e083e14e7fa7416ff0
I20260130 02:58:13.578402 20581 ts_tablet_manager.cc:1950] T e364f4e70e5749e083e14e7fa7416ff0 P 371a36d219164974abdf848841528152: Deleting consensus metadata
I20260130 02:58:13.563570 20538 raft_consensus.cc:515] T 274530359b174ff79f046c2d4a6afa76 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:13.579831 20538 leader_election.cc:290] T 274530359b174ff79f046c2d4a6afa76 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), d870164263bb44b6b780a557daea3656 (127.18.100.67:39571)
I20260130 02:58:13.580996 20562 leader_election.cc:290] T 345eaaa5625c48d099280b7fbb2e514b P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767)
I20260130 02:58:13.583491 19375 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "345eaaa5625c48d099280b7fbb2e514b" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74" is_pre_election: true
I20260130 02:58:13.584097 19375 raft_consensus.cc:2468] T 345eaaa5625c48d099280b7fbb2e514b P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d870164263bb44b6b780a557daea3656 in term 0.
I20260130 02:58:13.585207 19462 leader_election.cc:304] T 345eaaa5625c48d099280b7fbb2e514b P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78c2c46c7520492db35d51c244fb4f74, d870164263bb44b6b780a557daea3656; no voters:
I20260130 02:58:13.586835 19525 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "274530359b174ff79f046c2d4a6afa76" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
I20260130 02:58:13.589788 20530 raft_consensus.cc:2804] T 345eaaa5625c48d099280b7fbb2e514b P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260130 02:58:13.590121 20530 raft_consensus.cc:493] T 345eaaa5625c48d099280b7fbb2e514b P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260130 02:58:13.590408 20530 raft_consensus.cc:3060] T 345eaaa5625c48d099280b7fbb2e514b P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:13.591576 19221 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "345eaaa5625c48d099280b7fbb2e514b" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152" is_pre_election: true
W20260130 02:58:13.592895 19463 leader_election.cc:343] T 345eaaa5625c48d099280b7fbb2e514b P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Illegal state: must be running to vote when last-logged opid is not known
W20260130 02:58:13.595099 19313 leader_election.cc:343] T 274530359b174ff79f046c2d4a6afa76 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:13.596468 20581 tablet_replica.cc:333] T 3b148691090e4ee581d46f9845a6e49a P 371a36d219164974abdf848841528152: stopping tablet replica
I20260130 02:58:13.596899 18861 catalog_manager.cc:4985] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021): tablet e364f4e70e5749e083e14e7fa7416ff0 (table default.Table_bc11cb1bce0348ed9ea0912b3ee521b9 [id=df7d3d68e36142ff8564e63ffe0b8797]) successfully deleted
I20260130 02:58:13.597473 20581 raft_consensus.cc:2243] T 3b148691090e4ee581d46f9845a6e49a P 371a36d219164974abdf848841528152 [term 1 LEADER]: Raft consensus shutting down.
I20260130 02:58:13.598238 20581 raft_consensus.cc:2272] T 3b148691090e4ee581d46f9845a6e49a P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260130 02:58:13.603721 19225 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "03025d13f0e545cfa6288677a2715343" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152" is_pre_election: true
I20260130 02:58:13.605948 19224 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "274530359b174ff79f046c2d4a6afa76" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152" is_pre_election: true
I20260130 02:58:13.606981 20581 ts_tablet_manager.cc:1916] T 3b148691090e4ee581d46f9845a6e49a P 371a36d219164974abdf848841528152: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260130 02:58:13.608532 19462 leader_election.cc:343] T f54addb1a4304ce69c711ae0cc3b452c P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767): Not found: Tablet not found: f54addb1a4304ce69c711ae0cc3b452c
I20260130 02:58:13.608922 19462 leader_election.cc:304] T f54addb1a4304ce69c711ae0cc3b452c P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d870164263bb44b6b780a557daea3656; no voters: 371a36d219164974abdf848841528152, 78c2c46c7520492db35d51c244fb4f74
I20260130 02:58:13.610260 20562 raft_consensus.cc:2749] T f54addb1a4304ce69c711ae0cc3b452c P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260130 02:58:13.612185 19314 leader_election.cc:343] T 03025d13f0e545cfa6288677a2715343 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:13.612581 19314 leader_election.cc:304] T 03025d13f0e545cfa6288677a2715343 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78c2c46c7520492db35d51c244fb4f74; no voters: 371a36d219164974abdf848841528152, d870164263bb44b6b780a557daea3656
I20260130 02:58:13.613180 20479 raft_consensus.cc:2749] T 03025d13f0e545cfa6288677a2715343 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260130 02:58:13.607103 20530 raft_consensus.cc:515] T 345eaaa5625c48d099280b7fbb2e514b P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:13.622797 19625 tablet_bootstrap.cc:492] T 43cec19c51ec4f05b63800c07f9534ef P d870164263bb44b6b780a557daea3656: No bootstrap required, opened a new log
I20260130 02:58:13.624022 19225 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "345eaaa5625c48d099280b7fbb2e514b" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152"
I20260130 02:58:13.635268 19197 tablet_service.cc:1552] Processing DeleteTablet for tablet e364f4e70e5749e083e14e7fa7416ff0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:58:10 UTC) from {username='slave'} at 127.0.0.1:42870
I20260130 02:58:13.524508 20584 tablet.cc:1722] T 127a84346b9f473d9e7f739b1e65feb8 P 78c2c46c7520492db35d51c244fb4f74: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 5
I20260130 02:58:13.637287 20479 raft_consensus.cc:493] T f41f4cff7dd4489f948700a45e9a7057 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:13.638067 20479 raft_consensus.cc:515] T f41f4cff7dd4489f948700a45e9a7057 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:13.640623 20479 leader_election.cc:290] T f41f4cff7dd4489f948700a45e9a7057 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), d870164263bb44b6b780a557daea3656 (127.18.100.67:39571)
W20260130 02:58:13.643247 19314 leader_election.cc:343] T 274530359b174ff79f046c2d4a6afa76 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:13.647964 19840 tablet_bootstrap.cc:492] T 78e9bc3e55fe4e4a98e8eb01da59d365 P 78c2c46c7520492db35d51c244fb4f74: No bootstrap required, opened a new log
W20260130 02:58:13.626109 19463 leader_election.cc:343] T 345eaaa5625c48d099280b7fbb2e514b P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:13.648360 19840 ts_tablet_manager.cc:1403] T 78e9bc3e55fe4e4a98e8eb01da59d365 P 78c2c46c7520492db35d51c244fb4f74: Time spent bootstrapping tablet: real 0.325s user 0.028s sys 0.006s
I20260130 02:58:13.642100 19525 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f41f4cff7dd4489f948700a45e9a7057" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
I20260130 02:58:13.625732 19521 raft_consensus.cc:3060] T 16691c9315e94253bb9b3fb5058e5636 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:13.651913 19197 tablet_service.cc:1552] Processing DeleteTablet for tablet f54addb1a4304ce69c711ae0cc3b452c with delete_type TABLET_DATA_DELETED (replaced by tablet 7783db3568f4434e91e9e456799cc312) from {username='slave'} at 127.0.0.1:42870
I20260130 02:58:13.650497 19840 raft_consensus.cc:359] T 78e9bc3e55fe4e4a98e8eb01da59d365 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:13.653652 19840 raft_consensus.cc:385] T 78e9bc3e55fe4e4a98e8eb01da59d365 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:13.653946 19840 raft_consensus.cc:740] T 78e9bc3e55fe4e4a98e8eb01da59d365 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78c2c46c7520492db35d51c244fb4f74, State: Initialized, Role: FOLLOWER
I20260130 02:58:13.623193 19625 ts_tablet_manager.cc:1403] T 43cec19c51ec4f05b63800c07f9534ef P d870164263bb44b6b780a557daea3656: Time spent bootstrapping tablet: real 0.287s user 0.009s sys 0.000s
I20260130 02:58:13.654659 19840 consensus_queue.cc:260] T 78e9bc3e55fe4e4a98e8eb01da59d365 P 78c2c46c7520492db35d51c244fb4f74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:13.655839 20562 raft_consensus.cc:493] T b2ac91da01954c2390b6b7dcc06c394f P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 371a36d219164974abdf848841528152)
I20260130 02:58:13.656093 19197 tablet_service.cc:1552] Processing DeleteTablet for tablet 730bd1d7d15b4cbe945b3ef03b2455f7 with delete_type TABLET_DATA_DELETED (replaced by tablet 3b02480d9c6f4328af83f7940716d698) from {username='slave'} at 127.0.0.1:42870
I20260130 02:58:13.656277 20562 raft_consensus.cc:515] T b2ac91da01954c2390b6b7dcc06c394f P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:13.657574 19840 ts_tablet_manager.cc:1434] T 78e9bc3e55fe4e4a98e8eb01da59d365 P 78c2c46c7520492db35d51c244fb4f74: Time spent starting tablet: real 0.009s user 0.005s sys 0.000s
I20260130 02:58:13.658448 19840 tablet_bootstrap.cc:492] T e2aaaf00f53d418b87d5c7af286e4d84 P 78c2c46c7520492db35d51c244fb4f74: Bootstrap starting.
I20260130 02:58:13.660434 20562 leader_election.cc:290] T b2ac91da01954c2390b6b7dcc06c394f P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767)
I20260130 02:58:13.663371 19496 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0855d88806914a36b4bf6b49036f0ad0. 1 dirs total, 0 dirs full, 0 dirs failed
W20260130 02:58:13.650490 19313 leader_election.cc:343] T f41f4cff7dd4489f948700a45e9a7057 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:13.669435 19502 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1fc8f09d6329407c8078542b4123ef4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:13.674925 20538 raft_consensus.cc:493] T b1e8b78eff0d4a0b92059adb2de4a3ce P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:13.675410 20538 raft_consensus.cc:515] T b1e8b78eff0d4a0b92059adb2de4a3ce P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:13.677296 20538 leader_election.cc:290] T b1e8b78eff0d4a0b92059adb2de4a3ce P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), d870164263bb44b6b780a557daea3656 (127.18.100.67:39571)
I20260130 02:58:13.686944 19378 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "345eaaa5625c48d099280b7fbb2e514b" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74"
I20260130 02:58:13.687608 19378 raft_consensus.cc:3060] T 345eaaa5625c48d099280b7fbb2e514b P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:13.690923 19840 tablet_bootstrap.cc:654] T e2aaaf00f53d418b87d5c7af286e4d84 P 78c2c46c7520492db35d51c244fb4f74: Neither blocks nor log segments found. Creating new log.
W20260130 02:58:13.693331 19462 leader_election.cc:343] T b2ac91da01954c2390b6b7dcc06c394f P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767): Not found: Tablet not found: b2ac91da01954c2390b6b7dcc06c394f
I20260130 02:58:13.697682 20571 raft_consensus.cc:2243] T 3b148691090e4ee581d46f9845a6e49a P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260130 02:58:13.698290 20571 raft_consensus.cc:2272] T 3b148691090e4ee581d46f9845a6e49a P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260130 02:58:13.700318 20571 ts_tablet_manager.cc:1916] T 3b148691090e4ee581d46f9845a6e49a P d870164263bb44b6b780a557daea3656: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260130 02:58:13.706166 19463 leader_election.cc:343] T b2ac91da01954c2390b6b7dcc06c394f P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Not found: Tablet not found: b2ac91da01954c2390b6b7dcc06c394f
I20260130 02:58:13.706200 20581 ts_tablet_manager.cc:1929] T 3b148691090e4ee581d46f9845a6e49a P 371a36d219164974abdf848841528152: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260130 02:58:13.706633 20581 log.cc:1199] T 3b148691090e4ee581d46f9845a6e49a P 371a36d219164974abdf848841528152: Deleting WAL directory at /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-0/wal/wals/3b148691090e4ee581d46f9845a6e49a
I20260130 02:58:13.706713 19463 leader_election.cc:304] T b2ac91da01954c2390b6b7dcc06c394f P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d870164263bb44b6b780a557daea3656; no voters: 371a36d219164974abdf848841528152, 78c2c46c7520492db35d51c244fb4f74
I20260130 02:58:13.707734 20562 raft_consensus.cc:2749] T b2ac91da01954c2390b6b7dcc06c394f P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260130 02:58:13.643589 19314 leader_election.cc:304] T 274530359b174ff79f046c2d4a6afa76 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78c2c46c7520492db35d51c244fb4f74; no voters: 371a36d219164974abdf848841528152, d870164263bb44b6b780a557daea3656
I20260130 02:58:13.721957 20538 raft_consensus.cc:2749] T 274530359b174ff79f046c2d4a6afa76 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260130 02:58:13.722910 19314 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.100.65:42021, user_credentials={real_user=slave}} blocked reactor thread for 79681.8us
I20260130 02:58:13.714994 20581 ts_tablet_manager.cc:1950] T 3b148691090e4ee581d46f9845a6e49a P 371a36d219164974abdf848841528152: Deleting consensus metadata
I20260130 02:58:13.737115 19807 tablet_bootstrap.cc:492] T 345eaaa5625c48d099280b7fbb2e514b P 371a36d219164974abdf848841528152: Bootstrap starting.
I20260130 02:58:13.739688 20530 leader_election.cc:290] T 345eaaa5625c48d099280b7fbb2e514b P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 election: Requested vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767)
I20260130 02:58:13.742007 19807 tablet_bootstrap.cc:654] T 345eaaa5625c48d099280b7fbb2e514b P 371a36d219164974abdf848841528152: Neither blocks nor log segments found. Creating new log.
W20260130 02:58:13.708556 19159 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.100.67:39571, user_credentials={real_user=slave}} blocked reactor thread for 55110.3us
I20260130 02:58:13.736325 19197 tablet_service.cc:1552] Processing DeleteTablet for tablet 7b80b22bd44c4233837c951aa97e6996 with delete_type TABLET_DATA_DELETED (replaced by tablet 1fc8f09d6329407c8078542b4123ef4d) from {username='slave'} at 127.0.0.1:42870
I20260130 02:58:13.745438 19840 tablet_bootstrap.cc:492] T e2aaaf00f53d418b87d5c7af286e4d84 P 78c2c46c7520492db35d51c244fb4f74: No bootstrap required, opened a new log
I20260130 02:58:13.745862 19840 ts_tablet_manager.cc:1403] T e2aaaf00f53d418b87d5c7af286e4d84 P 78c2c46c7520492db35d51c244fb4f74: Time spent bootstrapping tablet: real 0.088s user 0.011s sys 0.004s
I20260130 02:58:13.750744 19625 raft_consensus.cc:359] T 43cec19c51ec4f05b63800c07f9534ef P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:13.751400 19625 raft_consensus.cc:385] T 43cec19c51ec4f05b63800c07f9534ef P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:13.751766 19625 raft_consensus.cc:740] T 43cec19c51ec4f05b63800c07f9534ef P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d870164263bb44b6b780a557daea3656, State: Initialized, Role: FOLLOWER
I20260130 02:58:13.752408 19625 consensus_queue.cc:260] T 43cec19c51ec4f05b63800c07f9534ef P d870164263bb44b6b780a557daea3656 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:13.756162 19528 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b1e8b78eff0d4a0b92059adb2de4a3ce" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
I20260130 02:58:13.759351 18861 catalog_manager.cc:4985] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021): tablet 3b148691090e4ee581d46f9845a6e49a (table default.Table_8edea1fae34d4d18b54b0f801a145225 [id=2009496525c94d7f8ee6ce59694b652d]) successfully deleted
W20260130 02:58:13.761778 18861 catalog_manager.cc:4952] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021): delete failed for tablet e364f4e70e5749e083e14e7fa7416ff0 because the tablet was not found. No further retry: Not found: Tablet not found: e364f4e70e5749e083e14e7fa7416ff0
I20260130 02:58:13.764900 20498 consensus_queue.cc:1048] T a8a5038e9def44eb8b3b1e4aeae25652 P 371a36d219164974abdf848841528152 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260130 02:58:13.767946 18861 catalog_manager.cc:4952] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021): delete failed for tablet f54addb1a4304ce69c711ae0cc3b452c because the tablet was not found. No further retry: Not found: Tablet not found: f54addb1a4304ce69c711ae0cc3b452c
I20260130 02:58:13.773023 19378 raft_consensus.cc:2468] T 345eaaa5625c48d099280b7fbb2e514b P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d870164263bb44b6b780a557daea3656 in term 1.
I20260130 02:58:13.780463 20590 raft_consensus.cc:493] T 43cec19c51ec4f05b63800c07f9534ef P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260130 02:58:13.780929 18861 catalog_manager.cc:4952] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021): delete failed for tablet 730bd1d7d15b4cbe945b3ef03b2455f7 because the tablet was not found. No further retry: Not found: Tablet not found: 730bd1d7d15b4cbe945b3ef03b2455f7
I20260130 02:58:13.780915 20590 raft_consensus.cc:515] T 43cec19c51ec4f05b63800c07f9534ef P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:13.776461 19840 raft_consensus.cc:359] T e2aaaf00f53d418b87d5c7af286e4d84 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:13.783233 19840 raft_consensus.cc:385] T e2aaaf00f53d418b87d5c7af286e4d84 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:13.783273 20590 leader_election.cc:290] T 43cec19c51ec4f05b63800c07f9534ef P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767)
I20260130 02:58:13.783540 19840 raft_consensus.cc:740] T e2aaaf00f53d418b87d5c7af286e4d84 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78c2c46c7520492db35d51c244fb4f74, State: Initialized, Role: FOLLOWER
W20260130 02:58:13.784130 18861 catalog_manager.cc:4952] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021): delete failed for tablet 7b80b22bd44c4233837c951aa97e6996 because the tablet was not found. No further retry: Not found: Tablet not found: 7b80b22bd44c4233837c951aa97e6996
I20260130 02:58:13.784138 19840 consensus_queue.cc:260] T e2aaaf00f53d418b87d5c7af286e4d84 P 78c2c46c7520492db35d51c244fb4f74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:13.785368 19197 tablet_service.cc:1552] Processing DeleteTablet for tablet 6573198830ad45628e71952a7d9c00a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:58:03 UTC) from {username='slave'} at 127.0.0.1:42870
I20260130 02:58:13.785714 19225 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f41f4cff7dd4489f948700a45e9a7057" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152" is_pre_election: true
I20260130 02:58:13.786796 19224 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b1e8b78eff0d4a0b92059adb2de4a3ce" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152" is_pre_election: true
I20260130 02:58:13.785677 19840 ts_tablet_manager.cc:1434] T e2aaaf00f53d418b87d5c7af286e4d84 P 78c2c46c7520492db35d51c244fb4f74: Time spent starting tablet: real 0.039s user 0.003s sys 0.004s
I20260130 02:58:13.794876 19807 tablet_bootstrap.cc:492] T 345eaaa5625c48d099280b7fbb2e514b P 371a36d219164974abdf848841528152: No bootstrap required, opened a new log
I20260130 02:58:13.795321 19807 ts_tablet_manager.cc:1403] T 345eaaa5625c48d099280b7fbb2e514b P 371a36d219164974abdf848841528152: Time spent bootstrapping tablet: real 0.058s user 0.007s sys 0.009s
I20260130 02:58:13.797636 19807 raft_consensus.cc:359] T 345eaaa5625c48d099280b7fbb2e514b P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:13.798255 19807 raft_consensus.cc:385] T 345eaaa5625c48d099280b7fbb2e514b P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:13.798512 19807 raft_consensus.cc:740] T 345eaaa5625c48d099280b7fbb2e514b P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 371a36d219164974abdf848841528152, State: Initialized, Role: FOLLOWER
I20260130 02:58:13.799328 19807 consensus_queue.cc:260] T 345eaaa5625c48d099280b7fbb2e514b P 371a36d219164974abdf848841528152 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
W20260130 02:58:13.800619 19314 leader_election.cc:343] T f41f4cff7dd4489f948700a45e9a7057 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:13.800904 19222 raft_consensus.cc:3060] T 3389db58493448aea3cf00aca411047c P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:13.801039 19314 leader_election.cc:304] T f41f4cff7dd4489f948700a45e9a7057 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78c2c46c7520492db35d51c244fb4f74; no voters: 371a36d219164974abdf848841528152, d870164263bb44b6b780a557daea3656
I20260130 02:58:13.801295 19807 ts_tablet_manager.cc:1434] T 345eaaa5625c48d099280b7fbb2e514b P 371a36d219164974abdf848841528152: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20260130 02:58:13.801687 20538 raft_consensus.cc:2749] T f41f4cff7dd4489f948700a45e9a7057 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260130 02:58:13.802162 19807 tablet_bootstrap.cc:492] T ba91a508b1cb4430a4d8ca26946aecc8 P 371a36d219164974abdf848841528152: Bootstrap starting.
W20260130 02:58:13.779125 19313 leader_election.cc:343] T b1e8b78eff0d4a0b92059adb2de4a3ce P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:13.806146 19462 leader_election.cc:304] T 345eaaa5625c48d099280b7fbb2e514b P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 78c2c46c7520492db35d51c244fb4f74, d870164263bb44b6b780a557daea3656; no voters: 371a36d219164974abdf848841528152
W20260130 02:58:13.816823 18870 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.18.100.65:51021 (request call id 33) took 3057 ms (client timeout 1000 ms). Trace:
I20260130 02:58:13.823084 19523 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3389db58493448aea3cf00aca411047c" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
I20260130 02:58:13.823858 19523 raft_consensus.cc:2393] T 3389db58493448aea3cf00aca411047c P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 371a36d219164974abdf848841528152 in current term 1: Already voted for candidate 78c2c46c7520492db35d51c244fb4f74 in this term.
I20260130 02:58:13.825809 19625 ts_tablet_manager.cc:1434] T 43cec19c51ec4f05b63800c07f9534ef P d870164263bb44b6b780a557daea3656: Time spent starting tablet: real 0.171s user 0.003s sys 0.003s
W20260130 02:58:13.825773 18861 catalog_manager.cc:4952] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021): delete failed for tablet 6573198830ad45628e71952a7d9c00a0 because the tablet was not found. No further retry: Not found: Tablet not found: 6573198830ad45628e71952a7d9c00a0
W20260130 02:58:13.826742 19314 leader_election.cc:343] T b1e8b78eff0d4a0b92059adb2de4a3ce P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:13.827121 19314 leader_election.cc:304] T b1e8b78eff0d4a0b92059adb2de4a3ce P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78c2c46c7520492db35d51c244fb4f74; no voters: 371a36d219164974abdf848841528152, d870164263bb44b6b780a557daea3656
I20260130 02:58:13.827744 20538 raft_consensus.cc:2749] T b1e8b78eff0d4a0b92059adb2de4a3ce P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260130 02:58:13.828555 19807 tablet_bootstrap.cc:654] T ba91a508b1cb4430a4d8ca26946aecc8 P 371a36d219164974abdf848841528152: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:13.834087 19376 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "43cec19c51ec4f05b63800c07f9534ef" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74" is_pre_election: true
I20260130 02:58:13.839010 19840 tablet_bootstrap.cc:492] T f9e5f01e69254d658c369f3e21cef523 P 78c2c46c7520492db35d51c244fb4f74: Bootstrap starting.
I20260130 02:58:13.838833 19380 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3389db58493448aea3cf00aca411047c" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74" is_pre_election: true
I20260130 02:58:13.845230 19840 tablet_bootstrap.cc:654] T f9e5f01e69254d658c369f3e21cef523 P 78c2c46c7520492db35d51c244fb4f74: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:13.854121 19197 tablet_service.cc:1552] Processing DeleteTablet for tablet b2ac91da01954c2390b6b7dcc06c394f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:58:04 UTC) from {username='slave'} at 127.0.0.1:42870
I20260130 02:58:13.855424 20538 consensus_queue.cc:1048] T 9fe0448bcc4a4346b3c67fc3f626ed3c P 78c2c46c7520492db35d51c244fb4f74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260130 02:58:13.860033 20474 consensus_queue.cc:1048] T 3389db58493448aea3cf00aca411047c P 78c2c46c7520492db35d51c244fb4f74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260130 02:58:13.861349 19204 tablet_service.cc:1552] Processing DeleteTablet for tablet 7d96e93d95f14af79cfb268626559b1e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:58:04 UTC) from {username='slave'} at 127.0.0.1:42870
I20260130 02:58:13.864281 19807 tablet_bootstrap.cc:492] T ba91a508b1cb4430a4d8ca26946aecc8 P 371a36d219164974abdf848841528152: No bootstrap required, opened a new log
I20260130 02:58:13.867374 19222 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "43cec19c51ec4f05b63800c07f9534ef" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152" is_pre_election: true
I20260130 02:58:13.868275 19807 ts_tablet_manager.cc:1403] T ba91a508b1cb4430a4d8ca26946aecc8 P 371a36d219164974abdf848841528152: Time spent bootstrapping tablet: real 0.066s user 0.009s sys 0.001s
I20260130 02:58:13.870597 19807 raft_consensus.cc:359] T ba91a508b1cb4430a4d8ca26946aecc8 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:13.871198 19807 raft_consensus.cc:385] T ba91a508b1cb4430a4d8ca26946aecc8 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:13.871452 19807 raft_consensus.cc:740] T ba91a508b1cb4430a4d8ca26946aecc8 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 371a36d219164974abdf848841528152, State: Initialized, Role: FOLLOWER
I20260130 02:58:13.871985 19807 consensus_queue.cc:260] T ba91a508b1cb4430a4d8ca26946aecc8 P 371a36d219164974abdf848841528152 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:13.887228 19840 tablet_bootstrap.cc:492] T f9e5f01e69254d658c369f3e21cef523 P 78c2c46c7520492db35d51c244fb4f74: No bootstrap required, opened a new log
I20260130 02:58:13.887635 19840 ts_tablet_manager.cc:1403] T f9e5f01e69254d658c369f3e21cef523 P 78c2c46c7520492db35d51c244fb4f74: Time spent bootstrapping tablet: real 0.049s user 0.002s sys 0.008s
I20260130 02:58:13.890232 19840 raft_consensus.cc:359] T f9e5f01e69254d658c369f3e21cef523 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:13.894892 19840 raft_consensus.cc:385] T f9e5f01e69254d658c369f3e21cef523 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:13.895148 19840 raft_consensus.cc:740] T f9e5f01e69254d658c369f3e21cef523 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78c2c46c7520492db35d51c244fb4f74, State: Initialized, Role: FOLLOWER
I20260130 02:58:13.895712 19840 consensus_queue.cc:260] T f9e5f01e69254d658c369f3e21cef523 P 78c2c46c7520492db35d51c244fb4f74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:13.826349 20590 raft_consensus.cc:2804] T 345eaaa5625c48d099280b7fbb2e514b P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Leader election won for term 1
I20260130 02:58:13.897744 20590 raft_consensus.cc:697] T 345eaaa5625c48d099280b7fbb2e514b P d870164263bb44b6b780a557daea3656 [term 1 LEADER]: Becoming Leader. State: Replica: d870164263bb44b6b780a557daea3656, State: Running, Role: LEADER
W20260130 02:58:13.817425 18870 rpcz_store.cc:269] 0130 02:58:10.758940 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:58:10.761892 (+ 2952us) service_pool.cc:225] Handling call
0130 02:58:13.816752 (+3054860us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0130 02:58:12.219476 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0130 02:58:12.219850 (+ 374us) write_op.cc:432] Acquiring schema lock in shared mode
0130 02:58:12.219872 (+ 22us) write_op.cc:435] Acquired schema lock
0130 02:58:12.219884 (+ 12us) tablet.cc:660] Decoding operations
0130 02:58:12.220294 (+ 410us) write_op.cc:620] Acquiring the partition lock for write op
0130 02:58:12.220334 (+ 40us) write_op.cc:641] Partition lock acquired for write op
0130 02:58:12.220358 (+ 24us) tablet.cc:683] Acquiring locks for 3 operations
0130 02:58:12.220590 (+ 232us) tablet.cc:699] Row locks acquired
0130 02:58:12.220603 (+ 13us) write_op.cc:260] PREPARE: finished
0130 02:58:12.220706 (+ 103us) write_op.cc:270] Start()
0130 02:58:12.220794 (+ 88us) write_op.cc:276] Timestamp: P: 1769741892220653 usec, L: 0
0130 02:58:12.220808 (+ 14us) op_driver.cc:348] REPLICATION: starting
0130 02:58:12.221313 (+ 505us) log.cc:844] Serialized 1293 byte log entry
0130 02:58:12.276288 (+ 54975us) op_driver.cc:464] REPLICATION: finished
0130 02:58:12.276698 (+ 410us) write_op.cc:301] APPLY: starting
0130 02:58:12.276772 (+ 74us) tablet.cc:1365] starting BulkCheckPresence
0130 02:58:12.276892 (+ 120us) tablet.cc:1368] finished BulkCheckPresence
0130 02:58:12.276987 (+ 95us) tablet.cc:1370] starting ApplyRowOperation cycle
0130 02:58:12.277311 (+ 324us) tablet.cc:1381] finished ApplyRowOperation cycle
0130 02:58:12.277459 (+ 148us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=3
0130 02:58:12.277501 (+ 42us) write_op.cc:312] APPLY: finished
0130 02:58:12.277718 (+ 217us) log.cc:844] Serialized 35 byte log entry
0130 02:58:12.277870 (+ 152us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:58:12.277956 (+ 86us) write_op.cc:454] Released schema lock
0130 02:58:12.277993 (+ 37us) write_op.cc:341] FINISH: Updating metrics
0130 02:58:12.278250 (+ 257us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:58:12.278275 (+ 25us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":21591,"reported_tablets":39,"child_traces":[["op",{"apply.queue_time_us":265,"apply.run_cpu_time_us":1499,"apply.run_wall_time_us":1499,"num_ops":3,"prepare.queue_time_us":84,"prepare.run_cpu_time_us":2088,"prepare.run_wall_time_us":2089,"raft.queue_time_us":2239,"raft.run_cpu_time_us":562,"raft.run_wall_time_us":563,"replication_time_us":55377}]]}
I20260130 02:58:13.903023 20590 consensus_queue.cc:237] T 345eaaa5625c48d099280b7fbb2e514b P d870164263bb44b6b780a557daea3656 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:13.897404 20538 raft_consensus.cc:493] T 78e9bc3e55fe4e4a98e8eb01da59d365 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:13.911871 20538 raft_consensus.cc:515] T 78e9bc3e55fe4e4a98e8eb01da59d365 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:13.912273 20571 ts_tablet_manager.cc:1929] T 3b148691090e4ee581d46f9845a6e49a P d870164263bb44b6b780a557daea3656: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260130 02:58:13.912480 19840 ts_tablet_manager.cc:1434] T f9e5f01e69254d658c369f3e21cef523 P 78c2c46c7520492db35d51c244fb4f74: Time spent starting tablet: real 0.025s user 0.007s sys 0.000s
I20260130 02:58:13.912676 20571 log.cc:1199] T 3b148691090e4ee581d46f9845a6e49a P d870164263bb44b6b780a557daea3656: Deleting WAL directory at /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-2/wal/wals/3b148691090e4ee581d46f9845a6e49a
I20260130 02:58:13.913440 19840 tablet_bootstrap.cc:492] T cf1b66a20fce4c4d96bc0bc32a7bafa8 P 78c2c46c7520492db35d51c244fb4f74: Bootstrap starting.
I20260130 02:58:13.913530 20571 ts_tablet_manager.cc:1950] T 3b148691090e4ee581d46f9845a6e49a P d870164263bb44b6b780a557daea3656: Deleting consensus metadata
I20260130 02:58:13.914011 20538 leader_election.cc:290] T 78e9bc3e55fe4e4a98e8eb01da59d365 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), d870164263bb44b6b780a557daea3656 (127.18.100.67:39571)
I20260130 02:58:13.921459 19807 ts_tablet_manager.cc:1434] T ba91a508b1cb4430a4d8ca26946aecc8 P 371a36d219164974abdf848841528152: Time spent starting tablet: real 0.053s user 0.004s sys 0.000s
I20260130 02:58:13.922297 19807 tablet_bootstrap.cc:492] T 0e8018c094024c4788c484e8f5466e00 P 371a36d219164974abdf848841528152: Bootstrap starting.
I20260130 02:58:13.931403 19625 tablet_bootstrap.cc:492] T 4c139aa8cd184f488cec4e05ef7dcada P d870164263bb44b6b780a557daea3656: Bootstrap starting.
I20260130 02:58:13.931779 19526 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "78e9bc3e55fe4e4a98e8eb01da59d365" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
I20260130 02:58:13.945304 18867 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:50284 (request call id 229) took 2456 ms. Trace:
I20260130 02:58:13.948786 18875 catalog_manager.cc:5941] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4: Sending DeleteTablet for 3 replicas of tablet d55291d7cea64a0d9a345ce712fbfa80
I20260130 02:58:13.951154 19355 tablet_service.cc:1552] Processing DeleteTablet for tablet d55291d7cea64a0d9a345ce712fbfa80 with delete_type TABLET_DATA_DELETED (replaced by tablet cd30d7adcc4545848e3501a205a10293) from {username='slave'} at 127.0.0.1:60110
I20260130 02:58:13.952119 18875 catalog_manager.cc:6724] ReplaceTablet: tablet d55291d7cea64a0d9a345ce712fbfa80 deleted and replaced by tablet cd30d7adcc4545848e3501a205a10293
I20260130 02:58:13.952190 19496 tablet_service.cc:1552] Processing DeleteTablet for tablet d55291d7cea64a0d9a345ce712fbfa80 with delete_type TABLET_DATA_DELETED (replaced by tablet cd30d7adcc4545848e3501a205a10293) from {username='slave'} at 127.0.0.1:59068
I20260130 02:58:13.952507 18875 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:50284 (request call id 230) took 2313 ms. Trace:
I20260130 02:58:13.955986 19807 tablet_bootstrap.cc:654] T 0e8018c094024c4788c484e8f5466e00 P 371a36d219164974abdf848841528152: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:13.952806 18875 rpcz_store.cc:276] 0130 02:58:11.639079 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:58:13.513547 (+1874468us) service_pool.cc:225] Handling call
0130 02:58:13.952454 (+438907us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0130 02:58:13.798625 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0130 02:58:13.812010 (+ 13385us) write_op.cc:432] Acquiring schema lock in shared mode
0130 02:58:13.812040 (+ 30us) write_op.cc:435] Acquired schema lock
0130 02:58:13.812055 (+ 15us) tablet.cc:660] Decoding operations
0130 02:58:13.812433 (+ 378us) write_op.cc:620] Acquiring the partition lock for write op
0130 02:58:13.812479 (+ 46us) write_op.cc:641] Partition lock acquired for write op
0130 02:58:13.812505 (+ 26us) tablet.cc:683] Acquiring locks for 2 operations
0130 02:58:13.812717 (+ 212us) tablet.cc:699] Row locks acquired
0130 02:58:13.812734 (+ 17us) write_op.cc:260] PREPARE: finished
0130 02:58:13.812866 (+ 132us) write_op.cc:270] Start()
0130 02:58:13.812960 (+ 94us) write_op.cc:276] Timestamp: P: 1769741893812800 usec, L: 0
0130 02:58:13.812974 (+ 14us) op_driver.cc:348] REPLICATION: starting
0130 02:58:13.813555 (+ 581us) log.cc:844] Serialized 662 byte log entry
0130 02:58:13.845411 (+ 31856us) op_driver.cc:464] REPLICATION: finished
0130 02:58:13.871104 (+ 25693us) write_op.cc:301] APPLY: starting
0130 02:58:13.871266 (+ 162us) tablet.cc:1365] starting BulkCheckPresence
0130 02:58:13.878008 (+ 6742us) tablet.cc:1368] finished BulkCheckPresence
0130 02:58:13.878105 (+ 97us) tablet.cc:1370] starting ApplyRowOperation cycle
0130 02:58:13.878443 (+ 338us) tablet.cc:1381] finished ApplyRowOperation cycle
0130 02:58:13.878656 (+ 213us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0130 02:58:13.878698 (+ 42us) write_op.cc:312] APPLY: finished
0130 02:58:13.878866 (+ 168us) log.cc:844] Serialized 29 byte log entry
0130 02:58:13.880028 (+ 1162us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:58:13.880110 (+ 82us) write_op.cc:454] Released schema lock
0130 02:58:13.880143 (+ 33us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":25570,"apply.run_cpu_time_us":1616,"apply.run_wall_time_us":9221,"num_ops":2,"prepare.queue_time_us":59400,"prepare.run_cpu_time_us":2431,"prepare.run_wall_time_us":28382,"raft.queue_time_us":84,"raft.run_cpu_time_us":12,"raft.run_wall_time_us":13,"replication_time_us":32349,"thread_start_us":61762,"threads_started":1}]]}
I20260130 02:58:13.957927 18875 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50284:
name: "default.Table_4ee03db7d0064d96a53d4f261bd12722"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260130 02:58:13.961249 20570 ts_tablet_manager.cc:1929] T 3b148691090e4ee581d46f9845a6e49a P 78c2c46c7520492db35d51c244fb4f74: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260130 02:58:13.964849 19204 tablet_service.cc:1552] Processing DeleteTablet for tablet d55291d7cea64a0d9a345ce712fbfa80 with delete_type TABLET_DATA_DELETED (replaced by tablet cd30d7adcc4545848e3501a205a10293) from {username='slave'} at 127.0.0.1:42870
I20260130 02:58:13.965564 20581 tablet_replica.cc:333] T d55291d7cea64a0d9a345ce712fbfa80 P 371a36d219164974abdf848841528152: stopping tablet replica
I20260130 02:58:13.965745 19625 tablet_bootstrap.cc:654] T 4c139aa8cd184f488cec4e05ef7dcada P d870164263bb44b6b780a557daea3656: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:13.966079 19463 leader_election.cc:304] T 43cec19c51ec4f05b63800c07f9534ef P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d870164263bb44b6b780a557daea3656; no voters: 371a36d219164974abdf848841528152, 78c2c46c7520492db35d51c244fb4f74
I20260130 02:58:13.966404 20581 raft_consensus.cc:2243] T d55291d7cea64a0d9a345ce712fbfa80 P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260130 02:58:13.966876 20581 raft_consensus.cc:2272] T d55291d7cea64a0d9a345ce712fbfa80 P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260130 02:58:13.967295 20530 raft_consensus.cc:2749] T 43cec19c51ec4f05b63800c07f9534ef P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260130 02:58:13.968806 20581 ts_tablet_manager.cc:1916] T d55291d7cea64a0d9a345ce712fbfa80 P 371a36d219164974abdf848841528152: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260130 02:58:13.976749 19807 tablet_bootstrap.cc:492] T 0e8018c094024c4788c484e8f5466e00 P 371a36d219164974abdf848841528152: No bootstrap required, opened a new log
I20260130 02:58:13.977174 19807 ts_tablet_manager.cc:1403] T 0e8018c094024c4788c484e8f5466e00 P 371a36d219164974abdf848841528152: Time spent bootstrapping tablet: real 0.055s user 0.006s sys 0.001s
W20260130 02:58:13.983197 19159 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.100.66:40767, user_credentials={real_user=slave}} blocked reactor thread for 52250.1us
I20260130 02:58:13.988139 19625 tablet_bootstrap.cc:492] T 4c139aa8cd184f488cec4e05ef7dcada P d870164263bb44b6b780a557daea3656: No bootstrap required, opened a new log
I20260130 02:58:13.988528 19625 ts_tablet_manager.cc:1403] T 4c139aa8cd184f488cec4e05ef7dcada P d870164263bb44b6b780a557daea3656: Time spent bootstrapping tablet: real 0.057s user 0.005s sys 0.003s
I20260130 02:58:13.990710 19223 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "78e9bc3e55fe4e4a98e8eb01da59d365" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152" is_pre_election: true
I20260130 02:58:13.990772 19625 raft_consensus.cc:359] T 4c139aa8cd184f488cec4e05ef7dcada P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:13.991330 19625 raft_consensus.cc:385] T 4c139aa8cd184f488cec4e05ef7dcada P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:13.991549 19625 raft_consensus.cc:740] T 4c139aa8cd184f488cec4e05ef7dcada P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d870164263bb44b6b780a557daea3656, State: Initialized, Role: FOLLOWER
I20260130 02:58:13.979368 19807 raft_consensus.cc:359] T 0e8018c094024c4788c484e8f5466e00 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:13.992270 19807 raft_consensus.cc:385] T 0e8018c094024c4788c484e8f5466e00 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:13.992533 19807 raft_consensus.cc:740] T 0e8018c094024c4788c484e8f5466e00 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 371a36d219164974abdf848841528152, State: Initialized, Role: FOLLOWER
I20260130 02:58:13.993079 19807 consensus_queue.cc:260] T 0e8018c094024c4788c484e8f5466e00 P 371a36d219164974abdf848841528152 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:13.995011 19807 ts_tablet_manager.cc:1434] T 0e8018c094024c4788c484e8f5466e00 P 371a36d219164974abdf848841528152: Time spent starting tablet: real 0.018s user 0.006s sys 0.001s
I20260130 02:58:13.995656 19625 consensus_queue.cc:260] T 4c139aa8cd184f488cec4e05ef7dcada P d870164263bb44b6b780a557daea3656 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
W20260130 02:58:13.962365 18861 catalog_manager.cc:4952] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021): delete failed for tablet b2ac91da01954c2390b6b7dcc06c394f because the tablet was not found. No further retry: Not found: Tablet not found: b2ac91da01954c2390b6b7dcc06c394f
W20260130 02:58:13.997656 18861 catalog_manager.cc:4952] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021): delete failed for tablet 7d96e93d95f14af79cfb268626559b1e because the tablet was not found. No further retry: Not found: Tablet not found: 7d96e93d95f14af79cfb268626559b1e
I20260130 02:58:13.985934 18870 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50284:
table { table_name: "default.Table_f4b80e13bbe24163a77efb7b60f5dc2c" } new_table_name: "default.Table_f17d8e4d9f25449b84aa751af93936b2" modify_external_catalogs: true
I20260130 02:58:13.945727 18867 rpcz_store.cc:276] 0130 02:58:11.489060 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:58:13.504554 (+2015494us) service_pool.cc:225] Handling call
0130 02:58:13.728056 (+223502us) catalog_manager.cc:2101] Acquired catalog manager lock
0130 02:58:13.729646 (+ 1590us) catalog_manager.cc:2147] Created new table and tablet info
0130 02:58:13.871549 (+141903us) catalog_manager.cc:2207] Wrote table and tablets to system table
0130 02:58:13.944949 (+ 73400us) catalog_manager.cc:2241] Inserted table and tablets into CatalogManager maps
0130 02:58:13.945237 (+ 288us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0130 02:58:13.731198 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0130 02:58:13.731643 (+ 445us) write_op.cc:432] Acquiring schema lock in shared mode
0130 02:58:13.731668 (+ 25us) write_op.cc:435] Acquired schema lock
0130 02:58:13.731684 (+ 16us) tablet.cc:660] Decoding operations
0130 02:58:13.731977 (+ 293us) write_op.cc:620] Acquiring the partition lock for write op
0130 02:58:13.732031 (+ 54us) write_op.cc:641] Partition lock acquired for write op
0130 02:58:13.732055 (+ 24us) tablet.cc:683] Acquiring locks for 2 operations
0130 02:58:13.732266 (+ 211us) tablet.cc:699] Row locks acquired
0130 02:58:13.732282 (+ 16us) write_op.cc:260] PREPARE: finished
0130 02:58:13.732407 (+ 125us) write_op.cc:270] Start()
0130 02:58:13.732502 (+ 95us) write_op.cc:276] Timestamp: P: 1769741893732355 usec, L: 0
0130 02:58:13.732520 (+ 18us) op_driver.cc:348] REPLICATION: starting
0130 02:58:13.770925 (+ 38405us) log.cc:844] Serialized 433 byte log entry
0130 02:58:13.773341 (+ 2416us) spinlock_profiling.cc:243] Waited 37.6 ms on lock 0x7b5400030554. stack: 00007fb0bd3f7373 00007fb0bd3f7270 00007fb0bcf6c574 0000000000508744 00000000005086d9 00007fb0c44485a1 00007fb0c44daaa8 00007fb0c494c4d1 00007fb0c494bca8 00007fb0c4950941 00007fb0c49508f9 00007fb0c4950889 00007fb0c4950851 00007fb0c494fa7d 00000000005aa2b4 00000000005aa0e9
0130 02:58:13.833541 (+ 60200us) op_driver.cc:464] REPLICATION: finished
0130 02:58:13.864767 (+ 31226us) write_op.cc:301] APPLY: starting
0130 02:58:13.864826 (+ 59us) tablet.cc:1365] starting BulkCheckPresence
0130 02:58:13.864920 (+ 94us) tablet.cc:1368] finished BulkCheckPresence
0130 02:58:13.864937 (+ 17us) tablet.cc:1370] starting ApplyRowOperation cycle
0130 02:58:13.865373 (+ 436us) tablet.cc:1381] finished ApplyRowOperation cycle
0130 02:58:13.865557 (+ 184us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0130 02:58:13.865597 (+ 40us) write_op.cc:312] APPLY: finished
0130 02:58:13.865770 (+ 173us) log.cc:844] Serialized 29 byte log entry
0130 02:58:13.870492 (+ 4722us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:58:13.870559 (+ 67us) write_op.cc:454] Released schema lock
0130 02:58:13.870619 (+ 60us) write_op.cc:341] FINISH: Updating metrics
0130 02:58:13.870843 (+ 224us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:58:13.870867 (+ 24us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":31056,"apply.run_cpu_time_us":1533,"apply.run_wall_time_us":6056,"num_ops":2,"prepare.queue_time_us":136,"prepare.run_cpu_time_us":5108,"prepare.run_wall_time_us":67194,"raft.queue_time_us":390,"raft.run_cpu_time_us":639,"raft.run_wall_time_us":1286,"replication_time_us":100932,"spinlock_wait_cycles":86503168}]]}
I20260130 02:58:14.001924 18867 master_service.cc:954] ReplaceTablet: received request to replace tablet 2c79a3aecab844698e70834999e1caac from {username='slave'} at 127.0.0.1:50284
W20260130 02:58:14.006707 18875 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4ee03db7d0064d96a53d4f261bd12722 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260130 02:58:14.013105 19625 ts_tablet_manager.cc:1434] T 4c139aa8cd184f488cec4e05ef7dcada P d870164263bb44b6b780a557daea3656: Time spent starting tablet: real 0.024s user 0.006s sys 0.000s
I20260130 02:58:14.013983 19625 tablet_bootstrap.cc:492] T d55291d7cea64a0d9a345ce712fbfa80 P d870164263bb44b6b780a557daea3656: Bootstrap starting.
I20260130 02:58:14.027402 19625 tablet_bootstrap.cc:654] T d55291d7cea64a0d9a345ce712fbfa80 P d870164263bb44b6b780a557daea3656: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:14.030074 20571 tablet_replica.cc:333] T e364f4e70e5749e083e14e7fa7416ff0 P d870164263bb44b6b780a557daea3656: stopping tablet replica
I20260130 02:58:14.032027 20571 raft_consensus.cc:2243] T e364f4e70e5749e083e14e7fa7416ff0 P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260130 02:58:14.037293 18860 catalog_manager.cc:4985] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): tablet 3b148691090e4ee581d46f9845a6e49a (table default.Table_8edea1fae34d4d18b54b0f801a145225 [id=2009496525c94d7f8ee6ce59694b652d]) successfully deleted
I20260130 02:58:14.054764 19625 tablet_bootstrap.cc:492] T d55291d7cea64a0d9a345ce712fbfa80 P d870164263bb44b6b780a557daea3656: No bootstrap required, opened a new log
I20260130 02:58:14.055356 19625 ts_tablet_manager.cc:1403] T d55291d7cea64a0d9a345ce712fbfa80 P d870164263bb44b6b780a557daea3656: Time spent bootstrapping tablet: real 0.042s user 0.006s sys 0.004s
I20260130 02:58:14.055465 20571 raft_consensus.cc:2272] T e364f4e70e5749e083e14e7fa7416ff0 P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260130 02:58:14.057940 20571 ts_tablet_manager.cc:1916] T e364f4e70e5749e083e14e7fa7416ff0 P d870164263bb44b6b780a557daea3656: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260130 02:58:14.057703 19625 raft_consensus.cc:359] T d55291d7cea64a0d9a345ce712fbfa80 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:14.058698 19625 raft_consensus.cc:385] T d55291d7cea64a0d9a345ce712fbfa80 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:14.058952 19625 raft_consensus.cc:740] T d55291d7cea64a0d9a345ce712fbfa80 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d870164263bb44b6b780a557daea3656, State: Initialized, Role: FOLLOWER
I20260130 02:58:14.059511 19625 consensus_queue.cc:260] T d55291d7cea64a0d9a345ce712fbfa80 P d870164263bb44b6b780a557daea3656 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:13.965008 19840 tablet_bootstrap.cc:654] T cf1b66a20fce4c4d96bc0bc32a7bafa8 P 78c2c46c7520492db35d51c244fb4f74: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:14.063283 20570 log.cc:1199] T 3b148691090e4ee581d46f9845a6e49a P 78c2c46c7520492db35d51c244fb4f74: Deleting WAL directory at /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-1/wal/wals/3b148691090e4ee581d46f9845a6e49a
I20260130 02:58:13.965492 20474 consensus_queue.cc:1048] T 4c139aa8cd184f488cec4e05ef7dcada P 78c2c46c7520492db35d51c244fb4f74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260130 02:58:14.064399 20570 ts_tablet_manager.cc:1950] T 3b148691090e4ee581d46f9845a6e49a P 78c2c46c7520492db35d51c244fb4f74: Deleting consensus metadata
I20260130 02:58:13.980861 20538 consensus_queue.cc:1048] T 3389db58493448aea3cf00aca411047c P 78c2c46c7520492db35d51c244fb4f74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260130 02:58:13.981618 19313 leader_election.cc:343] T 78e9bc3e55fe4e4a98e8eb01da59d365 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): Illegal state: must be running to vote when last-logged opid is not known
W20260130 02:58:14.067929 19313 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.100.67:39571, user_credentials={real_user=slave}} blocked reactor thread for 86337.6us
I20260130 02:58:14.070006 19625 ts_tablet_manager.cc:1434] T d55291d7cea64a0d9a345ce712fbfa80 P d870164263bb44b6b780a557daea3656: Time spent starting tablet: real 0.014s user 0.004s sys 0.001s
I20260130 02:58:14.076686 19625 tablet_bootstrap.cc:492] T ba91a508b1cb4430a4d8ca26946aecc8 P d870164263bb44b6b780a557daea3656: Bootstrap starting.
W20260130 02:58:14.077239 19314 leader_election.cc:343] T 78e9bc3e55fe4e4a98e8eb01da59d365 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:14.079118 19314 leader_election.cc:304] T 78e9bc3e55fe4e4a98e8eb01da59d365 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78c2c46c7520492db35d51c244fb4f74; no voters: 371a36d219164974abdf848841528152, d870164263bb44b6b780a557daea3656
W20260130 02:58:14.079948 19314 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.100.65:42021, user_credentials={real_user=slave}} blocked reactor thread for 83146.8us
I20260130 02:58:14.081866 20538 raft_consensus.cc:2749] T 78e9bc3e55fe4e4a98e8eb01da59d365 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260130 02:58:14.088032 19524 raft_consensus.cc:3060] T 4c139aa8cd184f488cec4e05ef7dcada P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:14.075244 20501 consensus_queue.cc:1048] T a8a5038e9def44eb8b3b1e4aeae25652 P 371a36d219164974abdf848841528152 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.016s
I20260130 02:58:14.096354 19625 tablet_bootstrap.cc:654] T ba91a508b1cb4430a4d8ca26946aecc8 P d870164263bb44b6b780a557daea3656: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:14.097139 20591 consensus_queue.cc:1048] T 7caf7cbad5054396b5617094e4d536a0 P d870164263bb44b6b780a557daea3656 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260130 02:58:14.100729 20474 raft_consensus.cc:493] T f9e5f01e69254d658c369f3e21cef523 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:14.101770 20474 raft_consensus.cc:515] T f9e5f01e69254d658c369f3e21cef523 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:14.105321 20474 leader_election.cc:290] T f9e5f01e69254d658c369f3e21cef523 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), d870164263bb44b6b780a557daea3656 (127.18.100.67:39571)
I20260130 02:58:14.111068 19807 tablet_bootstrap.cc:492] T 7caf7cbad5054396b5617094e4d536a0 P 371a36d219164974abdf848841528152: Bootstrap starting.
I20260130 02:58:14.113950 19521 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f9e5f01e69254d658c369f3e21cef523" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
W20260130 02:58:14.118294 19313 leader_election.cc:343] T f9e5f01e69254d658c369f3e21cef523 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:14.119982 19625 tablet_bootstrap.cc:492] T ba91a508b1cb4430a4d8ca26946aecc8 P d870164263bb44b6b780a557daea3656: No bootstrap required, opened a new log
I20260130 02:58:14.120400 19625 ts_tablet_manager.cc:1403] T ba91a508b1cb4430a4d8ca26946aecc8 P d870164263bb44b6b780a557daea3656: Time spent bootstrapping tablet: real 0.044s user 0.006s sys 0.005s
I20260130 02:58:14.120568 19222 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f9e5f01e69254d658c369f3e21cef523" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152" is_pre_election: true
W20260130 02:58:14.121915 19314 leader_election.cc:343] T f9e5f01e69254d658c369f3e21cef523 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:14.122347 19314 leader_election.cc:304] T f9e5f01e69254d658c369f3e21cef523 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78c2c46c7520492db35d51c244fb4f74; no voters: 371a36d219164974abdf848841528152, d870164263bb44b6b780a557daea3656
I20260130 02:58:14.123754 20479 raft_consensus.cc:2749] T f9e5f01e69254d658c369f3e21cef523 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260130 02:58:14.124665 19625 raft_consensus.cc:359] T ba91a508b1cb4430a4d8ca26946aecc8 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:14.125248 19625 raft_consensus.cc:385] T ba91a508b1cb4430a4d8ca26946aecc8 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:14.125468 19807 tablet_bootstrap.cc:654] T 7caf7cbad5054396b5617094e4d536a0 P 371a36d219164974abdf848841528152: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:14.125506 19625 raft_consensus.cc:740] T ba91a508b1cb4430a4d8ca26946aecc8 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d870164263bb44b6b780a557daea3656, State: Initialized, Role: FOLLOWER
I20260130 02:58:14.132081 19625 consensus_queue.cc:260] T ba91a508b1cb4430a4d8ca26946aecc8 P d870164263bb44b6b780a557daea3656 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:14.134135 19625 ts_tablet_manager.cc:1434] T ba91a508b1cb4430a4d8ca26946aecc8 P d870164263bb44b6b780a557daea3656: Time spent starting tablet: real 0.013s user 0.003s sys 0.001s
I20260130 02:58:14.135074 19625 tablet_bootstrap.cc:492] T 4907294442c145c6b03eb1de057c66a8 P d870164263bb44b6b780a557daea3656: Bootstrap starting.
I20260130 02:58:14.139422 20609 tablet.cc:1722] T 127a84346b9f473d9e7f739b1e65feb8 P 371a36d219164974abdf848841528152: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 5 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 6
I20260130 02:58:14.143952 20474 raft_consensus.cc:493] T e2aaaf00f53d418b87d5c7af286e4d84 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:14.146165 20474 raft_consensus.cc:515] T e2aaaf00f53d418b87d5c7af286e4d84 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
W20260130 02:58:14.152065 19159 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.100.66:40767, user_credentials={real_user=slave}} blocked reactor thread for 126709us
I20260130 02:58:14.153690 20590 raft_consensus.cc:493] T 4c139aa8cd184f488cec4e05ef7dcada P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:14.154212 20590 raft_consensus.cc:515] T 4c139aa8cd184f488cec4e05ef7dcada P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:14.157538 20590 leader_election.cc:290] T 4c139aa8cd184f488cec4e05ef7dcada P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767)
I20260130 02:58:14.157806 19625 tablet_bootstrap.cc:654] T 4907294442c145c6b03eb1de057c66a8 P d870164263bb44b6b780a557daea3656: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:14.159842 19221 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4c139aa8cd184f488cec4e05ef7dcada" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152" is_pre_election: true
I20260130 02:58:14.161448 19222 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e2aaaf00f53d418b87d5c7af286e4d84" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152" is_pre_election: true
W20260130 02:58:14.170099 19314 leader_election.cc:343] T e2aaaf00f53d418b87d5c7af286e4d84 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:14.171392 19625 tablet_bootstrap.cc:492] T 4907294442c145c6b03eb1de057c66a8 P d870164263bb44b6b780a557daea3656: No bootstrap required, opened a new log
I20260130 02:58:14.171794 19625 ts_tablet_manager.cc:1403] T 4907294442c145c6b03eb1de057c66a8 P d870164263bb44b6b780a557daea3656: Time spent bootstrapping tablet: real 0.037s user 0.010s sys 0.000s
I20260130 02:58:14.173632 18875 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:50284 (request call id 232) took 2472 ms. Trace:
I20260130 02:58:14.173564 19376 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4c139aa8cd184f488cec4e05ef7dcada" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74" is_pre_election: true
I20260130 02:58:14.173962 19625 raft_consensus.cc:359] T 4907294442c145c6b03eb1de057c66a8 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:14.173950 18875 rpcz_store.cc:276] 0130 02:58:11.700988 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:58:13.957563 (+2256575us) service_pool.cc:225] Handling call
0130 02:58:14.007452 (+ 49889us) catalog_manager.cc:2101] Acquired catalog manager lock
0130 02:58:14.010565 (+ 3113us) catalog_manager.cc:2147] Created new table and tablet info
0130 02:58:14.173025 (+162460us) catalog_manager.cc:2207] Wrote table and tablets to system table
0130 02:58:14.173308 (+ 283us) catalog_manager.cc:2241] Inserted table and tablets into CatalogManager maps
0130 02:58:14.173551 (+ 243us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0130 02:58:14.019754 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0130 02:58:14.020278 (+ 524us) write_op.cc:432] Acquiring schema lock in shared mode
0130 02:58:14.020305 (+ 27us) write_op.cc:435] Acquired schema lock
0130 02:58:14.020321 (+ 16us) tablet.cc:660] Decoding operations
0130 02:58:14.020634 (+ 313us) write_op.cc:620] Acquiring the partition lock for write op
0130 02:58:14.020673 (+ 39us) write_op.cc:641] Partition lock acquired for write op
0130 02:58:14.020698 (+ 25us) tablet.cc:683] Acquiring locks for 2 operations
0130 02:58:14.020920 (+ 222us) tablet.cc:699] Row locks acquired
0130 02:58:14.020936 (+ 16us) write_op.cc:260] PREPARE: finished
0130 02:58:14.021045 (+ 109us) write_op.cc:270] Start()
0130 02:58:14.021131 (+ 86us) write_op.cc:276] Timestamp: P: 1769741894020963 usec, L: 0
0130 02:58:14.021152 (+ 21us) op_driver.cc:348] REPLICATION: starting
0130 02:58:14.021682 (+ 530us) log.cc:844] Serialized 433 byte log entry
0130 02:58:14.170310 (+148628us) op_driver.cc:464] REPLICATION: finished
0130 02:58:14.171495 (+ 1185us) write_op.cc:301] APPLY: starting
0130 02:58:14.171552 (+ 57us) tablet.cc:1365] starting BulkCheckPresence
0130 02:58:14.171649 (+ 97us) tablet.cc:1368] finished BulkCheckPresence
0130 02:58:14.171738 (+ 89us) tablet.cc:1370] starting ApplyRowOperation cycle
0130 02:58:14.172147 (+ 409us) tablet.cc:1381] finished ApplyRowOperation cycle
0130 02:58:14.172273 (+ 126us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0130 02:58:14.172413 (+ 140us) write_op.cc:312] APPLY: finished
0130 02:58:14.172592 (+ 179us) log.cc:844] Serialized 29 byte log entry
0130 02:58:14.172738 (+ 146us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:58:14.172801 (+ 63us) write_op.cc:454] Released schema lock
0130 02:58:14.172830 (+ 29us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":1032,"num_ops":2,"prepare.queue_time_us":2615,"prepare.run_cpu_time_us":2176,"prepare.run_wall_time_us":8267,"raft.queue_time_us":1819,"raft.run_cpu_time_us":13,"raft.run_wall_time_us":14,"replication_time_us":149056}]]}
I20260130 02:58:14.177035 18875 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50284:
name: "default.Table_a132cf962cfc4ed8a6a8c177e088daeb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260130 02:58:14.177404 20538 consensus_queue.cc:1048] T 4c139aa8cd184f488cec4e05ef7dcada P 78c2c46c7520492db35d51c244fb4f74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260130 02:58:14.180962 19462 leader_election.cc:304] T 4c139aa8cd184f488cec4e05ef7dcada P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d870164263bb44b6b780a557daea3656; no voters: 371a36d219164974abdf848841528152, 78c2c46c7520492db35d51c244fb4f74
I20260130 02:58:14.174552 19625 raft_consensus.cc:385] T 4907294442c145c6b03eb1de057c66a8 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:14.187521 19625 raft_consensus.cc:740] T 4907294442c145c6b03eb1de057c66a8 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d870164263bb44b6b780a557daea3656, State: Initialized, Role: FOLLOWER
I20260130 02:58:14.189419 19625 consensus_queue.cc:260] T 4907294442c145c6b03eb1de057c66a8 P d870164263bb44b6b780a557daea3656 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:14.191569 19625 ts_tablet_manager.cc:1434] T 4907294442c145c6b03eb1de057c66a8 P d870164263bb44b6b780a557daea3656: Time spent starting tablet: real 0.020s user 0.001s sys 0.005s
I20260130 02:58:14.192430 19625 tablet_bootstrap.cc:492] T 9fe0448bcc4a4346b3c67fc3f626ed3c P d870164263bb44b6b780a557daea3656: Bootstrap starting.
I20260130 02:58:14.195384 19204 tablet_service.cc:1505] Processing CreateTablet for tablet 306452e208ca4e719609f67345114a13 (DEFAULT_TABLE table=default.Table_fa63c9eff99d4ac1b185a4c4ce2df031 [id=2e8b378422c74abc939955d110737316]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:14.196280 19204 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 306452e208ca4e719609f67345114a13. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:14.196239 20474 leader_election.cc:290] T e2aaaf00f53d418b87d5c7af286e4d84 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), d870164263bb44b6b780a557daea3656 (127.18.100.67:39571)
I20260130 02:58:14.197906 19521 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e2aaaf00f53d418b87d5c7af286e4d84" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
W20260130 02:58:14.199302 19313 leader_election.cc:343] T e2aaaf00f53d418b87d5c7af286e4d84 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:14.199718 19313 leader_election.cc:304] T e2aaaf00f53d418b87d5c7af286e4d84 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78c2c46c7520492db35d51c244fb4f74; no voters: 371a36d219164974abdf848841528152, d870164263bb44b6b780a557daea3656
I20260130 02:58:14.200572 19496 tablet_service.cc:1505] Processing CreateTablet for tablet 306452e208ca4e719609f67345114a13 (DEFAULT_TABLE table=default.Table_fa63c9eff99d4ac1b185a4c4ce2df031 [id=2e8b378422c74abc939955d110737316]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:14.201465 19496 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 306452e208ca4e719609f67345114a13. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:14.205945 20474 raft_consensus.cc:2749] T e2aaaf00f53d418b87d5c7af286e4d84 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260130 02:58:14.209412 19355 tablet_service.cc:1505] Processing CreateTablet for tablet 306452e208ca4e719609f67345114a13 (DEFAULT_TABLE table=default.Table_fa63c9eff99d4ac1b185a4c4ce2df031 [id=2e8b378422c74abc939955d110737316]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:14.210367 19355 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 306452e208ca4e719609f67345114a13. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:14.213258 20608 consensus_queue.cc:1048] T dc09a0f0cecf49738bab04596944c0f3 P 371a36d219164974abdf848841528152 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.004s
W20260130 02:58:14.178455 18875 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a132cf962cfc4ed8a6a8c177e088daeb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260130 02:58:14.221191 20616 raft_consensus.cc:493] T 0e8018c094024c4788c484e8f5466e00 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:14.221742 20616 raft_consensus.cc:515] T 0e8018c094024c4788c484e8f5466e00 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:14.223857 20616 leader_election.cc:290] T 0e8018c094024c4788c484e8f5466e00 P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767), d870164263bb44b6b780a557daea3656 (127.18.100.67:39571)
I20260130 02:58:14.226349 20581 ts_tablet_manager.cc:1929] T d55291d7cea64a0d9a345ce712fbfa80 P 371a36d219164974abdf848841528152: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260130 02:58:14.228314 20581 log.cc:1199] T d55291d7cea64a0d9a345ce712fbfa80 P 371a36d219164974abdf848841528152: Deleting WAL directory at /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-0/wal/wals/d55291d7cea64a0d9a345ce712fbfa80
I20260130 02:58:14.229058 19625 tablet_bootstrap.cc:654] T 9fe0448bcc4a4346b3c67fc3f626ed3c P d870164263bb44b6b780a557daea3656: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:14.241904 20538 raft_consensus.cc:493] T 7783db3568f4434e91e9e456799cc312 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:14.242409 20538 raft_consensus.cc:515] T 7783db3568f4434e91e9e456799cc312 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:14.244719 20538 leader_election.cc:290] T 7783db3568f4434e91e9e456799cc312 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), d870164263bb44b6b780a557daea3656 (127.18.100.67:39571)
I20260130 02:58:14.247469 19521 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7783db3568f4434e91e9e456799cc312" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
W20260130 02:58:14.248732 19313 leader_election.cc:343] T 7783db3568f4434e91e9e456799cc312 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:14.264665 19807 tablet_bootstrap.cc:492] T 7caf7cbad5054396b5617094e4d536a0 P 371a36d219164974abdf848841528152: No bootstrap required, opened a new log
I20260130 02:58:14.265168 19807 ts_tablet_manager.cc:1403] T 7caf7cbad5054396b5617094e4d536a0 P 371a36d219164974abdf848841528152: Time spent bootstrapping tablet: real 0.154s user 0.010s sys 0.003s
I20260130 02:58:14.267976 19159 leader_election.cc:304] T 3389db58493448aea3cf00aca411047c P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 371a36d219164974abdf848841528152; no voters: 78c2c46c7520492db35d51c244fb4f74, d870164263bb44b6b780a557daea3656
I20260130 02:58:14.276219 20608 consensus_queue.cc:1048] T dc09a0f0cecf49738bab04596944c0f3 P 371a36d219164974abdf848841528152 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260130 02:58:14.277595 20497 raft_consensus.cc:2749] T 3389db58493448aea3cf00aca411047c P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260130 02:58:14.279902 19225 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7783db3568f4434e91e9e456799cc312" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152" is_pre_election: true
I20260130 02:58:14.229107 20581 ts_tablet_manager.cc:1950] T d55291d7cea64a0d9a345ce712fbfa80 P 371a36d219164974abdf848841528152: Deleting consensus metadata
I20260130 02:58:14.282021 18861 catalog_manager.cc:4985] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021): tablet d55291d7cea64a0d9a345ce712fbfa80 (table default.Table_b85cb876f25740bfaaab01949ac26b38 [id=839e292e040c4aeb832f3fc0258c32ec]) successfully deleted
I20260130 02:58:14.286405 19807 raft_consensus.cc:359] T 7caf7cbad5054396b5617094e4d536a0 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:14.304157 20590 raft_consensus.cc:2749] T 4c139aa8cd184f488cec4e05ef7dcada P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260130 02:58:14.315579 19625 tablet_bootstrap.cc:492] T 9fe0448bcc4a4346b3c67fc3f626ed3c P d870164263bb44b6b780a557daea3656: No bootstrap required, opened a new log
I20260130 02:58:14.316007 19625 ts_tablet_manager.cc:1403] T 9fe0448bcc4a4346b3c67fc3f626ed3c P d870164263bb44b6b780a557daea3656: Time spent bootstrapping tablet: real 0.124s user 0.009s sys 0.001s
I20260130 02:58:14.318696 19521 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0e8018c094024c4788c484e8f5466e00" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
I20260130 02:58:14.319231 19625 raft_consensus.cc:359] T 9fe0448bcc4a4346b3c67fc3f626ed3c P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:14.319908 19625 raft_consensus.cc:385] T 9fe0448bcc4a4346b3c67fc3f626ed3c P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:14.320173 19625 raft_consensus.cc:740] T 9fe0448bcc4a4346b3c67fc3f626ed3c P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d870164263bb44b6b780a557daea3656, State: Initialized, Role: FOLLOWER
I20260130 02:58:14.320814 19625 consensus_queue.cc:260] T 9fe0448bcc4a4346b3c67fc3f626ed3c P d870164263bb44b6b780a557daea3656 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:14.287078 19807 raft_consensus.cc:385] T 7caf7cbad5054396b5617094e4d536a0 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:14.323472 19807 raft_consensus.cc:740] T 7caf7cbad5054396b5617094e4d536a0 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 371a36d219164974abdf848841528152, State: Initialized, Role: FOLLOWER
I20260130 02:58:14.324553 19625 ts_tablet_manager.cc:1434] T 9fe0448bcc4a4346b3c67fc3f626ed3c P d870164263bb44b6b780a557daea3656: Time spent starting tablet: real 0.008s user 0.006s sys 0.001s
I20260130 02:58:14.324067 19807 consensus_queue.cc:260] T 7caf7cbad5054396b5617094e4d536a0 P 371a36d219164974abdf848841528152 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:14.325297 20609 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.AlterSchema from 127.0.0.1:42870 (request call id 274) took 1289 ms. Trace:
I20260130 02:58:14.326786 19625 tablet_bootstrap.cc:492] T 78e9bc3e55fe4e4a98e8eb01da59d365 P d870164263bb44b6b780a557daea3656: Bootstrap starting.
I20260130 02:58:14.325670 20609 rpcz_store.cc:276] 0130 02:58:13.035985 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:58:13.036114 (+ 129us) service_pool.cc:225] Handling call
0130 02:58:14.325150 (+1289036us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0130 02:58:13.037201 (+ 0us) alter_schema_op.cc:104] PREPARE ALTER-SCHEMA: Starting
0130 02:58:13.037777 (+ 576us) alter_schema_op.cc:63] Acquiring schema lock in exclusive mode
0130 02:58:13.037813 (+ 36us) alter_schema_op.cc:66] Acquired schema lock
0130 02:58:13.037886 (+ 73us) alter_schema_op.cc:119] PREPARE ALTER-SCHEMA: finished
0130 02:58:13.037991 (+ 105us) alter_schema_op.cc:128] START. Timestamp: 1769741893037901
0130 02:58:13.038007 (+ 16us) op_driver.cc:348] REPLICATION: starting
0130 02:58:13.038526 (+ 519us) log.cc:844] Serialized 173 byte log entry
0130 02:58:13.985728 (+947202us) op_driver.cc:464] REPLICATION: finished
0130 02:58:14.139302 (+153574us) alter_schema_op.cc:133] APPLY ALTER-SCHEMA: Starting
0130 02:58:14.140925 (+ 1623us) log.cc:844] Serialized 5 byte log entry
0130 02:58:14.296045 (+155120us) tablet_metadata.cc:641] Metadata flushed
0130 02:58:14.296837 (+ 792us) log.cc:844] Serialized 14 byte log entry
0130 02:58:14.325023 (+ 28186us) alter_schema_op.cc:73] Released schema lock
0130 02:58:14.325054 (+ 31us) alter_schema_op.cc:186] AlterSchemaCommitCallback: making alter schema visible
Metrics: {"child_traces":[["op",{"apply.queue_time_us":153404,"prepare.queue_time_us":281,"prepare.run_cpu_time_us":1558,"prepare.run_wall_time_us":1558,"replication_time_us":947621,"thread_start_us":207549,"threads_started":2,"wal-append.queue_time_us":74042}]]}
I20260130 02:58:14.332888 19625 tablet_bootstrap.cc:654] T 78e9bc3e55fe4e4a98e8eb01da59d365 P d870164263bb44b6b780a557daea3656: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:14.339097 19807 ts_tablet_manager.cc:1434] T 7caf7cbad5054396b5617094e4d536a0 P 371a36d219164974abdf848841528152: Time spent starting tablet: real 0.073s user 0.007s sys 0.000s
I20260130 02:58:14.340054 19807 tablet_bootstrap.cc:492] T 7783db3568f4434e91e9e456799cc312 P 371a36d219164974abdf848841528152: Bootstrap starting.
I20260130 02:58:14.352766 19225 raft_consensus.cc:3060] T 7caf7cbad5054396b5617094e4d536a0 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:14.360168 20530 consensus_queue.cc:1048] T 7caf7cbad5054396b5617094e4d536a0 P d870164263bb44b6b780a557daea3656 [LEADER]: Connected to new peer: Peer: permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260130 02:58:14.391530 19840 tablet_bootstrap.cc:492] T cf1b66a20fce4c4d96bc0bc32a7bafa8 P 78c2c46c7520492db35d51c244fb4f74: No bootstrap required, opened a new log
I20260130 02:58:14.391963 19840 ts_tablet_manager.cc:1403] T cf1b66a20fce4c4d96bc0bc32a7bafa8 P 78c2c46c7520492db35d51c244fb4f74: Time spent bootstrapping tablet: real 0.479s user 0.042s sys 0.032s
I20260130 02:58:14.387668 20538 raft_consensus.cc:493] T 6e12a8eeb3204325a9a8b7779f72ab29 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:14.392475 18874 catalog_manager.cc:5746] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571) does not have the latest schema for tablet ee7c8ff92ca9465fa18be6eae63d78ca (table default.Table_dd22e9dc63f84d8688c0071f427f0b25 [id=77979d498aa040828de415129cb15a06]). Expected version 1 got 0
I20260130 02:58:14.393540 18874 catalog_manager.cc:5746] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571) does not have the latest schema for tablet 54be22352e404ff9b798a075693f7f21 (table default.Table_3a858a3cb886427cb8268f42c953ad4c [id=17150cb73e374ff9a1e92c43e0c3f2c1]). Expected version 1 got 0
I20260130 02:58:14.394701 20538 raft_consensus.cc:515] T 6e12a8eeb3204325a9a8b7779f72ab29 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:14.396440 20538 leader_election.cc:290] T 6e12a8eeb3204325a9a8b7779f72ab29 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), d870164263bb44b6b780a557daea3656 (127.18.100.67:39571)
I20260130 02:58:14.397989 20570 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60110 (request call id 245) took 1268 ms. Trace:
I20260130 02:58:14.398447 20570 rpcz_store.cc:276] 0130 02:58:13.124381 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:58:13.131182 (+ 6801us) service_pool.cc:225] Handling call
0130 02:58:13.297125 (+165943us) ts_tablet_manager.cc:1151] Deleting tablet 3b148691090e4ee581d46f9845a6e49a
0130 02:58:13.297172 (+ 47us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0130 02:58:13.789471 (+492299us) tablet_metadata.cc:641] Metadata flushed
0130 02:58:13.960991 (+171520us) tablet_metadata.cc:641] Metadata flushed
0130 02:58:14.390226 (+429235us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":144088,"tablet-delete.queue_time_us":165334,"thread_start_us":188557,"threads_started":1}
I20260130 02:58:14.399937 20570 tablet_replica.cc:333] T e364f4e70e5749e083e14e7fa7416ff0 P 78c2c46c7520492db35d51c244fb4f74: stopping tablet replica
I20260130 02:58:14.400867 20570 raft_consensus.cc:2243] T e364f4e70e5749e083e14e7fa7416ff0 P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260130 02:58:14.401403 20570 raft_consensus.cc:2272] T e364f4e70e5749e083e14e7fa7416ff0 P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260130 02:58:14.402824 19807 tablet_bootstrap.cc:654] T 7783db3568f4434e91e9e456799cc312 P 371a36d219164974abdf848841528152: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:14.413952 20570 ts_tablet_manager.cc:1916] T e364f4e70e5749e083e14e7fa7416ff0 P 78c2c46c7520492db35d51c244fb4f74: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260130 02:58:14.402151 19840 raft_consensus.cc:359] T cf1b66a20fce4c4d96bc0bc32a7bafa8 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:14.435221 19840 raft_consensus.cc:385] T cf1b66a20fce4c4d96bc0bc32a7bafa8 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:14.435561 19840 raft_consensus.cc:740] T cf1b66a20fce4c4d96bc0bc32a7bafa8 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78c2c46c7520492db35d51c244fb4f74, State: Initialized, Role: FOLLOWER
I20260130 02:58:14.436182 19840 consensus_queue.cc:260] T cf1b66a20fce4c4d96bc0bc32a7bafa8 P 78c2c46c7520492db35d51c244fb4f74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:14.401238 18867 catalog_manager.cc:5941] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4: Sending DeleteTablet for 3 replicas of tablet 2c79a3aecab844698e70834999e1caac
I20260130 02:58:14.438299 19840 ts_tablet_manager.cc:1434] T cf1b66a20fce4c4d96bc0bc32a7bafa8 P 78c2c46c7520492db35d51c244fb4f74: Time spent starting tablet: real 0.046s user 0.005s sys 0.000s
I20260130 02:58:14.438773 18867 catalog_manager.cc:6724] ReplaceTablet: tablet 2c79a3aecab844698e70834999e1caac deleted and replaced by tablet 94355fc1338f479e8798070aff04fcba
I20260130 02:58:14.439141 19840 tablet_bootstrap.cc:492] T dcba6b534f9d406dad4fb502be40921c P 78c2c46c7520492db35d51c244fb4f74: Bootstrap starting.
I20260130 02:58:14.439079 18867 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:50284 (request call id 233) took 2465 ms. Trace:
I20260130 02:58:14.439329 18867 rpcz_store.cc:276] 0130 02:58:11.973245 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:58:14.001794 (+2028549us) service_pool.cc:225] Handling call
0130 02:58:14.439026 (+437232us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0130 02:58:14.260295 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0130 02:58:14.260759 (+ 464us) write_op.cc:432] Acquiring schema lock in shared mode
0130 02:58:14.260788 (+ 29us) write_op.cc:435] Acquired schema lock
0130 02:58:14.260804 (+ 16us) tablet.cc:660] Decoding operations
0130 02:58:14.261135 (+ 331us) write_op.cc:620] Acquiring the partition lock for write op
0130 02:58:14.261181 (+ 46us) write_op.cc:641] Partition lock acquired for write op
0130 02:58:14.261206 (+ 25us) tablet.cc:683] Acquiring locks for 2 operations
0130 02:58:14.261415 (+ 209us) tablet.cc:699] Row locks acquired
0130 02:58:14.261426 (+ 11us) write_op.cc:260] PREPARE: finished
0130 02:58:14.261664 (+ 238us) write_op.cc:270] Start()
0130 02:58:14.261754 (+ 90us) write_op.cc:276] Timestamp: P: 1769741894261604 usec, L: 0
0130 02:58:14.261773 (+ 19us) op_driver.cc:348] REPLICATION: starting
0130 02:58:14.376646 (+114873us) log.cc:844] Serialized 628 byte log entry
0130 02:58:14.389795 (+ 13149us) op_driver.cc:464] REPLICATION: finished
0130 02:58:14.399077 (+ 9282us) write_op.cc:301] APPLY: starting
0130 02:58:14.399149 (+ 72us) tablet.cc:1365] starting BulkCheckPresence
0130 02:58:14.399252 (+ 103us) tablet.cc:1368] finished BulkCheckPresence
0130 02:58:14.399339 (+ 87us) tablet.cc:1370] starting ApplyRowOperation cycle
0130 02:58:14.399664 (+ 325us) tablet.cc:1381] finished ApplyRowOperation cycle
0130 02:58:14.399867 (+ 203us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0130 02:58:14.399903 (+ 36us) write_op.cc:312] APPLY: finished
0130 02:58:14.400099 (+ 196us) log.cc:844] Serialized 29 byte log entry
0130 02:58:14.400241 (+ 142us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:58:14.400309 (+ 68us) write_op.cc:454] Released schema lock
0130 02:58:14.400340 (+ 31us) write_op.cc:341] FINISH: Updating metrics
0130 02:58:14.400570 (+ 230us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:58:14.400595 (+ 25us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":149,"apply.run_cpu_time_us":1469,"apply.run_wall_time_us":1469,"num_ops":2,"prepare.queue_time_us":66234,"prepare.run_cpu_time_us":2973,"prepare.run_wall_time_us":119653,"raft.queue_time_us":81,"raft.run_cpu_time_us":640,"raft.run_wall_time_us":641,"replication_time_us":127927}]]}
I20260130 02:58:14.443542 19204 tablet_service.cc:1552] Processing DeleteTablet for tablet 2c79a3aecab844698e70834999e1caac with delete_type TABLET_DATA_DELETED (replaced by tablet 94355fc1338f479e8798070aff04fcba) from {username='slave'} at 127.0.0.1:42870
I20260130 02:58:14.442358 18867 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50284:
name: "default.Table_e2480022cdae4429b710395eebae7fec"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260130 02:58:14.445173 18867 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e2480022cdae4429b710395eebae7fec in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260130 02:58:14.445263 18861 catalog_manager.cc:4965] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021): delete failed for tablet 2c79a3aecab844698e70834999e1caac because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2c79a3aecab844698e70834999e1caac already in progress: creating tablet
I20260130 02:58:14.420768 18875 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:50284 (request call id 234) took 2435 ms. Trace:
I20260130 02:58:14.426090 18860 catalog_manager.cc:4985] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767): tablet 3b148691090e4ee581d46f9845a6e49a (table default.Table_8edea1fae34d4d18b54b0f801a145225 [id=2009496525c94d7f8ee6ce59694b652d]) successfully deleted
I20260130 02:58:14.448120 18875 rpcz_store.cc:276] 0130 02:58:11.985956 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:58:14.176668 (+2190712us) service_pool.cc:225] Handling call
0130 02:58:14.270877 (+ 94209us) catalog_manager.cc:2101] Acquired catalog manager lock
0130 02:58:14.272363 (+ 1486us) catalog_manager.cc:2147] Created new table and tablet info
0130 02:58:14.420197 (+147834us) catalog_manager.cc:2207] Wrote table and tablets to system table
0130 02:58:14.420466 (+ 269us) catalog_manager.cc:2241] Inserted table and tablets into CatalogManager maps
0130 02:58:14.420709 (+ 243us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0130 02:58:14.380110 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0130 02:58:14.381554 (+ 1444us) write_op.cc:432] Acquiring schema lock in shared mode
0130 02:58:14.381585 (+ 31us) write_op.cc:435] Acquired schema lock
0130 02:58:14.381596 (+ 11us) tablet.cc:660] Decoding operations
0130 02:58:14.402921 (+ 21325us) write_op.cc:620] Acquiring the partition lock for write op
0130 02:58:14.402971 (+ 50us) write_op.cc:641] Partition lock acquired for write op
0130 02:58:14.402997 (+ 26us) tablet.cc:683] Acquiring locks for 2 operations
0130 02:58:14.403287 (+ 290us) tablet.cc:699] Row locks acquired
0130 02:58:14.403299 (+ 12us) write_op.cc:260] PREPARE: finished
0130 02:58:14.403430 (+ 131us) write_op.cc:270] Start()
0130 02:58:14.403529 (+ 99us) write_op.cc:276] Timestamp: P: 1769741894403357 usec, L: 0
0130 02:58:14.403549 (+ 20us) op_driver.cc:348] REPLICATION: starting
0130 02:58:14.404266 (+ 717us) log.cc:844] Serialized 433 byte log entry
0130 02:58:14.412963 (+ 8697us) op_driver.cc:464] REPLICATION: finished
0130 02:58:14.413947 (+ 984us) write_op.cc:301] APPLY: starting
0130 02:58:14.414009 (+ 62us) tablet.cc:1365] starting BulkCheckPresence
0130 02:58:14.414131 (+ 122us) tablet.cc:1368] finished BulkCheckPresence
0130 02:58:14.414219 (+ 88us) tablet.cc:1370] starting ApplyRowOperation cycle
0130 02:58:14.414675 (+ 456us) tablet.cc:1381] finished ApplyRowOperation cycle
0130 02:58:14.414803 (+ 128us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0130 02:58:14.414845 (+ 42us) write_op.cc:312] APPLY: finished
0130 02:58:14.415020 (+ 175us) log.cc:844] Serialized 29 byte log entry
0130 02:58:14.415164 (+ 144us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:58:14.415233 (+ 69us) write_op.cc:454] Released schema lock
0130 02:58:14.415264 (+ 31us) write_op.cc:341] FINISH: Updating metrics
0130 02:58:14.415474 (+ 210us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:58:14.415497 (+ 23us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":822,"apply.run_cpu_time_us":1495,"apply.run_wall_time_us":1495,"num_ops":2,"prepare.queue_time_us":106532,"prepare.run_cpu_time_us":2426,"prepare.run_wall_time_us":27029,"raft.queue_time_us":77,"raft.run_cpu_time_us":653,"raft.run_wall_time_us":653,"replication_time_us":9321}]]}
I20260130 02:58:14.453533 18875 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50284:
table { table_name: "default.Table_68be882f76b74def9608a4147dd883d0" } new_table_name: "default.Table_d95381c590c94802bdc83acef0f0df96" modify_external_catalogs: true
W20260130 02:58:14.429584 19313 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.100.67:39571, user_credentials={real_user=slave}} blocked reactor thread for 92178.9us
W20260130 02:58:14.430020 19314 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.100.65:42021, user_credentials={real_user=slave}} blocked reactor thread for 150802us
I20260130 02:58:14.478307 19350 tablet_service.cc:1552] Processing DeleteTablet for tablet 2c79a3aecab844698e70834999e1caac with delete_type TABLET_DATA_DELETED (replaced by tablet 94355fc1338f479e8798070aff04fcba) from {username='slave'} at 127.0.0.1:60110
I20260130 02:58:14.480455 19225 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6e12a8eeb3204325a9a8b7779f72ab29" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152" is_pre_election: true
I20260130 02:58:14.483394 18867 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:50284 (request call id 235) took 2202 ms. Trace:
I20260130 02:58:14.483741 18867 rpcz_store.cc:276] 0130 02:58:12.281073 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:58:14.441968 (+2160895us) service_pool.cc:225] Handling call
0130 02:58:14.445534 (+ 3566us) catalog_manager.cc:2101] Acquired catalog manager lock
0130 02:58:14.446905 (+ 1371us) catalog_manager.cc:2147] Created new table and tablet info
0130 02:58:14.482786 (+ 35881us) catalog_manager.cc:2207] Wrote table and tablets to system table
0130 02:58:14.483110 (+ 324us) catalog_manager.cc:2241] Inserted table and tablets into CatalogManager maps
0130 02:58:14.483338 (+ 228us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0130 02:58:14.458051 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0130 02:58:14.458497 (+ 446us) write_op.cc:432] Acquiring schema lock in shared mode
0130 02:58:14.458523 (+ 26us) write_op.cc:435] Acquired schema lock
0130 02:58:14.458534 (+ 11us) tablet.cc:660] Decoding operations
0130 02:58:14.458890 (+ 356us) write_op.cc:620] Acquiring the partition lock for write op
0130 02:58:14.458931 (+ 41us) write_op.cc:641] Partition lock acquired for write op
0130 02:58:14.458956 (+ 25us) tablet.cc:683] Acquiring locks for 2 operations
0130 02:58:14.459179 (+ 223us) tablet.cc:699] Row locks acquired
0130 02:58:14.459192 (+ 13us) write_op.cc:260] PREPARE: finished
0130 02:58:14.459309 (+ 117us) write_op.cc:270] Start()
0130 02:58:14.459394 (+ 85us) write_op.cc:276] Timestamp: P: 1769741894459232 usec, L: 0
0130 02:58:14.459412 (+ 18us) op_driver.cc:348] REPLICATION: starting
0130 02:58:14.459924 (+ 512us) log.cc:844] Serialized 433 byte log entry
0130 02:58:14.474006 (+ 14082us) op_driver.cc:464] REPLICATION: finished
0130 02:58:14.474966 (+ 960us) write_op.cc:301] APPLY: starting
0130 02:58:14.475018 (+ 52us) tablet.cc:1365] starting BulkCheckPresence
0130 02:58:14.475131 (+ 113us) tablet.cc:1368] finished BulkCheckPresence
0130 02:58:14.475226 (+ 95us) tablet.cc:1370] starting ApplyRowOperation cycle
0130 02:58:14.475615 (+ 389us) tablet.cc:1381] finished ApplyRowOperation cycle
0130 02:58:14.475754 (+ 139us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0130 02:58:14.475797 (+ 43us) write_op.cc:312] APPLY: finished
0130 02:58:14.475971 (+ 174us) log.cc:844] Serialized 29 byte log entry
0130 02:58:14.476123 (+ 152us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:58:14.476197 (+ 74us) write_op.cc:454] Released schema lock
0130 02:58:14.476228 (+ 31us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":814,"apply.run_cpu_time_us":1445,"apply.run_wall_time_us":1445,"num_ops":2,"prepare.queue_time_us":9994,"raft.queue_time_us":84,"raft.run_cpu_time_us":565,"raft.run_wall_time_us":566,"replication_time_us":14497}]]}
I20260130 02:58:14.487773 18874 catalog_manager.cc:5746] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571) does not have the latest schema for tablet 127a84346b9f473d9e7f739b1e65feb8 (table default.Table_1cbb7ef2df504c1083290b333440c9f7 [id=1515db389f284e2f9ed1a3cd0061d0f1]). Expected version 6 got 0
I20260130 02:58:14.374305 20571 ts_tablet_manager.cc:1929] T e364f4e70e5749e083e14e7fa7416ff0 P d870164263bb44b6b780a557daea3656: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260130 02:58:14.494871 20571 log.cc:1199] T e364f4e70e5749e083e14e7fa7416ff0 P d870164263bb44b6b780a557daea3656: Deleting WAL directory at /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-2/wal/wals/e364f4e70e5749e083e14e7fa7416ff0
I20260130 02:58:14.495782 20571 ts_tablet_manager.cc:1950] T e364f4e70e5749e083e14e7fa7416ff0 P d870164263bb44b6b780a557daea3656: Deleting consensus metadata
I20260130 02:58:14.421936 20643 tablet.cc:1722] T 127a84346b9f473d9e7f739b1e65feb8 P d870164263bb44b6b780a557daea3656: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 5 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 6
I20260130 02:58:14.451802 19496 tablet_service.cc:1552] Processing DeleteTablet for tablet 2c79a3aecab844698e70834999e1caac with delete_type TABLET_DATA_DELETED (replaced by tablet 94355fc1338f479e8798070aff04fcba) from {username='slave'} at 127.0.0.1:59068
I20260130 02:58:14.505103 19521 raft_consensus.cc:3060] T 43cec19c51ec4f05b63800c07f9534ef P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:14.515924 20640 tablet.cc:1722] T 127a84346b9f473d9e7f739b1e65feb8 P 78c2c46c7520492db35d51c244fb4f74: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 5 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 6
I20260130 02:58:14.520277 19625 tablet_bootstrap.cc:492] T 78e9bc3e55fe4e4a98e8eb01da59d365 P d870164263bb44b6b780a557daea3656: No bootstrap required, opened a new log
I20260130 02:58:14.520749 19625 ts_tablet_manager.cc:1403] T 78e9bc3e55fe4e4a98e8eb01da59d365 P d870164263bb44b6b780a557daea3656: Time spent bootstrapping tablet: real 0.195s user 0.006s sys 0.004s
W20260130 02:58:14.490500 19314 leader_election.cc:343] T 7783db3568f4434e91e9e456799cc312 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:14.521114 19314 leader_election.cc:304] T 7783db3568f4434e91e9e456799cc312 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78c2c46c7520492db35d51c244fb4f74; no voters: 371a36d219164974abdf848841528152, d870164263bb44b6b780a557daea3656
I20260130 02:58:14.522418 19524 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6e12a8eeb3204325a9a8b7779f72ab29" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
I20260130 02:58:14.540700 19840 tablet_bootstrap.cc:654] T dcba6b534f9d406dad4fb502be40921c P 78c2c46c7520492db35d51c244fb4f74: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:14.545153 18872 catalog_manager.cc:5654] T dc09a0f0cecf49738bab04596944c0f3 P 371a36d219164974abdf848841528152 reported cstate change: term changed from 0 to 1, leader changed from <none> to 371a36d219164974abdf848841528152 (127.18.100.65). New cstate: current_term: 1 leader_uuid: "371a36d219164974abdf848841528152" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } health_report { overall_health: UNKNOWN } } }
I20260130 02:58:14.556602 18867 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50284:
table { table_name: "default.Table_f359d249f8274f319f5ff4e4ea89dd22" } modify_external_catalogs: true
I20260130 02:58:14.560953 20611 raft_consensus.cc:2749] T 7783db3568f4434e91e9e456799cc312 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260130 02:58:14.562636 19380 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0e8018c094024c4788c484e8f5466e00" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74" is_pre_election: true
W20260130 02:58:14.560497 19314 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.100.65:42021, user_credentials={real_user=slave}} blocked reactor thread for 70026us
I20260130 02:58:14.572765 20571 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59068 (request call id 254) took 1341 ms. Trace:
W20260130 02:58:14.573432 19314 leader_election.cc:343] T 6e12a8eeb3204325a9a8b7779f72ab29 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:14.574903 19625 raft_consensus.cc:359] T 78e9bc3e55fe4e4a98e8eb01da59d365 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:14.576306 19807 tablet_bootstrap.cc:492] T 7783db3568f4434e91e9e456799cc312 P 371a36d219164974abdf848841528152: No bootstrap required, opened a new log
I20260130 02:58:14.576702 19807 ts_tablet_manager.cc:1403] T 7783db3568f4434e91e9e456799cc312 P 371a36d219164974abdf848841528152: Time spent bootstrapping tablet: real 0.237s user 0.009s sys 0.002s
I20260130 02:58:14.576705 18867 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50284:
table { table_name: "default.Table_f359d249f8274f319f5ff4e4ea89dd22" } modify_external_catalogs: true
I20260130 02:58:14.579591 19807 raft_consensus.cc:359] T 7783db3568f4434e91e9e456799cc312 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:14.580190 19807 raft_consensus.cc:385] T 7783db3568f4434e91e9e456799cc312 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:14.580432 19807 raft_consensus.cc:740] T 7783db3568f4434e91e9e456799cc312 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 371a36d219164974abdf848841528152, State: Initialized, Role: FOLLOWER
W20260130 02:58:14.582413 19313 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.100.67:39571, user_credentials={real_user=slave}} blocked reactor thread for 70813.9us
W20260130 02:58:14.585486 19313 leader_election.cc:343] T 6e12a8eeb3204325a9a8b7779f72ab29 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:14.585829 19313 leader_election.cc:304] T 6e12a8eeb3204325a9a8b7779f72ab29 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78c2c46c7520492db35d51c244fb4f74; no voters: 371a36d219164974abdf848841528152, d870164263bb44b6b780a557daea3656
I20260130 02:58:14.595613 19840 tablet_bootstrap.cc:492] T dcba6b534f9d406dad4fb502be40921c P 78c2c46c7520492db35d51c244fb4f74: No bootstrap required, opened a new log
I20260130 02:58:14.595432 19159 leader_election.cc:304] T 0e8018c094024c4788c484e8f5466e00 P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 371a36d219164974abdf848841528152; no voters: 78c2c46c7520492db35d51c244fb4f74, d870164263bb44b6b780a557daea3656
I20260130 02:58:14.596000 19840 ts_tablet_manager.cc:1403] T dcba6b534f9d406dad4fb502be40921c P 78c2c46c7520492db35d51c244fb4f74: Time spent bootstrapping tablet: real 0.157s user 0.007s sys 0.002s
I20260130 02:58:14.596086 20616 raft_consensus.cc:2749] T 0e8018c094024c4788c484e8f5466e00 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260130 02:58:14.581041 19807 consensus_queue.cc:260] T 7783db3568f4434e91e9e456799cc312 P 371a36d219164974abdf848841528152 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:14.600911 19807 ts_tablet_manager.cc:1434] T 7783db3568f4434e91e9e456799cc312 P 371a36d219164974abdf848841528152: Time spent starting tablet: real 0.024s user 0.002s sys 0.005s
I20260130 02:58:14.601774 19807 tablet_bootstrap.cc:492] T 03025d13f0e545cfa6288677a2715343 P 371a36d219164974abdf848841528152: Bootstrap starting.
I20260130 02:58:14.607007 20637 raft_consensus.cc:2749] T 6e12a8eeb3204325a9a8b7779f72ab29 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260130 02:58:14.597498 18872 catalog_manager.cc:5654] T a8a5038e9def44eb8b3b1e4aeae25652 P 371a36d219164974abdf848841528152 reported cstate change: term changed from 0 to 1, leader changed from <none> to 371a36d219164974abdf848841528152 (127.18.100.65). New cstate: current_term: 1 leader_uuid: "371a36d219164974abdf848841528152" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } health_report { overall_health: UNKNOWN } } }
I20260130 02:58:14.617343 18872 catalog_manager.cc:5746] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021) does not have the latest schema for tablet a8a5038e9def44eb8b3b1e4aeae25652 (table default.Table_dee034aec0b24982b8dc2a3fa1b326f1 [id=66033e75b3814a8584311cf7e4a07ad3]). Expected version 1 got 0
W20260130 02:58:14.607178 18875 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2b0edaad6877460cb7982efc62b00b66 consensus state has no leader
I20260130 02:58:14.621325 18875 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:50284 (request call id 236) took 1376 ms. Trace:
I20260130 02:58:14.621773 18875 rpcz_store.cc:276] 0130 02:58:13.244964 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:58:14.453146 (+1208182us) service_pool.cc:225] Handling call
0130 02:58:14.454135 (+ 989us) catalog_manager.cc:2393] Looking up, locking, and authorizing table
0130 02:58:14.456032 (+ 1897us) catalog_manager.cc:3708] Apply alter schema
0130 02:58:14.456646 (+ 614us) catalog_manager.cc:3740] Acquired catalog manager lock
0130 02:58:14.456856 (+ 210us) catalog_manager.cc:3898] Updating metadata on disk
0130 02:58:14.595765 (+138909us) catalog_manager.cc:3929] Committing alterations to in-memory state
0130 02:58:14.621255 (+ 25490us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0130 02:58:14.525512 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0130 02:58:14.526080 (+ 568us) write_op.cc:432] Acquiring schema lock in shared mode
0130 02:58:14.526118 (+ 38us) write_op.cc:435] Acquired schema lock
0130 02:58:14.526134 (+ 16us) tablet.cc:660] Decoding operations
0130 02:58:14.526433 (+ 299us) write_op.cc:620] Acquiring the partition lock for write op
0130 02:58:14.526478 (+ 45us) write_op.cc:641] Partition lock acquired for write op
0130 02:58:14.526504 (+ 26us) tablet.cc:683] Acquiring locks for 1 operations
0130 02:58:14.526688 (+ 184us) tablet.cc:699] Row locks acquired
0130 02:58:14.526705 (+ 17us) write_op.cc:260] PREPARE: finished
0130 02:58:14.526830 (+ 125us) write_op.cc:270] Start()
0130 02:58:14.526925 (+ 95us) write_op.cc:276] Timestamp: P: 1769741894526744 usec, L: 0
0130 02:58:14.526944 (+ 19us) op_driver.cc:348] REPLICATION: starting
0130 02:58:14.527515 (+ 571us) log.cc:844] Serialized 405 byte log entry
0130 02:58:14.579756 (+ 52241us) op_driver.cc:464] REPLICATION: finished
0130 02:58:14.580070 (+ 314us) write_op.cc:301] APPLY: starting
0130 02:58:14.580136 (+ 66us) tablet.cc:1365] starting BulkCheckPresence
0130 02:58:14.580220 (+ 84us) tablet.cc:1368] finished BulkCheckPresence
0130 02:58:14.580318 (+ 98us) tablet.cc:1370] starting ApplyRowOperation cycle
0130 02:58:14.580487 (+ 169us) tablet.cc:1381] finished ApplyRowOperation cycle
0130 02:58:14.580614 (+ 127us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0130 02:58:14.580660 (+ 46us) write_op.cc:312] APPLY: finished
0130 02:58:14.580828 (+ 168us) log.cc:844] Serialized 23 byte log entry
0130 02:58:14.580968 (+ 140us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:58:14.581030 (+ 62us) write_op.cc:454] Released schema lock
0130 02:58:14.581061 (+ 31us) write_op.cc:341] FINISH: Updating metrics
0130 02:58:14.581286 (+ 225us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:58:14.581313 (+ 27us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":173,"apply.run_cpu_time_us":1186,"apply.run_wall_time_us":1186,"num_ops":1,"prepare.queue_time_us":67553,"prepare.run_cpu_time_us":2404,"prepare.run_wall_time_us":41636,"raft.queue_time_us":6429,"raft.run_cpu_time_us":590,"raft.run_wall_time_us":590,"replication_time_us":52723}]]}
I20260130 02:58:14.630777 19625 raft_consensus.cc:385] T 78e9bc3e55fe4e4a98e8eb01da59d365 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:14.622442 19524 raft_consensus.cc:3060] T 4907294442c145c6b03eb1de057c66a8 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:14.635952 19840 raft_consensus.cc:359] T dcba6b534f9d406dad4fb502be40921c P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:14.636574 19840 raft_consensus.cc:385] T dcba6b534f9d406dad4fb502be40921c P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:14.636804 19840 raft_consensus.cc:740] T dcba6b534f9d406dad4fb502be40921c P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78c2c46c7520492db35d51c244fb4f74, State: Initialized, Role: FOLLOWER
I20260130 02:58:14.637270 19625 raft_consensus.cc:740] T 78e9bc3e55fe4e4a98e8eb01da59d365 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d870164263bb44b6b780a557daea3656, State: Initialized, Role: FOLLOWER
I20260130 02:58:14.573254 20571 rpcz_store.cc:276] 0130 02:58:13.231073 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:58:13.231174 (+ 101us) service_pool.cc:225] Handling call
0130 02:58:14.029918 (+798744us) ts_tablet_manager.cc:1151] Deleting tablet e364f4e70e5749e083e14e7fa7416ff0
0130 02:58:14.029961 (+ 43us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0130 02:58:14.140785 (+110824us) tablet_metadata.cc:641] Metadata flushed
0130 02:58:14.374057 (+233272us) tablet_metadata.cc:641] Metadata flushed
0130 02:58:14.572702 (+198645us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":798124}
I20260130 02:58:14.627439 19525 raft_consensus.cc:3060] T d55291d7cea64a0d9a345ce712fbfa80 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:14.646289 19840 consensus_queue.cc:260] T dcba6b534f9d406dad4fb502be40921c P 78c2c46c7520492db35d51c244fb4f74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:14.647974 20629 consensus_queue.cc:1048] T 4907294442c145c6b03eb1de057c66a8 P 78c2c46c7520492db35d51c244fb4f74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260130 02:58:14.649418 20571 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59068 (request call id 255) took 1295 ms. Trace:
I20260130 02:58:14.650915 20571 rpcz_store.cc:276] 0130 02:58:13.353581 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:58:13.353680 (+ 99us) service_pool.cc:225] Handling call
0130 02:58:14.649138 (+1295458us) ts_tablet_manager.cc:1151] Deleting tablet 3b4908bd1a634ca0907234dd3c39e1c3
0130 02:58:14.649181 (+ 43us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0130 02:58:14.649363 (+ 182us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1294028}
I20260130 02:58:14.655052 20571 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59068 (request call id 257) took 1297 ms. Trace:
I20260130 02:58:14.665273 19840 ts_tablet_manager.cc:1434] T dcba6b534f9d406dad4fb502be40921c P 78c2c46c7520492db35d51c244fb4f74: Time spent starting tablet: real 0.069s user 0.006s sys 0.000s
I20260130 02:58:14.666200 19840 tablet_bootstrap.cc:492] T 6218f2f7d73a42428b57a45858f53afd P 78c2c46c7520492db35d51c244fb4f74: Bootstrap starting.
I20260130 02:58:14.666957 20637 raft_consensus.cc:493] T dcba6b534f9d406dad4fb502be40921c P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:14.667363 20637 raft_consensus.cc:515] T dcba6b534f9d406dad4fb502be40921c P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:14.669231 20637 leader_election.cc:290] T dcba6b534f9d406dad4fb502be40921c P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), d870164263bb44b6b780a557daea3656 (127.18.100.67:39571)
I20260130 02:58:14.671660 19225 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dcba6b534f9d406dad4fb502be40921c" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152" is_pre_election: true
W20260130 02:58:14.672996 19314 leader_election.cc:343] T dcba6b534f9d406dad4fb502be40921c P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:14.678221 20570 ts_tablet_manager.cc:1929] T e364f4e70e5749e083e14e7fa7416ff0 P 78c2c46c7520492db35d51c244fb4f74: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260130 02:58:14.678675 20570 log.cc:1199] T e364f4e70e5749e083e14e7fa7416ff0 P 78c2c46c7520492db35d51c244fb4f74: Deleting WAL directory at /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-1/wal/wals/e364f4e70e5749e083e14e7fa7416ff0
I20260130 02:58:14.678680 20571 rpcz_store.cc:276] 0130 02:58:13.357809 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:58:13.358630 (+ 821us) service_pool.cc:225] Handling call
0130 02:58:14.654774 (+1296144us) ts_tablet_manager.cc:1151] Deleting tablet 1bbc55bbf2254d8bb6b721d7e8f69d85
0130 02:58:14.654818 (+ 44us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0130 02:58:14.654996 (+ 178us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1295578}
I20260130 02:58:14.679992 20571 tablet_replica.cc:333] T f54addb1a4304ce69c711ae0cc3b452c P d870164263bb44b6b780a557daea3656: stopping tablet replica
I20260130 02:58:14.680806 20571 raft_consensus.cc:2243] T f54addb1a4304ce69c711ae0cc3b452c P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260130 02:58:14.682127 20571 raft_consensus.cc:2272] T f54addb1a4304ce69c711ae0cc3b452c P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260130 02:58:14.656584 19625 consensus_queue.cc:260] T 78e9bc3e55fe4e4a98e8eb01da59d365 P d870164263bb44b6b780a557daea3656 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:14.677402 19526 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dcba6b534f9d406dad4fb502be40921c" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
I20260130 02:58:14.684216 19625 ts_tablet_manager.cc:1434] T 78e9bc3e55fe4e4a98e8eb01da59d365 P d870164263bb44b6b780a557daea3656: Time spent starting tablet: real 0.163s user 0.006s sys 0.000s
I20260130 02:58:14.686764 20570 ts_tablet_manager.cc:1950] T e364f4e70e5749e083e14e7fa7416ff0 P 78c2c46c7520492db35d51c244fb4f74: Deleting consensus metadata
I20260130 02:58:14.690194 19526 raft_consensus.cc:3060] T 9fe0448bcc4a4346b3c67fc3f626ed3c P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:14.635100 18872 catalog_manager.cc:5654] T 3b02480d9c6f4328af83f7940716d698 P 371a36d219164974abdf848841528152 reported cstate change: term changed from 0 to 1, leader changed from <none> to d870164263bb44b6b780a557daea3656 (127.18.100.67). New cstate: current_term: 1 leader_uuid: "d870164263bb44b6b780a557daea3656" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } }
W20260130 02:58:14.647650 18860 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet 2b0edaad6877460cb7982efc62b00b66 (table default.Table_d95381c590c94802bdc83acef0f0df96 [id=64a8e0e079f34806aa68795f3d548c5e], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 2b0edaad6877460cb7982efc62b00b66 consensus state has no leader
I20260130 02:58:14.694106 18860 catalog_manager.cc:4985] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): tablet e364f4e70e5749e083e14e7fa7416ff0 (table default.Table_bc11cb1bce0348ed9ea0912b3ee521b9 [id=df7d3d68e36142ff8564e63ffe0b8797]) successfully deleted
W20260130 02:58:14.694736 18860 catalog_manager.cc:4952] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): delete failed for tablet 3b4908bd1a634ca0907234dd3c39e1c3 because the tablet was not found. No further retry: Not found: Tablet not found: 3b4908bd1a634ca0907234dd3c39e1c3
W20260130 02:58:14.695289 18860 catalog_manager.cc:4952] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): delete failed for tablet 1bbc55bbf2254d8bb6b721d7e8f69d85 because the tablet was not found. No further retry: Not found: Tablet not found: 1bbc55bbf2254d8bb6b721d7e8f69d85
I20260130 02:58:14.697889 18872 catalog_manager.cc:5746] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021) does not have the latest schema for tablet ee7c8ff92ca9465fa18be6eae63d78ca (table default.Table_dd22e9dc63f84d8688c0071f427f0b25 [id=77979d498aa040828de415129cb15a06]). Expected version 1 got 0
I20260130 02:58:14.707641 19625 tablet_bootstrap.cc:492] T b1e8b78eff0d4a0b92059adb2de4a3ce P d870164263bb44b6b780a557daea3656: Bootstrap starting.
I20260130 02:58:14.709601 19807 tablet_bootstrap.cc:654] T 03025d13f0e545cfa6288677a2715343 P 371a36d219164974abdf848841528152: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:14.710394 19840 tablet_bootstrap.cc:654] T 6218f2f7d73a42428b57a45858f53afd P 78c2c46c7520492db35d51c244fb4f74: Neither blocks nor log segments found. Creating new log.
W20260130 02:58:14.710762 19313 leader_election.cc:343] T dcba6b534f9d406dad4fb502be40921c P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:14.711133 19313 leader_election.cc:304] T dcba6b534f9d406dad4fb502be40921c P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78c2c46c7520492db35d51c244fb4f74; no voters: 371a36d219164974abdf848841528152, d870164263bb44b6b780a557daea3656
I20260130 02:58:14.712906 20497 raft_consensus.cc:493] T 7783db3568f4434e91e9e456799cc312 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:14.713382 20497 raft_consensus.cc:515] T 7783db3568f4434e91e9e456799cc312 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:14.715384 20497 leader_election.cc:290] T 7783db3568f4434e91e9e456799cc312 P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767), d870164263bb44b6b780a557daea3656 (127.18.100.67:39571)
I20260130 02:58:14.716516 20629 raft_consensus.cc:2749] T dcba6b534f9d406dad4fb502be40921c P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260130 02:58:14.717105 19378 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7783db3568f4434e91e9e456799cc312" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74" is_pre_election: true
I20260130 02:58:14.717643 19378 raft_consensus.cc:2468] T 7783db3568f4434e91e9e456799cc312 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 371a36d219164974abdf848841528152 in term 0.
I20260130 02:58:14.718989 19159 leader_election.cc:304] T 7783db3568f4434e91e9e456799cc312 P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 371a36d219164974abdf848841528152, 78c2c46c7520492db35d51c244fb4f74; no voters:
I20260130 02:58:14.719720 20497 raft_consensus.cc:2804] T 7783db3568f4434e91e9e456799cc312 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260130 02:58:14.720052 20497 raft_consensus.cc:493] T 7783db3568f4434e91e9e456799cc312 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260130 02:58:14.720352 20497 raft_consensus.cc:3060] T 7783db3568f4434e91e9e456799cc312 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:14.722999 20616 raft_consensus.cc:493] T 345eaaa5625c48d099280b7fbb2e514b P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:14.723721 19807 tablet_bootstrap.cc:492] T 03025d13f0e545cfa6288677a2715343 P 371a36d219164974abdf848841528152: No bootstrap required, opened a new log
I20260130 02:58:14.723440 20616 raft_consensus.cc:515] T 345eaaa5625c48d099280b7fbb2e514b P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:14.724112 19807 ts_tablet_manager.cc:1403] T 03025d13f0e545cfa6288677a2715343 P 371a36d219164974abdf848841528152: Time spent bootstrapping tablet: real 0.123s user 0.011s sys 0.002s
I20260130 02:58:14.725214 20616 leader_election.cc:290] T 345eaaa5625c48d099280b7fbb2e514b P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767), d870164263bb44b6b780a557daea3656 (127.18.100.67:39571)
I20260130 02:58:14.726312 19807 raft_consensus.cc:359] T 03025d13f0e545cfa6288677a2715343 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:14.728138 19528 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "345eaaa5625c48d099280b7fbb2e514b" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
I20260130 02:58:14.728220 19378 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "345eaaa5625c48d099280b7fbb2e514b" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74" is_pre_election: true
I20260130 02:58:14.728822 19378 raft_consensus.cc:2393] T 345eaaa5625c48d099280b7fbb2e514b P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 371a36d219164974abdf848841528152 in current term 1: Already voted for candidate d870164263bb44b6b780a557daea3656 in this term.
I20260130 02:58:14.729596 20571 ts_tablet_manager.cc:1916] T f54addb1a4304ce69c711ae0cc3b452c P d870164263bb44b6b780a557daea3656: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260130 02:58:14.730233 19159 leader_election.cc:304] T 345eaaa5625c48d099280b7fbb2e514b P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 371a36d219164974abdf848841528152; no voters: 78c2c46c7520492db35d51c244fb4f74, d870164263bb44b6b780a557daea3656
I20260130 02:58:14.731479 20616 raft_consensus.cc:3060] T 345eaaa5625c48d099280b7fbb2e514b P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:14.738929 18872 catalog_manager.cc:5654] T 16691c9315e94253bb9b3fb5058e5636 P 371a36d219164974abdf848841528152 reported cstate change: term changed from 0 to 1, leader changed from <none> to 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66). New cstate: current_term: 1 leader_uuid: "78c2c46c7520492db35d51c244fb4f74" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } }
I20260130 02:58:14.740022 18872 catalog_manager.cc:5746] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021) does not have the latest schema for tablet 16691c9315e94253bb9b3fb5058e5636 (table default.Table_8f311c5878744ef692b496ecfdfc3ede [id=4f499357303d464aafc17679ffcaaa07]). Expected version 1 got 0
I20260130 02:58:14.740756 18872 catalog_manager.cc:5746] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021) does not have the latest schema for tablet 127a84346b9f473d9e7f739b1e65feb8 (table default.Table_1cbb7ef2df504c1083290b333440c9f7 [id=1515db389f284e2f9ed1a3cd0061d0f1]). Expected version 6 got 0
I20260130 02:58:14.741882 20570 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60110 (request call id 246) took 1509 ms. Trace:
I20260130 02:58:14.742692 20616 raft_consensus.cc:2749] T 345eaaa5625c48d099280b7fbb2e514b P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260130 02:58:14.743351 20629 consensus_queue.cc:1048] T 4907294442c145c6b03eb1de057c66a8 P 78c2c46c7520492db35d51c244fb4f74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260130 02:58:14.740603 19525 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7783db3568f4434e91e9e456799cc312" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
I20260130 02:58:14.756911 19625 tablet_bootstrap.cc:654] T b1e8b78eff0d4a0b92059adb2de4a3ce P d870164263bb44b6b780a557daea3656: Neither blocks nor log segments found. Creating new log.
W20260130 02:58:14.752314 19159 leader_election.cc:343] T 7783db3568f4434e91e9e456799cc312 P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:14.753283 20570 rpcz_store.cc:276] 0130 02:58:13.232999 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:58:13.233127 (+ 128us) service_pool.cc:225] Handling call
0130 02:58:14.399780 (+1166653us) ts_tablet_manager.cc:1151] Deleting tablet e364f4e70e5749e083e14e7fa7416ff0
0130 02:58:14.399826 (+ 46us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0130 02:58:14.450335 (+ 50509us) tablet_metadata.cc:641] Metadata flushed
0130 02:58:14.676512 (+226177us) tablet_metadata.cc:641] Metadata flushed
0130 02:58:14.741818 (+ 65306us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1166132}
I20260130 02:58:14.758766 19807 raft_consensus.cc:385] T 03025d13f0e545cfa6288677a2715343 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:14.759104 19807 raft_consensus.cc:740] T 03025d13f0e545cfa6288677a2715343 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 371a36d219164974abdf848841528152, State: Initialized, Role: FOLLOWER
I20260130 02:58:14.759753 18860 catalog_manager.cc:4985] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767): tablet e364f4e70e5749e083e14e7fa7416ff0 (table default.Table_bc11cb1bce0348ed9ea0912b3ee521b9 [id=df7d3d68e36142ff8564e63ffe0b8797]) successfully deleted
I20260130 02:58:14.763232 19807 consensus_queue.cc:260] T 03025d13f0e545cfa6288677a2715343 P 371a36d219164974abdf848841528152 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:14.764323 20570 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60110 (request call id 249) took 1209 ms. Trace:
I20260130 02:58:14.768857 20570 rpcz_store.cc:276] 0130 02:58:13.555391 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:58:13.555519 (+ 128us) service_pool.cc:225] Handling call
0130 02:58:14.764026 (+1208507us) ts_tablet_manager.cc:1151] Deleting tablet f54addb1a4304ce69c711ae0cc3b452c
0130 02:58:14.764086 (+ 60us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0130 02:58:14.764263 (+ 177us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1205633}
W20260130 02:58:14.773535 18860 catalog_manager.cc:4952] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767): delete failed for tablet f54addb1a4304ce69c711ae0cc3b452c because the tablet was not found. No further retry: Not found: Tablet not found: f54addb1a4304ce69c711ae0cc3b452c
I20260130 02:58:14.774652 20570 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60110 (request call id 250) took 1219 ms. Trace:
I20260130 02:58:14.782872 20570 rpcz_store.cc:276] 0130 02:58:13.555705 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:58:13.556205 (+ 500us) service_pool.cc:225] Handling call
0130 02:58:14.774352 (+1218147us) ts_tablet_manager.cc:1151] Deleting tablet 7b80b22bd44c4233837c951aa97e6996
0130 02:58:14.774396 (+ 44us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0130 02:58:14.774592 (+ 196us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1217659}
W20260130 02:58:14.784327 18860 catalog_manager.cc:4952] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767): delete failed for tablet 7b80b22bd44c4233837c951aa97e6996 because the tablet was not found. No further retry: Not found: Tablet not found: 7b80b22bd44c4233837c951aa97e6996
I20260130 02:58:14.785437 20497 raft_consensus.cc:515] T 7783db3568f4434e91e9e456799cc312 P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:14.786720 20570 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60110 (request call id 251) took 1230 ms. Trace:
I20260130 02:58:14.787240 19625 tablet_bootstrap.cc:492] T b1e8b78eff0d4a0b92059adb2de4a3ce P d870164263bb44b6b780a557daea3656: No bootstrap required, opened a new log
I20260130 02:58:14.787623 19625 ts_tablet_manager.cc:1403] T b1e8b78eff0d4a0b92059adb2de4a3ce P d870164263bb44b6b780a557daea3656: Time spent bootstrapping tablet: real 0.080s user 0.009s sys 0.000s
I20260130 02:58:14.787225 20570 rpcz_store.cc:276] 0130 02:58:13.555957 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:58:13.556755 (+ 798us) service_pool.cc:225] Handling call
0130 02:58:14.786365 (+1229610us) ts_tablet_manager.cc:1151] Deleting tablet b2ac91da01954c2390b6b7dcc06c394f
0130 02:58:14.786410 (+ 45us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0130 02:58:14.786653 (+ 243us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1229147}
I20260130 02:58:14.789978 19625 raft_consensus.cc:359] T b1e8b78eff0d4a0b92059adb2de4a3ce P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:14.792658 19807 ts_tablet_manager.cc:1434] T 03025d13f0e545cfa6288677a2715343 P 371a36d219164974abdf848841528152: Time spent starting tablet: real 0.068s user 0.006s sys 0.000s
I20260130 02:58:14.793532 19807 tablet_bootstrap.cc:492] T b1e8b78eff0d4a0b92059adb2de4a3ce P 371a36d219164974abdf848841528152: Bootstrap starting.
I20260130 02:58:14.742504 18872 catalog_manager.cc:5654] T 753344737b5941d9a93b6779e72f63d4 P 371a36d219164974abdf848841528152 reported cstate change: term changed from 0 to 1, leader changed from <none> to 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66). New cstate: current_term: 1 leader_uuid: "78c2c46c7520492db35d51c244fb4f74" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } }
I20260130 02:58:14.802294 18872 catalog_manager.cc:5746] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021) does not have the latest schema for tablet 9fe0448bcc4a4346b3c67fc3f626ed3c (table default.Table_5c1c447e85524ac5a61ffd8a48509d36 [id=70aaf21ad9bd4d959f46939c6b4a4880]). Expected version 1 got 0
I20260130 02:58:14.803153 18872 catalog_manager.cc:5746] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021) does not have the latest schema for tablet 4c139aa8cd184f488cec4e05ef7dcada (table default.Table_e4d07634d84a4678b1c94374e3adc476 [id=fcbd7e3a7fc64d89aa740494d7e1a0f7]). Expected version 1 got 0
I20260130 02:58:14.813934 20655 consensus_queue.cc:1048] T 345eaaa5625c48d099280b7fbb2e514b P d870164263bb44b6b780a557daea3656 [LEADER]: Connected to new peer: Peer: permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260130 02:58:14.790102 19378 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7783db3568f4434e91e9e456799cc312" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74"
I20260130 02:58:14.821800 19378 raft_consensus.cc:3060] T 7783db3568f4434e91e9e456799cc312 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:14.826922 19625 raft_consensus.cc:385] T b1e8b78eff0d4a0b92059adb2de4a3ce P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:14.827325 19625 raft_consensus.cc:740] T b1e8b78eff0d4a0b92059adb2de4a3ce P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d870164263bb44b6b780a557daea3656, State: Initialized, Role: FOLLOWER
I20260130 02:58:14.828176 19378 raft_consensus.cc:2468] T 7783db3568f4434e91e9e456799cc312 P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 371a36d219164974abdf848841528152 in term 1.
I20260130 02:58:14.829402 19159 leader_election.cc:304] T 7783db3568f4434e91e9e456799cc312 P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 371a36d219164974abdf848841528152, 78c2c46c7520492db35d51c244fb4f74; no voters:
I20260130 02:58:14.830376 20616 raft_consensus.cc:2804] T 7783db3568f4434e91e9e456799cc312 P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Leader election won for term 1
I20260130 02:58:14.831203 20616 raft_consensus.cc:697] T 7783db3568f4434e91e9e456799cc312 P 371a36d219164974abdf848841528152 [term 1 LEADER]: Becoming Leader. State: Replica: 371a36d219164974abdf848841528152, State: Running, Role: LEADER
I20260130 02:58:14.831943 20616 consensus_queue.cc:237] T 7783db3568f4434e91e9e456799cc312 P 371a36d219164974abdf848841528152 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:14.835299 19625 consensus_queue.cc:260] T b1e8b78eff0d4a0b92059adb2de4a3ce P d870164263bb44b6b780a557daea3656 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:14.839941 20570 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60110 (request call id 252) took 1283 ms. Trace:
I20260130 02:58:14.840287 20570 rpcz_store.cc:276] 0130 02:58:13.556469 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:58:13.559205 (+ 2736us) service_pool.cc:225] Handling call
0130 02:58:14.839666 (+1280461us) ts_tablet_manager.cc:1151] Deleting tablet 7d96e93d95f14af79cfb268626559b1e
0130 02:58:14.839716 (+ 50us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0130 02:58:14.839885 (+ 169us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1279929}
I20260130 02:58:14.843364 20497 leader_election.cc:290] T 7783db3568f4434e91e9e456799cc312 P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 election: Requested vote from peers 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767), d870164263bb44b6b780a557daea3656 (127.18.100.67:39571)
I20260130 02:58:14.844888 19526 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7783db3568f4434e91e9e456799cc312" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656"
W20260130 02:58:14.851414 19159 leader_election.cc:343] T 7783db3568f4434e91e9e456799cc312 P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:14.853641 19807 tablet_bootstrap.cc:654] T b1e8b78eff0d4a0b92059adb2de4a3ce P 371a36d219164974abdf848841528152: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:14.854128 19625 ts_tablet_manager.cc:1434] T b1e8b78eff0d4a0b92059adb2de4a3ce P d870164263bb44b6b780a557daea3656: Time spent starting tablet: real 0.066s user 0.007s sys 0.000s
I20260130 02:58:14.857244 19840 tablet_bootstrap.cc:492] T 6218f2f7d73a42428b57a45858f53afd P 78c2c46c7520492db35d51c244fb4f74: No bootstrap required, opened a new log
I20260130 02:58:14.857627 19840 ts_tablet_manager.cc:1403] T 6218f2f7d73a42428b57a45858f53afd P 78c2c46c7520492db35d51c244fb4f74: Time spent bootstrapping tablet: real 0.192s user 0.010s sys 0.000s
I20260130 02:58:14.857733 20562 consensus_queue.cc:1048] T 345eaaa5625c48d099280b7fbb2e514b P d870164263bb44b6b780a557daea3656 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260130 02:58:14.863631 19625 tablet_bootstrap.cc:492] T 5fbe92a2227d44d88b4531a7f8aa0e08 P d870164263bb44b6b780a557daea3656: Bootstrap starting.
I20260130 02:58:14.860208 20608 raft_consensus.cc:493] T ba91a508b1cb4430a4d8ca26946aecc8 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:14.865374 20608 raft_consensus.cc:515] T ba91a508b1cb4430a4d8ca26946aecc8 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:14.867172 20608 leader_election.cc:290] T ba91a508b1cb4430a4d8ca26946aecc8 P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767), d870164263bb44b6b780a557daea3656 (127.18.100.67:39571)
W20260130 02:58:14.867645 18860 catalog_manager.cc:4952] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767): delete failed for tablet b2ac91da01954c2390b6b7dcc06c394f because the tablet was not found. No further retry: Not found: Tablet not found: b2ac91da01954c2390b6b7dcc06c394f
W20260130 02:58:14.868264 18860 catalog_manager.cc:4952] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767): delete failed for tablet 7d96e93d95f14af79cfb268626559b1e because the tablet was not found. No further retry: Not found: Tablet not found: 7d96e93d95f14af79cfb268626559b1e
I20260130 02:58:14.873634 19625 tablet_bootstrap.cc:654] T 5fbe92a2227d44d88b4531a7f8aa0e08 P d870164263bb44b6b780a557daea3656: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:14.877411 19378 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ba91a508b1cb4430a4d8ca26946aecc8" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74" is_pre_election: true
I20260130 02:58:14.878155 19378 raft_consensus.cc:2468] T ba91a508b1cb4430a4d8ca26946aecc8 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 371a36d219164974abdf848841528152 in term 0.
I20260130 02:58:14.879211 19159 leader_election.cc:304] T ba91a508b1cb4430a4d8ca26946aecc8 P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 371a36d219164974abdf848841528152, 78c2c46c7520492db35d51c244fb4f74; no voters:
I20260130 02:58:14.879988 20616 raft_consensus.cc:2804] T ba91a508b1cb4430a4d8ca26946aecc8 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260130 02:58:14.880336 20616 raft_consensus.cc:493] T ba91a508b1cb4430a4d8ca26946aecc8 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260130 02:58:14.880646 20616 raft_consensus.cc:3060] T ba91a508b1cb4430a4d8ca26946aecc8 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:14.888664 20570 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60110 (request call id 253) took 1315 ms. Trace:
I20260130 02:58:14.889078 20570 rpcz_store.cc:276] 0130 02:58:13.572872 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:58:13.573057 (+ 185us) service_pool.cc:225] Handling call
0130 02:58:14.888359 (+1315302us) ts_tablet_manager.cc:1151] Deleting tablet 6573198830ad45628e71952a7d9c00a0
0130 02:58:14.888410 (+ 51us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0130 02:58:14.888604 (+ 194us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1314792}
I20260130 02:58:14.890863 20570 tablet_replica.cc:333] T d55291d7cea64a0d9a345ce712fbfa80 P 78c2c46c7520492db35d51c244fb4f74: stopping tablet replica
I20260130 02:58:14.892362 19840 raft_consensus.cc:359] T 6218f2f7d73a42428b57a45858f53afd P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:14.893006 19840 raft_consensus.cc:385] T 6218f2f7d73a42428b57a45858f53afd P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:14.893270 19840 raft_consensus.cc:740] T 6218f2f7d73a42428b57a45858f53afd P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78c2c46c7520492db35d51c244fb4f74, State: Initialized, Role: FOLLOWER
I20260130 02:58:14.893913 19840 consensus_queue.cc:260] T 6218f2f7d73a42428b57a45858f53afd P 78c2c46c7520492db35d51c244fb4f74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
W20260130 02:58:14.895231 18860 catalog_manager.cc:4952] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767): delete failed for tablet 6573198830ad45628e71952a7d9c00a0 because the tablet was not found. No further retry: Not found: Tablet not found: 6573198830ad45628e71952a7d9c00a0
I20260130 02:58:14.876647 19526 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ba91a508b1cb4430a4d8ca26946aecc8" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
I20260130 02:58:14.897902 19526 raft_consensus.cc:2468] T ba91a508b1cb4430a4d8ca26946aecc8 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 371a36d219164974abdf848841528152 in term 0.
I20260130 02:58:14.890271 18869 catalog_manager.cc:5746] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767) does not have the latest schema for tablet a8a5038e9def44eb8b3b1e4aeae25652 (table default.Table_dee034aec0b24982b8dc2a3fa1b326f1 [id=66033e75b3814a8584311cf7e4a07ad3]). Expected version 1 got 0
I20260130 02:58:14.901597 18869 catalog_manager.cc:5746] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767) does not have the latest schema for tablet 16691c9315e94253bb9b3fb5058e5636 (table default.Table_8f311c5878744ef692b496ecfdfc3ede [id=4f499357303d464aafc17679ffcaaa07]). Expected version 1 got 0
I20260130 02:58:14.907549 20571 ts_tablet_manager.cc:1929] T f54addb1a4304ce69c711ae0cc3b452c P d870164263bb44b6b780a557daea3656: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260130 02:58:14.908933 20570 raft_consensus.cc:2243] T d55291d7cea64a0d9a345ce712fbfa80 P 78c2c46c7520492db35d51c244fb4f74 [term 1 LEADER]: Raft consensus shutting down.
I20260130 02:58:14.910408 18869 catalog_manager.cc:5746] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767) does not have the latest schema for tablet 127a84346b9f473d9e7f739b1e65feb8 (table default.Table_1cbb7ef2df504c1083290b333440c9f7 [id=1515db389f284e2f9ed1a3cd0061d0f1]). Expected version 6 got 0
I20260130 02:58:14.914384 18869 catalog_manager.cc:5746] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767) does not have the latest schema for tablet ee7c8ff92ca9465fa18be6eae63d78ca (table default.Table_dd22e9dc63f84d8688c0071f427f0b25 [id=77979d498aa040828de415129cb15a06]). Expected version 1 got 0
I20260130 02:58:14.916852 19625 tablet_bootstrap.cc:492] T 5fbe92a2227d44d88b4531a7f8aa0e08 P d870164263bb44b6b780a557daea3656: No bootstrap required, opened a new log
W20260130 02:58:14.904168 18860 consensus_peers.cc:597] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4 -> Peer 3749baa287ed4f249329a74071781edd (127.18.100.124:34629): Couldn't send request to peer 3749baa287ed4f249329a74071781edd. Status: Network error: Client connection negotiation failed: client connection to 127.18.100.124:34629: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20260130 02:58:14.918946 19625 ts_tablet_manager.cc:1403] T 5fbe92a2227d44d88b4531a7f8aa0e08 P d870164263bb44b6b780a557daea3656: Time spent bootstrapping tablet: real 0.056s user 0.008s sys 0.001s
I20260130 02:58:14.920210 18869 catalog_manager.cc:5654] T 0e8018c094024c4788c484e8f5466e00 P 78c2c46c7520492db35d51c244fb4f74 reported cstate change: term changed from 0 to 1, leader changed from <none> to 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66). New cstate: current_term: 1 leader_uuid: "78c2c46c7520492db35d51c244fb4f74" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } health_report { overall_health: UNKNOWN } } }
I20260130 02:58:14.922786 18869 catalog_manager.cc:5746] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767) does not have the latest schema for tablet 9fe0448bcc4a4346b3c67fc3f626ed3c (table default.Table_5c1c447e85524ac5a61ffd8a48509d36 [id=70aaf21ad9bd4d959f46939c6b4a4880]). Expected version 1 got 0
I20260130 02:58:14.924383 19625 raft_consensus.cc:359] T 5fbe92a2227d44d88b4531a7f8aa0e08 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:14.925011 19625 raft_consensus.cc:385] T 5fbe92a2227d44d88b4531a7f8aa0e08 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:14.925324 19625 raft_consensus.cc:740] T 5fbe92a2227d44d88b4531a7f8aa0e08 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d870164263bb44b6b780a557daea3656, State: Initialized, Role: FOLLOWER
I20260130 02:58:14.926087 19225 raft_consensus.cc:3060] T 0e8018c094024c4788c484e8f5466e00 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:14.925887 19625 consensus_queue.cc:260] T 5fbe92a2227d44d88b4531a7f8aa0e08 P d870164263bb44b6b780a557daea3656 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:14.926759 20571 log.cc:1199] T f54addb1a4304ce69c711ae0cc3b452c P d870164263bb44b6b780a557daea3656: Deleting WAL directory at /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-2/wal/wals/f54addb1a4304ce69c711ae0cc3b452c
I20260130 02:58:14.930027 20571 ts_tablet_manager.cc:1950] T f54addb1a4304ce69c711ae0cc3b452c P d870164263bb44b6b780a557daea3656: Deleting consensus metadata
I20260130 02:58:14.930846 20570 raft_consensus.cc:2272] T d55291d7cea64a0d9a345ce712fbfa80 P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260130 02:58:14.940271 19807 tablet_bootstrap.cc:492] T b1e8b78eff0d4a0b92059adb2de4a3ce P 371a36d219164974abdf848841528152: No bootstrap required, opened a new log
I20260130 02:58:14.940728 19807 ts_tablet_manager.cc:1403] T b1e8b78eff0d4a0b92059adb2de4a3ce P 371a36d219164974abdf848841528152: Time spent bootstrapping tablet: real 0.147s user 0.010s sys 0.001s
I20260130 02:58:14.941077 18869 catalog_manager.cc:5746] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767) does not have the latest schema for tablet 4c139aa8cd184f488cec4e05ef7dcada (table default.Table_e4d07634d84a4678b1c94374e3adc476 [id=fcbd7e3a7fc64d89aa740494d7e1a0f7]). Expected version 1 got 0
I20260130 02:58:14.932191 19625 ts_tablet_manager.cc:1434] T 5fbe92a2227d44d88b4531a7f8aa0e08 P d870164263bb44b6b780a557daea3656: Time spent starting tablet: real 0.010s user 0.002s sys 0.003s
I20260130 02:58:14.943217 19807 raft_consensus.cc:359] T b1e8b78eff0d4a0b92059adb2de4a3ce P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:14.944025 19807 raft_consensus.cc:385] T b1e8b78eff0d4a0b92059adb2de4a3ce P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:14.944278 19807 raft_consensus.cc:740] T b1e8b78eff0d4a0b92059adb2de4a3ce P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 371a36d219164974abdf848841528152, State: Initialized, Role: FOLLOWER
I20260130 02:58:14.945227 20616 raft_consensus.cc:515] T ba91a508b1cb4430a4d8ca26946aecc8 P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:14.947583 19378 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ba91a508b1cb4430a4d8ca26946aecc8" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74"
I20260130 02:58:14.948073 19378 raft_consensus.cc:3060] T ba91a508b1cb4430a4d8ca26946aecc8 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:14.950889 19807 consensus_queue.cc:260] T b1e8b78eff0d4a0b92059adb2de4a3ce P 371a36d219164974abdf848841528152 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:14.953977 19807 ts_tablet_manager.cc:1434] T b1e8b78eff0d4a0b92059adb2de4a3ce P 371a36d219164974abdf848841528152: Time spent starting tablet: real 0.013s user 0.005s sys 0.000s
I20260130 02:58:14.958029 19840 ts_tablet_manager.cc:1434] T 6218f2f7d73a42428b57a45858f53afd P 78c2c46c7520492db35d51c244fb4f74: Time spent starting tablet: real 0.100s user 0.007s sys 0.000s
I20260130 02:58:14.958984 19840 tablet_bootstrap.cc:492] T 25fd48e693424537b91ba7c3fcc96878 P 78c2c46c7520492db35d51c244fb4f74: Bootstrap starting.
I20260130 02:58:14.945415 19625 tablet_bootstrap.cc:492] T f41f4cff7dd4489f948700a45e9a7057 P d870164263bb44b6b780a557daea3656: Bootstrap starting.
I20260130 02:58:14.954751 20571 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59068 (request call id 258) took 1593 ms. Trace:
I20260130 02:58:14.973757 20616 leader_election.cc:290] T ba91a508b1cb4430a4d8ca26946aecc8 P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 election: Requested vote from peers 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767), d870164263bb44b6b780a557daea3656 (127.18.100.67:39571)
I20260130 02:58:14.975404 19526 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ba91a508b1cb4430a4d8ca26946aecc8" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656"
I20260130 02:58:14.975901 19526 raft_consensus.cc:3060] T ba91a508b1cb4430a4d8ca26946aecc8 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:14.962524 20571 rpcz_store.cc:276] 0130 02:58:13.361109 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:58:13.361199 (+ 90us) service_pool.cc:225] Handling call
0130 02:58:14.679841 (+1318642us) ts_tablet_manager.cc:1151] Deleting tablet f54addb1a4304ce69c711ae0cc3b452c
0130 02:58:14.679880 (+ 39us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0130 02:58:14.837981 (+158101us) tablet_metadata.cc:641] Metadata flushed
0130 02:58:14.907282 (+ 69301us) tablet_metadata.cc:641] Metadata flushed
0130 02:58:14.954690 (+ 47408us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1318158}
I20260130 02:58:14.985056 18860 catalog_manager.cc:4985] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): tablet f54addb1a4304ce69c711ae0cc3b452c (table default.Table_2d345d0f0957486c86c2cf2e34b00941 [id=4c3e0d3ecd5642a3b7fc7d20e2521418]) successfully deleted
I20260130 02:58:14.985464 20571 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59068 (request call id 259) took 1623 ms. Trace:
I20260130 02:58:14.985780 20571 rpcz_store.cc:276] 0130 02:58:13.362417 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:58:13.362500 (+ 83us) service_pool.cc:225] Handling call
0130 02:58:14.985194 (+1622694us) ts_tablet_manager.cc:1151] Deleting tablet 6573198830ad45628e71952a7d9c00a0
0130 02:58:14.985237 (+ 43us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0130 02:58:14.985409 (+ 172us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1622219}
I20260130 02:58:14.986222 19526 raft_consensus.cc:2468] T ba91a508b1cb4430a4d8ca26946aecc8 P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 371a36d219164974abdf848841528152 in term 1.
W20260130 02:58:14.986824 18860 catalog_manager.cc:4952] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): delete failed for tablet 6573198830ad45628e71952a7d9c00a0 because the tablet was not found. No further retry: Not found: Tablet not found: 6573198830ad45628e71952a7d9c00a0
I20260130 02:58:14.987653 19159 leader_election.cc:304] T ba91a508b1cb4430a4d8ca26946aecc8 P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 371a36d219164974abdf848841528152, d870164263bb44b6b780a557daea3656; no voters:
I20260130 02:58:14.988559 20608 raft_consensus.cc:2804] T ba91a508b1cb4430a4d8ca26946aecc8 P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Leader election won for term 1
I20260130 02:58:14.988276 18866 catalog_manager.cc:5746] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571) does not have the latest schema for tablet a8a5038e9def44eb8b3b1e4aeae25652 (table default.Table_dee034aec0b24982b8dc2a3fa1b326f1 [id=66033e75b3814a8584311cf7e4a07ad3]). Expected version 1 got 0
I20260130 02:58:14.988962 20608 raft_consensus.cc:697] T ba91a508b1cb4430a4d8ca26946aecc8 P 371a36d219164974abdf848841528152 [term 1 LEADER]: Becoming Leader. State: Replica: 371a36d219164974abdf848841528152, State: Running, Role: LEADER
I20260130 02:58:14.989670 20608 consensus_queue.cc:237] T ba91a508b1cb4430a4d8ca26946aecc8 P 371a36d219164974abdf848841528152 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:14.990794 20571 tablet_replica.cc:333] T b2ac91da01954c2390b6b7dcc06c394f P d870164263bb44b6b780a557daea3656: stopping tablet replica
I20260130 02:58:14.994205 20570 ts_tablet_manager.cc:1916] T d55291d7cea64a0d9a345ce712fbfa80 P 78c2c46c7520492db35d51c244fb4f74: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260130 02:58:14.997958 18866 catalog_manager.cc:5746] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571) does not have the latest schema for tablet 345eaaa5625c48d099280b7fbb2e514b (table default.Table_c1886dd9a7934e7797e0e7e0b065d692 [id=7a2d5367224b492b844199088b01b149]). Expected version 1 got 0
I20260130 02:58:15.000036 18866 catalog_manager.cc:5746] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571) does not have the latest schema for tablet ee7c8ff92ca9465fa18be6eae63d78ca (table default.Table_dd22e9dc63f84d8688c0071f427f0b25 [id=77979d498aa040828de415129cb15a06]). Expected version 1 got 0
I20260130 02:58:15.001562 19807 tablet_bootstrap.cc:492] T 5fbe92a2227d44d88b4531a7f8aa0e08 P 371a36d219164974abdf848841528152: Bootstrap starting.
I20260130 02:58:15.002480 20571 raft_consensus.cc:2243] T b2ac91da01954c2390b6b7dcc06c394f P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260130 02:58:15.003108 20571 raft_consensus.cc:2272] T b2ac91da01954c2390b6b7dcc06c394f P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260130 02:58:15.005002 18866 catalog_manager.cc:5746] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571) does not have the latest schema for tablet 7caf7cbad5054396b5617094e4d536a0 (table default.Table_007e969f70d345a7bcb3dfbaabbc025d [id=77d4fa8705a3419cb0a84d3ca1bb4ce3]). Expected version 1 got 0
I20260130 02:58:15.008080 19807 tablet_bootstrap.cc:654] T 5fbe92a2227d44d88b4531a7f8aa0e08 P 371a36d219164974abdf848841528152: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:15.009100 19625 tablet_bootstrap.cc:654] T f41f4cff7dd4489f948700a45e9a7057 P d870164263bb44b6b780a557daea3656: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:15.009781 18866 catalog_manager.cc:5746] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571) does not have the latest schema for tablet 16691c9315e94253bb9b3fb5058e5636 (table default.Table_8f311c5878744ef692b496ecfdfc3ede [id=4f499357303d464aafc17679ffcaaa07]). Expected version 1 got 0
I20260130 02:58:15.017483 18866 catalog_manager.cc:5746] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571) does not have the latest schema for tablet 127a84346b9f473d9e7f739b1e65feb8 (table default.Table_1cbb7ef2df504c1083290b333440c9f7 [id=1515db389f284e2f9ed1a3cd0061d0f1]). Expected version 6 got 5
I20260130 02:58:15.010982 20571 ts_tablet_manager.cc:1916] T b2ac91da01954c2390b6b7dcc06c394f P d870164263bb44b6b780a557daea3656: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260130 02:58:15.021807 19378 raft_consensus.cc:2468] T ba91a508b1cb4430a4d8ca26946aecc8 P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 371a36d219164974abdf848841528152 in term 1.
I20260130 02:58:15.036113 19625 tablet_bootstrap.cc:492] T f41f4cff7dd4489f948700a45e9a7057 P d870164263bb44b6b780a557daea3656: No bootstrap required, opened a new log
I20260130 02:58:15.036598 19625 ts_tablet_manager.cc:1403] T f41f4cff7dd4489f948700a45e9a7057 P d870164263bb44b6b780a557daea3656: Time spent bootstrapping tablet: real 0.091s user 0.008s sys 0.003s
I20260130 02:58:15.037638 19807 tablet_bootstrap.cc:492] T 5fbe92a2227d44d88b4531a7f8aa0e08 P 371a36d219164974abdf848841528152: No bootstrap required, opened a new log
I20260130 02:58:15.038019 19807 ts_tablet_manager.cc:1403] T 5fbe92a2227d44d88b4531a7f8aa0e08 P 371a36d219164974abdf848841528152: Time spent bootstrapping tablet: real 0.037s user 0.007s sys 0.003s
I20260130 02:58:15.040277 20629 raft_consensus.cc:493] T cf1b66a20fce4c4d96bc0bc32a7bafa8 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:15.040727 20629 raft_consensus.cc:515] T cf1b66a20fce4c4d96bc0bc32a7bafa8 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:15.026454 19840 tablet_bootstrap.cc:654] T 25fd48e693424537b91ba7c3fcc96878 P 78c2c46c7520492db35d51c244fb4f74: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:15.043421 20629 leader_election.cc:290] T cf1b66a20fce4c4d96bc0bc32a7bafa8 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), d870164263bb44b6b780a557daea3656 (127.18.100.67:39571)
I20260130 02:58:15.044790 19526 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cf1b66a20fce4c4d96bc0bc32a7bafa8" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
W20260130 02:58:15.045830 19313 leader_election.cc:343] T cf1b66a20fce4c4d96bc0bc32a7bafa8 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:15.046564 19225 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cf1b66a20fce4c4d96bc0bc32a7bafa8" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152" is_pre_election: true
W20260130 02:58:15.047648 19314 leader_election.cc:343] T cf1b66a20fce4c4d96bc0bc32a7bafa8 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:15.047999 19314 leader_election.cc:304] T cf1b66a20fce4c4d96bc0bc32a7bafa8 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78c2c46c7520492db35d51c244fb4f74; no voters: 371a36d219164974abdf848841528152, d870164263bb44b6b780a557daea3656
I20260130 02:58:15.048909 20629 raft_consensus.cc:2749] T cf1b66a20fce4c4d96bc0bc32a7bafa8 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260130 02:58:15.050365 19807 raft_consensus.cc:359] T 5fbe92a2227d44d88b4531a7f8aa0e08 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:15.051086 19807 raft_consensus.cc:385] T 5fbe92a2227d44d88b4531a7f8aa0e08 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:15.051363 19807 raft_consensus.cc:740] T 5fbe92a2227d44d88b4531a7f8aa0e08 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 371a36d219164974abdf848841528152, State: Initialized, Role: FOLLOWER
I20260130 02:58:15.048813 19625 raft_consensus.cc:359] T f41f4cff7dd4489f948700a45e9a7057 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:15.052110 19625 raft_consensus.cc:385] T f41f4cff7dd4489f948700a45e9a7057 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:15.051939 19807 consensus_queue.cc:260] T 5fbe92a2227d44d88b4531a7f8aa0e08 P 371a36d219164974abdf848841528152 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:15.052416 19625 raft_consensus.cc:740] T f41f4cff7dd4489f948700a45e9a7057 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d870164263bb44b6b780a557daea3656, State: Initialized, Role: FOLLOWER
I20260130 02:58:15.054266 19807 ts_tablet_manager.cc:1434] T 5fbe92a2227d44d88b4531a7f8aa0e08 P 371a36d219164974abdf848841528152: Time spent starting tablet: real 0.016s user 0.006s sys 0.000s
I20260130 02:58:15.055140 19807 tablet_bootstrap.cc:492] T 274530359b174ff79f046c2d4a6afa76 P 371a36d219164974abdf848841528152: Bootstrap starting.
I20260130 02:58:15.054368 19625 consensus_queue.cc:260] T f41f4cff7dd4489f948700a45e9a7057 P d870164263bb44b6b780a557daea3656 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:15.056221 18873 catalog_manager.cc:5746] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767) does not have the latest schema for tablet 345eaaa5625c48d099280b7fbb2e514b (table default.Table_c1886dd9a7934e7797e0e7e0b065d692 [id=7a2d5367224b492b844199088b01b149]). Expected version 1 got 0
I20260130 02:58:15.061340 19807 tablet_bootstrap.cc:654] T 274530359b174ff79f046c2d4a6afa76 P 371a36d219164974abdf848841528152: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:15.076332 19625 ts_tablet_manager.cc:1434] T f41f4cff7dd4489f948700a45e9a7057 P d870164263bb44b6b780a557daea3656: Time spent starting tablet: real 0.035s user 0.005s sys 0.000s
I20260130 02:58:15.078615 19625 tablet_bootstrap.cc:492] T 274530359b174ff79f046c2d4a6afa76 P d870164263bb44b6b780a557daea3656: Bootstrap starting.
W20260130 02:58:15.079385 19872 thread.cc:641] rpc reactor (reactor) Time spent creating pthread: real 0.738s user 0.013s sys 0.035s
W20260130 02:58:15.079712 19872 thread.cc:608] rpc reactor (reactor) Time spent starting thread: real 0.739s user 0.013s sys 0.035s
I20260130 02:58:15.087702 18873 catalog_manager.cc:5746] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767) does not have the latest schema for tablet b1e8b78eff0d4a0b92059adb2de4a3ce (table default.Table_d0b4516d24384e5386c1debaf2543adb [id=f1342f9497b74e7a9ecad1de8082578d]). Expected version 1 got 0
I20260130 02:58:15.093806 19840 tablet_bootstrap.cc:492] T 25fd48e693424537b91ba7c3fcc96878 P 78c2c46c7520492db35d51c244fb4f74: No bootstrap required, opened a new log
I20260130 02:58:15.094285 19840 ts_tablet_manager.cc:1403] T 25fd48e693424537b91ba7c3fcc96878 P 78c2c46c7520492db35d51c244fb4f74: Time spent bootstrapping tablet: real 0.135s user 0.009s sys 0.001s
I20260130 02:58:15.096616 18873 catalog_manager.cc:5746] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767) does not have the latest schema for tablet 7caf7cbad5054396b5617094e4d536a0 (table default.Table_007e969f70d345a7bcb3dfbaabbc025d [id=77d4fa8705a3419cb0a84d3ca1bb4ce3]). Expected version 1 got 0
I20260130 02:58:15.099116 18873 catalog_manager.cc:5746] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767) does not have the latest schema for tablet 274530359b174ff79f046c2d4a6afa76 (table default.Table_c4f4904b6fac4b9880979cf0314e71c1 [id=c456c6418c93499a9d08f652720854ca]). Expected version 1 got 0
I20260130 02:58:15.104089 18873 catalog_manager.cc:5746] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767) does not have the latest schema for tablet 127a84346b9f473d9e7f739b1e65feb8 (table default.Table_1cbb7ef2df504c1083290b333440c9f7 [id=1515db389f284e2f9ed1a3cd0061d0f1]). Expected version 6 got 0
I20260130 02:58:15.105054 19840 raft_consensus.cc:359] T 25fd48e693424537b91ba7c3fcc96878 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:15.105741 19840 raft_consensus.cc:385] T 25fd48e693424537b91ba7c3fcc96878 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:15.105993 19840 raft_consensus.cc:740] T 25fd48e693424537b91ba7c3fcc96878 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78c2c46c7520492db35d51c244fb4f74, State: Initialized, Role: FOLLOWER
I20260130 02:58:15.112845 19625 tablet_bootstrap.cc:654] T 274530359b174ff79f046c2d4a6afa76 P d870164263bb44b6b780a557daea3656: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:15.114607 19840 consensus_queue.cc:260] T 25fd48e693424537b91ba7c3fcc96878 P 78c2c46c7520492db35d51c244fb4f74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:15.115794 18873 catalog_manager.cc:5746] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767) does not have the latest schema for tablet 16691c9315e94253bb9b3fb5058e5636 (table default.Table_8f311c5878744ef692b496ecfdfc3ede [id=4f499357303d464aafc17679ffcaaa07]). Expected version 1 got 0
I20260130 02:58:15.116896 20571 ts_tablet_manager.cc:1929] T b2ac91da01954c2390b6b7dcc06c394f P d870164263bb44b6b780a557daea3656: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260130 02:58:15.117295 20571 log.cc:1199] T b2ac91da01954c2390b6b7dcc06c394f P d870164263bb44b6b780a557daea3656: Deleting WAL directory at /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-2/wal/wals/b2ac91da01954c2390b6b7dcc06c394f
I20260130 02:58:15.117977 20571 ts_tablet_manager.cc:1950] T b2ac91da01954c2390b6b7dcc06c394f P d870164263bb44b6b780a557daea3656: Deleting consensus metadata
I20260130 02:58:15.120288 18873 catalog_manager.cc:5654] T 9fe0448bcc4a4346b3c67fc3f626ed3c P 78c2c46c7520492db35d51c244fb4f74 reported cstate change: term changed from 0 to 1, leader changed from <none> to 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66). New cstate: current_term: 1 leader_uuid: "78c2c46c7520492db35d51c244fb4f74" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } health_report { overall_health: UNKNOWN } } }
I20260130 02:58:15.124153 20571 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59068 (request call id 261) took 1756 ms. Trace:
I20260130 02:58:15.124341 19840 ts_tablet_manager.cc:1434] T 25fd48e693424537b91ba7c3fcc96878 P 78c2c46c7520492db35d51c244fb4f74: Time spent starting tablet: real 0.029s user 0.006s sys 0.000s
I20260130 02:58:15.124486 20571 rpcz_store.cc:276] 0130 02:58:13.367818 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:58:13.367921 (+ 103us) service_pool.cc:225] Handling call
0130 02:58:14.986921 (+1619000us) ts_tablet_manager.cc:1151] Deleting tablet b2ac91da01954c2390b6b7dcc06c394f
0130 02:58:14.990638 (+ 3717us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0130 02:58:15.053586 (+ 62948us) tablet_metadata.cc:641] Metadata flushed
0130 02:58:15.116663 (+ 63077us) tablet_metadata.cc:641] Metadata flushed
0130 02:58:15.124077 (+ 7414us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1618518}
I20260130 02:58:15.125808 18860 catalog_manager.cc:4985] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): tablet b2ac91da01954c2390b6b7dcc06c394f (table default.Table_f2f790e8e46840ecb199c34fae899ada [id=60b8e98e3f864716a6c1aa3816612211]) successfully deleted
I20260130 02:58:15.127512 20571 tablet_replica.cc:333] T 8bd3e1c06b89426da57e2b259e7983af P d870164263bb44b6b780a557daea3656: stopping tablet replica
I20260130 02:58:15.128089 18873 catalog_manager.cc:5746] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767) does not have the latest schema for tablet 9fe0448bcc4a4346b3c67fc3f626ed3c (table default.Table_5c1c447e85524ac5a61ffd8a48509d36 [id=70aaf21ad9bd4d959f46939c6b4a4880]). Expected version 1 got 0
I20260130 02:58:15.128355 20571 raft_consensus.cc:2243] T 8bd3e1c06b89426da57e2b259e7983af P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260130 02:58:15.128777 20571 raft_consensus.cc:2272] T 8bd3e1c06b89426da57e2b259e7983af P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260130 02:58:15.131224 18873 catalog_manager.cc:5654] T 4c139aa8cd184f488cec4e05ef7dcada P 78c2c46c7520492db35d51c244fb4f74 reported cstate change: term changed from 0 to 1, leader changed from <none> to 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66). New cstate: current_term: 1 leader_uuid: "78c2c46c7520492db35d51c244fb4f74" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } health_report { overall_health: UNKNOWN } } }
I20260130 02:58:15.136145 19807 tablet_bootstrap.cc:492] T 274530359b174ff79f046c2d4a6afa76 P 371a36d219164974abdf848841528152: No bootstrap required, opened a new log
I20260130 02:58:15.136538 19807 ts_tablet_manager.cc:1403] T 274530359b174ff79f046c2d4a6afa76 P 371a36d219164974abdf848841528152: Time spent bootstrapping tablet: real 0.082s user 0.025s sys 0.033s
I20260130 02:58:15.138717 19807 raft_consensus.cc:359] T 274530359b174ff79f046c2d4a6afa76 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:15.139312 19807 raft_consensus.cc:385] T 274530359b174ff79f046c2d4a6afa76 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:15.139565 19807 raft_consensus.cc:740] T 274530359b174ff79f046c2d4a6afa76 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 371a36d219164974abdf848841528152, State: Initialized, Role: FOLLOWER
I20260130 02:58:15.140137 19807 consensus_queue.cc:260] T 274530359b174ff79f046c2d4a6afa76 P 371a36d219164974abdf848841528152 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:15.141104 19840 tablet_bootstrap.cc:492] T 838e49e5e8874db393bdc15a1c92f87e P 78c2c46c7520492db35d51c244fb4f74: Bootstrap starting.
I20260130 02:58:15.146262 19625 tablet_bootstrap.cc:492] T 274530359b174ff79f046c2d4a6afa76 P d870164263bb44b6b780a557daea3656: No bootstrap required, opened a new log
I20260130 02:58:15.122351 20570 ts_tablet_manager.cc:1929] T d55291d7cea64a0d9a345ce712fbfa80 P 78c2c46c7520492db35d51c244fb4f74: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260130 02:58:15.146665 19625 ts_tablet_manager.cc:1403] T 274530359b174ff79f046c2d4a6afa76 P d870164263bb44b6b780a557daea3656: Time spent bootstrapping tablet: real 0.069s user 0.006s sys 0.004s
I20260130 02:58:15.146787 20570 log.cc:1199] T d55291d7cea64a0d9a345ce712fbfa80 P 78c2c46c7520492db35d51c244fb4f74: Deleting WAL directory at /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-1/wal/wals/d55291d7cea64a0d9a345ce712fbfa80
I20260130 02:58:15.147492 20570 ts_tablet_manager.cc:1950] T d55291d7cea64a0d9a345ce712fbfa80 P 78c2c46c7520492db35d51c244fb4f74: Deleting consensus metadata
I20260130 02:58:15.136691 18873 catalog_manager.cc:5746] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767) does not have the latest schema for tablet 4c139aa8cd184f488cec4e05ef7dcada (table default.Table_e4d07634d84a4678b1c94374e3adc476 [id=fcbd7e3a7fc64d89aa740494d7e1a0f7]). Expected version 1 got 0
I20260130 02:58:15.149629 18873 catalog_manager.cc:5746] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767) does not have the latest schema for tablet a8a5038e9def44eb8b3b1e4aeae25652 (table default.Table_dee034aec0b24982b8dc2a3fa1b326f1 [id=66033e75b3814a8584311cf7e4a07ad3]). Expected version 1 got 0
I20260130 02:58:15.149945 20570 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60110 (request call id 255) took 1199 ms. Trace:
I20260130 02:58:15.149869 19625 raft_consensus.cc:359] T 274530359b174ff79f046c2d4a6afa76 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:15.150269 20570 rpcz_store.cc:276] 0130 02:58:13.950921 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:58:13.951026 (+ 105us) service_pool.cc:225] Handling call
0130 02:58:14.890690 (+939664us) ts_tablet_manager.cc:1151] Deleting tablet d55291d7cea64a0d9a345ce712fbfa80
0130 02:58:14.890735 (+ 45us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0130 02:58:15.057693 (+166958us) tablet_metadata.cc:641] Metadata flushed
0130 02:58:15.122122 (+ 64429us) tablet_metadata.cc:641] Metadata flushed
0130 02:58:15.149883 (+ 27761us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":16880,"tablet-delete.queue_time_us":938670}
I20260130 02:58:15.151522 20571 ts_tablet_manager.cc:1916] T 8bd3e1c06b89426da57e2b259e7983af P d870164263bb44b6b780a557daea3656: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260130 02:58:15.146436 19840 tablet_bootstrap.cc:654] T 838e49e5e8874db393bdc15a1c92f87e P 78c2c46c7520492db35d51c244fb4f74: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:15.150444 19625 raft_consensus.cc:385] T 274530359b174ff79f046c2d4a6afa76 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:15.156401 19625 raft_consensus.cc:740] T 274530359b174ff79f046c2d4a6afa76 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d870164263bb44b6b780a557daea3656, State: Initialized, Role: FOLLOWER
I20260130 02:58:15.156301 20591 raft_consensus.cc:493] T f41f4cff7dd4489f948700a45e9a7057 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:15.156992 19807 ts_tablet_manager.cc:1434] T 274530359b174ff79f046c2d4a6afa76 P 371a36d219164974abdf848841528152: Time spent starting tablet: real 0.020s user 0.006s sys 0.001s
I20260130 02:58:15.157141 19625 consensus_queue.cc:260] T 274530359b174ff79f046c2d4a6afa76 P d870164263bb44b6b780a557daea3656 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:15.157235 20591 raft_consensus.cc:515] T f41f4cff7dd4489f948700a45e9a7057 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:15.159070 19625 ts_tablet_manager.cc:1434] T 274530359b174ff79f046c2d4a6afa76 P d870164263bb44b6b780a557daea3656: Time spent starting tablet: real 0.012s user 0.006s sys 0.000s
I20260130 02:58:15.159865 19625 tablet_bootstrap.cc:492] T 7783db3568f4434e91e9e456799cc312 P d870164263bb44b6b780a557daea3656: Bootstrap starting.
I20260130 02:58:15.163172 20591 leader_election.cc:290] T f41f4cff7dd4489f948700a45e9a7057 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767)
I20260130 02:58:15.165179 19378 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f41f4cff7dd4489f948700a45e9a7057" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74" is_pre_election: true
I20260130 02:58:15.165607 19225 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f41f4cff7dd4489f948700a45e9a7057" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152" is_pre_election: true
I20260130 02:58:15.165760 19378 raft_consensus.cc:2468] T f41f4cff7dd4489f948700a45e9a7057 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d870164263bb44b6b780a557daea3656 in term 0.
W20260130 02:58:15.166970 19463 leader_election.cc:343] T f41f4cff7dd4489f948700a45e9a7057 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:15.169257 19462 leader_election.cc:304] T f41f4cff7dd4489f948700a45e9a7057 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 78c2c46c7520492db35d51c244fb4f74, d870164263bb44b6b780a557daea3656; no voters: 371a36d219164974abdf848841528152
I20260130 02:58:15.173905 19625 tablet_bootstrap.cc:654] T 7783db3568f4434e91e9e456799cc312 P d870164263bb44b6b780a557daea3656: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:15.151654 18860 catalog_manager.cc:4985] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767): tablet d55291d7cea64a0d9a345ce712fbfa80 (table default.Table_b85cb876f25740bfaaab01949ac26b38 [id=839e292e040c4aeb832f3fc0258c32ec]) successfully deleted
W20260130 02:58:15.175491 18860 catalog_manager.cc:4965] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767): delete failed for tablet 2c79a3aecab844698e70834999e1caac because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2c79a3aecab844698e70834999e1caac already in progress: creating tablet
I20260130 02:58:15.175786 20562 raft_consensus.cc:2804] T f41f4cff7dd4489f948700a45e9a7057 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260130 02:58:15.176158 20562 raft_consensus.cc:493] T f41f4cff7dd4489f948700a45e9a7057 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260130 02:58:15.176463 20562 raft_consensus.cc:3060] T f41f4cff7dd4489f948700a45e9a7057 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:15.180776 19807 tablet_bootstrap.cc:492] T f41f4cff7dd4489f948700a45e9a7057 P 371a36d219164974abdf848841528152: Bootstrap starting.
I20260130 02:58:15.182235 20571 ts_tablet_manager.cc:1929] T 8bd3e1c06b89426da57e2b259e7983af P d870164263bb44b6b780a557daea3656: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260130 02:58:15.184746 20571 log.cc:1199] T 8bd3e1c06b89426da57e2b259e7983af P d870164263bb44b6b780a557daea3656: Deleting WAL directory at /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-2/wal/wals/8bd3e1c06b89426da57e2b259e7983af
I20260130 02:58:15.186540 19807 tablet_bootstrap.cc:654] T f41f4cff7dd4489f948700a45e9a7057 P 371a36d219164974abdf848841528152: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:15.186866 20571 ts_tablet_manager.cc:1950] T 8bd3e1c06b89426da57e2b259e7983af P d870164263bb44b6b780a557daea3656: Deleting consensus metadata
I20260130 02:58:15.198961 19625 tablet_bootstrap.cc:492] T 7783db3568f4434e91e9e456799cc312 P d870164263bb44b6b780a557daea3656: No bootstrap required, opened a new log
I20260130 02:58:15.200137 19625 ts_tablet_manager.cc:1403] T 7783db3568f4434e91e9e456799cc312 P d870164263bb44b6b780a557daea3656: Time spent bootstrapping tablet: real 0.040s user 0.003s sys 0.009s
I20260130 02:58:15.202337 19625 raft_consensus.cc:359] T 7783db3568f4434e91e9e456799cc312 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:15.202960 19625 raft_consensus.cc:385] T 7783db3568f4434e91e9e456799cc312 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:15.203246 19625 raft_consensus.cc:740] T 7783db3568f4434e91e9e456799cc312 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d870164263bb44b6b780a557daea3656, State: Initialized, Role: FOLLOWER
I20260130 02:58:15.205212 20571 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59068 (request call id 256) took 1849 ms. Trace:
I20260130 02:58:15.205557 20571 rpcz_store.cc:276] 0130 02:58:13.355848 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:58:13.370349 (+ 14501us) service_pool.cc:225] Handling call
0130 02:58:15.127350 (+1757001us) ts_tablet_manager.cc:1151] Deleting tablet 8bd3e1c06b89426da57e2b259e7983af
0130 02:58:15.127393 (+ 43us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0130 02:58:15.168770 (+ 41377us) tablet_metadata.cc:641] Metadata flushed
0130 02:58:15.181974 (+ 13204us) tablet_metadata.cc:641] Metadata flushed
0130 02:58:15.205152 (+ 23178us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1756385}
I20260130 02:58:15.206980 19625 consensus_queue.cc:260] T 7783db3568f4434e91e9e456799cc312 P d870164263bb44b6b780a557daea3656 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:15.210278 18860 catalog_manager.cc:4985] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): tablet 8bd3e1c06b89426da57e2b259e7983af (table default.Table_1d9d1f3b2b6048f8949fdba26f5fd3d7 [id=1b409bcef4ca4d248418597c1aa69fdb]) successfully deleted
I20260130 02:58:15.220350 20571 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59068 (request call id 264) took 1736 ms. Trace:
I20260130 02:58:15.220722 20571 rpcz_store.cc:276] 0130 02:58:13.483486 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:58:13.483615 (+ 129us) service_pool.cc:225] Handling call
0130 02:58:15.220041 (+1736426us) ts_tablet_manager.cc:1151] Deleting tablet 730bd1d7d15b4cbe945b3ef03b2455f7
0130 02:58:15.220098 (+ 57us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0130 02:58:15.220280 (+ 182us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1735813}
W20260130 02:58:15.221951 18860 catalog_manager.cc:4952] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): delete failed for tablet 730bd1d7d15b4cbe945b3ef03b2455f7 because the tablet was not found. No further retry: Not found: Tablet not found: 730bd1d7d15b4cbe945b3ef03b2455f7
I20260130 02:58:15.226851 20608 raft_consensus.cc:493] T b1e8b78eff0d4a0b92059adb2de4a3ce P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:15.227346 20608 raft_consensus.cc:515] T b1e8b78eff0d4a0b92059adb2de4a3ce P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:15.229804 19625 ts_tablet_manager.cc:1434] T 7783db3568f4434e91e9e456799cc312 P d870164263bb44b6b780a557daea3656: Time spent starting tablet: real 0.029s user 0.005s sys 0.001s
I20260130 02:58:15.231115 19625 tablet_bootstrap.cc:492] T 6e12a8eeb3204325a9a8b7779f72ab29 P d870164263bb44b6b780a557daea3656: Bootstrap starting.
I20260130 02:58:15.232744 19378 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b1e8b78eff0d4a0b92059adb2de4a3ce" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74" is_pre_election: true
I20260130 02:58:15.232095 19526 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b1e8b78eff0d4a0b92059adb2de4a3ce" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
I20260130 02:58:15.233407 19526 raft_consensus.cc:2468] T b1e8b78eff0d4a0b92059adb2de4a3ce P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 371a36d219164974abdf848841528152 in term 0.
I20260130 02:58:15.235635 19159 leader_election.cc:304] T b1e8b78eff0d4a0b92059adb2de4a3ce P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 371a36d219164974abdf848841528152, d870164263bb44b6b780a557daea3656; no voters:
I20260130 02:58:15.236249 19807 tablet_bootstrap.cc:492] T f41f4cff7dd4489f948700a45e9a7057 P 371a36d219164974abdf848841528152: No bootstrap required, opened a new log
I20260130 02:58:15.236398 20497 raft_consensus.cc:2804] T b1e8b78eff0d4a0b92059adb2de4a3ce P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260130 02:58:15.236694 20497 raft_consensus.cc:493] T b1e8b78eff0d4a0b92059adb2de4a3ce P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260130 02:58:15.237030 20497 raft_consensus.cc:3060] T b1e8b78eff0d4a0b92059adb2de4a3ce P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:15.230640 20608 leader_election.cc:290] T b1e8b78eff0d4a0b92059adb2de4a3ce P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767), d870164263bb44b6b780a557daea3656 (127.18.100.67:39571)
I20260130 02:58:15.239526 19378 raft_consensus.cc:2468] T b1e8b78eff0d4a0b92059adb2de4a3ce P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 371a36d219164974abdf848841528152 in term 0.
I20260130 02:58:15.240754 19625 tablet_bootstrap.cc:654] T 6e12a8eeb3204325a9a8b7779f72ab29 P d870164263bb44b6b780a557daea3656: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:15.244297 20497 raft_consensus.cc:515] T b1e8b78eff0d4a0b92059adb2de4a3ce P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:15.246269 20497 leader_election.cc:290] T b1e8b78eff0d4a0b92059adb2de4a3ce P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 election: Requested vote from peers 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767), d870164263bb44b6b780a557daea3656 (127.18.100.67:39571)
I20260130 02:58:15.236689 19807 ts_tablet_manager.cc:1403] T f41f4cff7dd4489f948700a45e9a7057 P 371a36d219164974abdf848841528152: Time spent bootstrapping tablet: real 0.056s user 0.014s sys 0.004s
I20260130 02:58:15.248252 19378 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b1e8b78eff0d4a0b92059adb2de4a3ce" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74"
I20260130 02:58:15.248829 19378 raft_consensus.cc:3060] T b1e8b78eff0d4a0b92059adb2de4a3ce P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:15.249974 19526 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b1e8b78eff0d4a0b92059adb2de4a3ce" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656"
I20260130 02:58:15.250485 19526 raft_consensus.cc:3060] T b1e8b78eff0d4a0b92059adb2de4a3ce P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:15.254220 19807 raft_consensus.cc:359] T f41f4cff7dd4489f948700a45e9a7057 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:15.254854 19807 raft_consensus.cc:385] T f41f4cff7dd4489f948700a45e9a7057 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:15.255095 19807 raft_consensus.cc:740] T f41f4cff7dd4489f948700a45e9a7057 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 371a36d219164974abdf848841528152, State: Initialized, Role: FOLLOWER
I20260130 02:58:15.255643 19807 consensus_queue.cc:260] T f41f4cff7dd4489f948700a45e9a7057 P 371a36d219164974abdf848841528152 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:15.257640 19807 ts_tablet_manager.cc:1434] T f41f4cff7dd4489f948700a45e9a7057 P 371a36d219164974abdf848841528152: Time spent starting tablet: real 0.011s user 0.003s sys 0.000s
I20260130 02:58:15.261987 19625 tablet_bootstrap.cc:492] T 6e12a8eeb3204325a9a8b7779f72ab29 P d870164263bb44b6b780a557daea3656: No bootstrap required, opened a new log
I20260130 02:58:15.262408 19625 ts_tablet_manager.cc:1403] T 6e12a8eeb3204325a9a8b7779f72ab29 P d870164263bb44b6b780a557daea3656: Time spent bootstrapping tablet: real 0.031s user 0.006s sys 0.009s
I20260130 02:58:15.258455 19807 tablet_bootstrap.cc:492] T 6e12a8eeb3204325a9a8b7779f72ab29 P 371a36d219164974abdf848841528152: Bootstrap starting.
I20260130 02:58:15.273504 19625 raft_consensus.cc:359] T 6e12a8eeb3204325a9a8b7779f72ab29 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:15.274115 19625 raft_consensus.cc:385] T 6e12a8eeb3204325a9a8b7779f72ab29 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:15.274343 19625 raft_consensus.cc:740] T 6e12a8eeb3204325a9a8b7779f72ab29 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d870164263bb44b6b780a557daea3656, State: Initialized, Role: FOLLOWER
I20260130 02:58:15.275365 19625 consensus_queue.cc:260] T 6e12a8eeb3204325a9a8b7779f72ab29 P d870164263bb44b6b780a557daea3656 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:15.277649 19526 raft_consensus.cc:2468] T b1e8b78eff0d4a0b92059adb2de4a3ce P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 371a36d219164974abdf848841528152 in term 1.
I20260130 02:58:15.232174 20571 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59068 (request call id 265) took 1747 ms. Trace:
I20260130 02:58:15.284258 19840 tablet_bootstrap.cc:492] T 838e49e5e8874db393bdc15a1c92f87e P 78c2c46c7520492db35d51c244fb4f74: No bootstrap required, opened a new log
I20260130 02:58:15.284675 19840 ts_tablet_manager.cc:1403] T 838e49e5e8874db393bdc15a1c92f87e P 78c2c46c7520492db35d51c244fb4f74: Time spent bootstrapping tablet: real 0.144s user 0.006s sys 0.023s
I20260130 02:58:15.287083 19840 raft_consensus.cc:359] T 838e49e5e8874db393bdc15a1c92f87e P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:15.287696 19840 raft_consensus.cc:385] T 838e49e5e8874db393bdc15a1c92f87e P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:15.287938 19840 raft_consensus.cc:740] T 838e49e5e8874db393bdc15a1c92f87e P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78c2c46c7520492db35d51c244fb4f74, State: Initialized, Role: FOLLOWER
I20260130 02:58:15.292724 19159 leader_election.cc:304] T b1e8b78eff0d4a0b92059adb2de4a3ce P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 371a36d219164974abdf848841528152, d870164263bb44b6b780a557daea3656; no voters:
I20260130 02:58:15.296943 19840 consensus_queue.cc:260] T 838e49e5e8874db393bdc15a1c92f87e P 78c2c46c7520492db35d51c244fb4f74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:15.278645 20562 raft_consensus.cc:515] T f41f4cff7dd4489f948700a45e9a7057 P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:15.282711 20571 rpcz_store.cc:276] 0130 02:58:13.484888 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:58:13.485002 (+ 114us) service_pool.cc:225] Handling call
0130 02:58:15.231893 (+1746891us) ts_tablet_manager.cc:1151] Deleting tablet 3cf5154d8cc44f2bb12978202699c193
0130 02:58:15.231939 (+ 46us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0130 02:58:15.232118 (+ 179us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1746412}
I20260130 02:58:15.285084 19625 ts_tablet_manager.cc:1434] T 6e12a8eeb3204325a9a8b7779f72ab29 P d870164263bb44b6b780a557daea3656: Time spent starting tablet: real 0.021s user 0.005s sys 0.000s
I20260130 02:58:15.306743 19625 tablet_bootstrap.cc:492] T 03025d13f0e545cfa6288677a2715343 P d870164263bb44b6b780a557daea3656: Bootstrap starting.
I20260130 02:58:15.315733 20571 tablet_replica.cc:333] T 7d96e93d95f14af79cfb268626559b1e P d870164263bb44b6b780a557daea3656: stopping tablet replica
W20260130 02:58:15.316125 18860 catalog_manager.cc:4952] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): delete failed for tablet 3cf5154d8cc44f2bb12978202699c193 because the tablet was not found. No further retry: Not found: Tablet not found: 3cf5154d8cc44f2bb12978202699c193
I20260130 02:58:15.316782 20571 raft_consensus.cc:2243] T 7d96e93d95f14af79cfb268626559b1e P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260130 02:58:15.317337 20571 raft_consensus.cc:2272] T 7d96e93d95f14af79cfb268626559b1e P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260130 02:58:15.319321 20571 ts_tablet_manager.cc:1916] T 7d96e93d95f14af79cfb268626559b1e P d870164263bb44b6b780a557daea3656: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260130 02:58:15.304879 19807 tablet_bootstrap.cc:654] T 6e12a8eeb3204325a9a8b7779f72ab29 P 371a36d219164974abdf848841528152: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:15.305442 19225 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f41f4cff7dd4489f948700a45e9a7057" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152"
I20260130 02:58:15.320135 19225 raft_consensus.cc:3060] T f41f4cff7dd4489f948700a45e9a7057 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:15.322748 20497 raft_consensus.cc:2804] T b1e8b78eff0d4a0b92059adb2de4a3ce P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Leader election won for term 1
I20260130 02:58:15.323194 20497 raft_consensus.cc:697] T b1e8b78eff0d4a0b92059adb2de4a3ce P 371a36d219164974abdf848841528152 [term 1 LEADER]: Becoming Leader. State: Replica: 371a36d219164974abdf848841528152, State: Running, Role: LEADER
I20260130 02:58:15.302539 19378 raft_consensus.cc:2468] T b1e8b78eff0d4a0b92059adb2de4a3ce P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 371a36d219164974abdf848841528152 in term 1.
I20260130 02:58:15.324013 20497 consensus_queue.cc:237] T b1e8b78eff0d4a0b92059adb2de4a3ce P 371a36d219164974abdf848841528152 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
W20260130 02:58:15.324646 18860 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.18.100.124:34629: connect: Connection refused (error 111) [suppressed 6 similar messages]
I20260130 02:58:15.333262 19225 raft_consensus.cc:2468] T f41f4cff7dd4489f948700a45e9a7057 P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d870164263bb44b6b780a557daea3656 in term 1.
I20260130 02:58:15.334336 19463 leader_election.cc:304] T f41f4cff7dd4489f948700a45e9a7057 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 371a36d219164974abdf848841528152, d870164263bb44b6b780a557daea3656; no voters:
I20260130 02:58:15.335098 20591 raft_consensus.cc:2804] T f41f4cff7dd4489f948700a45e9a7057 P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Leader election won for term 1
I20260130 02:58:15.322221 19380 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f41f4cff7dd4489f948700a45e9a7057" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74"
I20260130 02:58:15.335534 19380 raft_consensus.cc:3060] T f41f4cff7dd4489f948700a45e9a7057 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:15.320999 20562 leader_election.cc:290] T f41f4cff7dd4489f948700a45e9a7057 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 election: Requested vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767)
I20260130 02:58:15.324610 19625 tablet_bootstrap.cc:654] T 03025d13f0e545cfa6288677a2715343 P d870164263bb44b6b780a557daea3656: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:15.335520 20591 raft_consensus.cc:697] T f41f4cff7dd4489f948700a45e9a7057 P d870164263bb44b6b780a557daea3656 [term 1 LEADER]: Becoming Leader. State: Replica: d870164263bb44b6b780a557daea3656, State: Running, Role: LEADER
I20260130 02:58:15.337849 20591 consensus_queue.cc:237] T f41f4cff7dd4489f948700a45e9a7057 P d870164263bb44b6b780a557daea3656 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:15.308218 19840 ts_tablet_manager.cc:1434] T 838e49e5e8874db393bdc15a1c92f87e P 78c2c46c7520492db35d51c244fb4f74: Time spent starting tablet: real 0.023s user 0.004s sys 0.002s
I20260130 02:58:15.344717 19840 tablet_bootstrap.cc:492] T e12d130bcdf34965888753f12c1ff848 P 78c2c46c7520492db35d51c244fb4f74: Bootstrap starting.
I20260130 02:58:15.342315 19380 raft_consensus.cc:2468] T f41f4cff7dd4489f948700a45e9a7057 P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d870164263bb44b6b780a557daea3656 in term 1.
I20260130 02:58:15.366191 19204 tablet_service.cc:1505] Processing CreateTablet for tablet cfd12a72726846e587cff9d6a4d61bd6 (DEFAULT_TABLE table=default.Table_4ee03db7d0064d96a53d4f261bd12722 [id=07fe2931234d4888a524b52171b03f0a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:15.367233 19204 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cfd12a72726846e587cff9d6a4d61bd6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:15.367910 19625 tablet_bootstrap.cc:492] T 03025d13f0e545cfa6288677a2715343 P d870164263bb44b6b780a557daea3656: No bootstrap required, opened a new log
I20260130 02:58:15.368474 19625 ts_tablet_manager.cc:1403] T 03025d13f0e545cfa6288677a2715343 P d870164263bb44b6b780a557daea3656: Time spent bootstrapping tablet: real 0.062s user 0.009s sys 0.002s
I20260130 02:58:15.370713 19625 raft_consensus.cc:359] T 03025d13f0e545cfa6288677a2715343 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:15.371485 19840 tablet_bootstrap.cc:654] T e12d130bcdf34965888753f12c1ff848 P 78c2c46c7520492db35d51c244fb4f74: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:15.371464 19625 raft_consensus.cc:385] T 03025d13f0e545cfa6288677a2715343 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:15.371790 19496 tablet_service.cc:1505] Processing CreateTablet for tablet cfd12a72726846e587cff9d6a4d61bd6 (DEFAULT_TABLE table=default.Table_4ee03db7d0064d96a53d4f261bd12722 [id=07fe2931234d4888a524b52171b03f0a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:15.371803 19625 raft_consensus.cc:740] T 03025d13f0e545cfa6288677a2715343 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d870164263bb44b6b780a557daea3656, State: Initialized, Role: FOLLOWER
I20260130 02:58:15.374537 19355 tablet_service.cc:1505] Processing CreateTablet for tablet cfd12a72726846e587cff9d6a4d61bd6 (DEFAULT_TABLE table=default.Table_4ee03db7d0064d96a53d4f261bd12722 [id=07fe2931234d4888a524b52171b03f0a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:15.373608 19625 consensus_queue.cc:260] T 03025d13f0e545cfa6288677a2715343 P d870164263bb44b6b780a557daea3656 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:15.379271 19496 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cfd12a72726846e587cff9d6a4d61bd6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:15.382344 19625 ts_tablet_manager.cc:1434] T 03025d13f0e545cfa6288677a2715343 P d870164263bb44b6b780a557daea3656: Time spent starting tablet: real 0.014s user 0.004s sys 0.000s
I20260130 02:58:15.383307 19840 tablet_bootstrap.cc:492] T e12d130bcdf34965888753f12c1ff848 P 78c2c46c7520492db35d51c244fb4f74: No bootstrap required, opened a new log
I20260130 02:58:15.383705 19840 ts_tablet_manager.cc:1403] T e12d130bcdf34965888753f12c1ff848 P 78c2c46c7520492db35d51c244fb4f74: Time spent bootstrapping tablet: real 0.039s user 0.006s sys 0.004s
I20260130 02:58:15.384986 19872 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260130 02:58:15.388916 19355 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cfd12a72726846e587cff9d6a4d61bd6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:15.392840 19872 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260130 02:58:15.393483 19872 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260130 02:58:15.395290 19840 raft_consensus.cc:359] T e12d130bcdf34965888753f12c1ff848 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:15.395960 19840 raft_consensus.cc:385] T e12d130bcdf34965888753f12c1ff848 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:15.396200 19840 raft_consensus.cc:740] T e12d130bcdf34965888753f12c1ff848 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78c2c46c7520492db35d51c244fb4f74, State: Initialized, Role: FOLLOWER
I20260130 02:58:15.396152 19502 tablet_service.cc:1505] Processing CreateTablet for tablet cd30d7adcc4545848e3501a205a10293 (DEFAULT_TABLE table=default.Table_b85cb876f25740bfaaab01949ac26b38 [id=839e292e040c4aeb832f3fc0258c32ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:15.397084 19502 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cd30d7adcc4545848e3501a205a10293. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:15.399456 19202 tablet_service.cc:1505] Processing CreateTablet for tablet cd30d7adcc4545848e3501a205a10293 (DEFAULT_TABLE table=default.Table_b85cb876f25740bfaaab01949ac26b38 [id=839e292e040c4aeb832f3fc0258c32ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:15.400370 19202 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cd30d7adcc4545848e3501a205a10293. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:15.401832 19350 tablet_service.cc:1505] Processing CreateTablet for tablet cd30d7adcc4545848e3501a205a10293 (DEFAULT_TABLE table=default.Table_b85cb876f25740bfaaab01949ac26b38 [id=839e292e040c4aeb832f3fc0258c32ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:15.402753 19350 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cd30d7adcc4545848e3501a205a10293. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:15.383244 19625 tablet_bootstrap.cc:492] T a90712651a604b83ba187f893ab17711 P d870164263bb44b6b780a557daea3656: Bootstrap starting.
I20260130 02:58:15.406904 19840 consensus_queue.cc:260] T e12d130bcdf34965888753f12c1ff848 P 78c2c46c7520492db35d51c244fb4f74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:15.409780 19807 tablet_bootstrap.cc:492] T 6e12a8eeb3204325a9a8b7779f72ab29 P 371a36d219164974abdf848841528152: No bootstrap required, opened a new log
I20260130 02:58:15.410177 19807 ts_tablet_manager.cc:1403] T 6e12a8eeb3204325a9a8b7779f72ab29 P 371a36d219164974abdf848841528152: Time spent bootstrapping tablet: real 0.152s user 0.008s sys 0.008s
I20260130 02:58:15.412285 19807 raft_consensus.cc:359] T 6e12a8eeb3204325a9a8b7779f72ab29 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:15.412881 19807 raft_consensus.cc:385] T 6e12a8eeb3204325a9a8b7779f72ab29 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:15.412817 19197 tablet_service.cc:1505] Processing CreateTablet for tablet c76f0dc7762c49c389fa6fef41f02141 (DEFAULT_TABLE table=default.Table_e894e5560f194692bac730600c6bf5f4 [id=d5e1b2a45122429aabb4fd8653d71f0f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:15.413172 19807 raft_consensus.cc:740] T 6e12a8eeb3204325a9a8b7779f72ab29 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 371a36d219164974abdf848841528152, State: Initialized, Role: FOLLOWER
I20260130 02:58:15.413975 19197 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c76f0dc7762c49c389fa6fef41f02141. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:15.416000 19840 ts_tablet_manager.cc:1434] T e12d130bcdf34965888753f12c1ff848 P 78c2c46c7520492db35d51c244fb4f74: Time spent starting tablet: real 0.032s user 0.006s sys 0.000s
I20260130 02:58:15.416831 19840 tablet_bootstrap.cc:492] T 13f0ace4d9de43abab08eca32cdb30c1 P 78c2c46c7520492db35d51c244fb4f74: Bootstrap starting.
I20260130 02:58:15.423710 19807 consensus_queue.cc:260] T 6e12a8eeb3204325a9a8b7779f72ab29 P 371a36d219164974abdf848841528152 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:15.425634 18875 catalog_manager.cc:5746] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021) does not have the latest schema for tablet a8a5038e9def44eb8b3b1e4aeae25652 (table default.Table_dee034aec0b24982b8dc2a3fa1b326f1 [id=66033e75b3814a8584311cf7e4a07ad3]). Expected version 1 got 0
I20260130 02:58:15.427747 18875 catalog_manager.cc:5746] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021) does not have the latest schema for tablet 16691c9315e94253bb9b3fb5058e5636 (table default.Table_8f311c5878744ef692b496ecfdfc3ede [id=4f499357303d464aafc17679ffcaaa07]). Expected version 1 got 0
I20260130 02:58:15.438797 19350 tablet_service.cc:1505] Processing CreateTablet for tablet c76f0dc7762c49c389fa6fef41f02141 (DEFAULT_TABLE table=default.Table_e894e5560f194692bac730600c6bf5f4 [id=d5e1b2a45122429aabb4fd8653d71f0f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:15.440250 20629 raft_consensus.cc:493] T e12d130bcdf34965888753f12c1ff848 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:15.440744 19350 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c76f0dc7762c49c389fa6fef41f02141. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:15.440760 20629 raft_consensus.cc:515] T e12d130bcdf34965888753f12c1ff848 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:15.443220 20629 leader_election.cc:290] T e12d130bcdf34965888753f12c1ff848 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d870164263bb44b6b780a557daea3656 (127.18.100.67:39571), 371a36d219164974abdf848841528152 (127.18.100.65:42021)
I20260130 02:58:15.443673 19225 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e12d130bcdf34965888753f12c1ff848" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152" is_pre_election: true
W20260130 02:58:15.444761 19314 leader_election.cc:343] T e12d130bcdf34965888753f12c1ff848 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:15.450760 19807 ts_tablet_manager.cc:1434] T 6e12a8eeb3204325a9a8b7779f72ab29 P 371a36d219164974abdf848841528152: Time spent starting tablet: real 0.040s user 0.006s sys 0.001s
I20260130 02:58:15.451586 19496 tablet_service.cc:1505] Processing CreateTablet for tablet c76f0dc7762c49c389fa6fef41f02141 (DEFAULT_TABLE table=default.Table_e894e5560f194692bac730600c6bf5f4 [id=d5e1b2a45122429aabb4fd8653d71f0f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:15.452433 19526 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e12d130bcdf34965888753f12c1ff848" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
I20260130 02:58:15.454105 19496 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c76f0dc7762c49c389fa6fef41f02141. 1 dirs total, 0 dirs full, 0 dirs failed
W20260130 02:58:15.455886 19313 leader_election.cc:343] T e12d130bcdf34965888753f12c1ff848 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:15.456311 19313 leader_election.cc:304] T e12d130bcdf34965888753f12c1ff848 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78c2c46c7520492db35d51c244fb4f74; no voters: 371a36d219164974abdf848841528152, d870164263bb44b6b780a557daea3656
I20260130 02:58:15.458078 20611 raft_consensus.cc:2749] T e12d130bcdf34965888753f12c1ff848 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260130 02:58:15.461232 20611 raft_consensus.cc:493] T 838e49e5e8874db393bdc15a1c92f87e P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:15.461612 20611 raft_consensus.cc:515] T 838e49e5e8874db393bdc15a1c92f87e P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:15.463356 20611 leader_election.cc:290] T 838e49e5e8874db393bdc15a1c92f87e P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), d870164263bb44b6b780a557daea3656 (127.18.100.67:39571)
I20260130 02:58:15.463744 19807 tablet_bootstrap.cc:492] T 78e9bc3e55fe4e4a98e8eb01da59d365 P 371a36d219164974abdf848841528152: Bootstrap starting.
I20260130 02:58:15.465309 19225 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "838e49e5e8874db393bdc15a1c92f87e" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152" is_pre_election: true
W20260130 02:58:15.467759 19314 leader_election.cc:343] T 838e49e5e8874db393bdc15a1c92f87e P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:15.477219 19840 tablet_bootstrap.cc:654] T 13f0ace4d9de43abab08eca32cdb30c1 P 78c2c46c7520492db35d51c244fb4f74: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:15.477830 18875 catalog_manager.cc:5746] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021) does not have the latest schema for tablet 345eaaa5625c48d099280b7fbb2e514b (table default.Table_c1886dd9a7934e7797e0e7e0b065d692 [id=7a2d5367224b492b844199088b01b149]). Expected version 1 got 0
I20260130 02:58:15.500628 19807 tablet_bootstrap.cc:654] T 78e9bc3e55fe4e4a98e8eb01da59d365 P 371a36d219164974abdf848841528152: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:15.501842 19625 tablet_bootstrap.cc:654] T a90712651a604b83ba187f893ab17711 P d870164263bb44b6b780a557daea3656: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:15.503885 20571 ts_tablet_manager.cc:1929] T 7d96e93d95f14af79cfb268626559b1e P d870164263bb44b6b780a557daea3656: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260130 02:58:15.504218 20571 log.cc:1199] T 7d96e93d95f14af79cfb268626559b1e P d870164263bb44b6b780a557daea3656: Deleting WAL directory at /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-2/wal/wals/7d96e93d95f14af79cfb268626559b1e
I20260130 02:58:15.505688 19526 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "838e49e5e8874db393bdc15a1c92f87e" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
W20260130 02:58:15.506721 19313 leader_election.cc:343] T 838e49e5e8874db393bdc15a1c92f87e P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:15.507109 19313 leader_election.cc:304] T 838e49e5e8874db393bdc15a1c92f87e P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78c2c46c7520492db35d51c244fb4f74; no voters: 371a36d219164974abdf848841528152, d870164263bb44b6b780a557daea3656
I20260130 02:58:15.507776 20629 raft_consensus.cc:2749] T 838e49e5e8874db393bdc15a1c92f87e P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260130 02:58:15.509258 20571 ts_tablet_manager.cc:1950] T 7d96e93d95f14af79cfb268626559b1e P d870164263bb44b6b780a557daea3656: Deleting consensus metadata
I20260130 02:58:15.510133 19840 tablet_bootstrap.cc:492] T 13f0ace4d9de43abab08eca32cdb30c1 P 78c2c46c7520492db35d51c244fb4f74: No bootstrap required, opened a new log
I20260130 02:58:15.510536 19840 ts_tablet_manager.cc:1403] T 13f0ace4d9de43abab08eca32cdb30c1 P 78c2c46c7520492db35d51c244fb4f74: Time spent bootstrapping tablet: real 0.094s user 0.003s sys 0.005s
I20260130 02:58:15.511348 18875 catalog_manager.cc:5746] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021) does not have the latest schema for tablet 9fe0448bcc4a4346b3c67fc3f626ed3c (table default.Table_5c1c447e85524ac5a61ffd8a48509d36 [id=70aaf21ad9bd4d959f46939c6b4a4880]). Expected version 1 got 0
I20260130 02:58:15.512688 19840 raft_consensus.cc:359] T 13f0ace4d9de43abab08eca32cdb30c1 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:15.513289 19840 raft_consensus.cc:385] T 13f0ace4d9de43abab08eca32cdb30c1 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:15.513542 19840 raft_consensus.cc:740] T 13f0ace4d9de43abab08eca32cdb30c1 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78c2c46c7520492db35d51c244fb4f74, State: Initialized, Role: FOLLOWER
I20260130 02:58:15.513598 20591 raft_consensus.cc:493] T 03025d13f0e545cfa6288677a2715343 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:15.516366 20591 raft_consensus.cc:515] T 03025d13f0e545cfa6288677a2715343 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:15.519203 20571 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59068 (request call id 266) took 2018 ms. Trace:
I20260130 02:58:15.521143 20571 rpcz_store.cc:276] 0130 02:58:13.500410 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:58:13.500527 (+ 117us) service_pool.cc:225] Handling call
0130 02:58:15.315584 (+1815057us) ts_tablet_manager.cc:1151] Deleting tablet 7d96e93d95f14af79cfb268626559b1e
0130 02:58:15.315626 (+ 42us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0130 02:58:15.361443 (+ 45817us) tablet_metadata.cc:641] Metadata flushed
0130 02:58:15.503679 (+142236us) tablet_metadata.cc:641] Metadata flushed
0130 02:58:15.519147 (+ 15468us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1813567}
I20260130 02:58:15.522184 20571 tablet_replica.cc:333] T d55291d7cea64a0d9a345ce712fbfa80 P d870164263bb44b6b780a557daea3656: stopping tablet replica
I20260130 02:58:15.522833 19840 consensus_queue.cc:260] T 13f0ace4d9de43abab08eca32cdb30c1 P 78c2c46c7520492db35d51c244fb4f74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:15.523554 19225 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "03025d13f0e545cfa6288677a2715343" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152" is_pre_election: true
I20260130 02:58:15.523867 20591 leader_election.cc:290] T 03025d13f0e545cfa6288677a2715343 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767)
I20260130 02:58:15.524053 19225 raft_consensus.cc:2468] T 03025d13f0e545cfa6288677a2715343 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d870164263bb44b6b780a557daea3656 in term 0.
I20260130 02:58:15.525604 19463 leader_election.cc:304] T 03025d13f0e545cfa6288677a2715343 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 371a36d219164974abdf848841528152, d870164263bb44b6b780a557daea3656; no voters:
I20260130 02:58:15.525696 19380 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "03025d13f0e545cfa6288677a2715343" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74" is_pre_election: true
I20260130 02:58:15.526203 19380 raft_consensus.cc:2468] T 03025d13f0e545cfa6288677a2715343 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d870164263bb44b6b780a557daea3656 in term 0.
I20260130 02:58:15.526670 19807 tablet_bootstrap.cc:492] T 78e9bc3e55fe4e4a98e8eb01da59d365 P 371a36d219164974abdf848841528152: No bootstrap required, opened a new log
I20260130 02:58:15.527057 19807 ts_tablet_manager.cc:1403] T 78e9bc3e55fe4e4a98e8eb01da59d365 P 371a36d219164974abdf848841528152: Time spent bootstrapping tablet: real 0.063s user 0.009s sys 0.001s
I20260130 02:58:15.527411 20591 raft_consensus.cc:2804] T 03025d13f0e545cfa6288677a2715343 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260130 02:58:15.527719 20591 raft_consensus.cc:493] T 03025d13f0e545cfa6288677a2715343 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260130 02:58:15.527988 20591 raft_consensus.cc:3060] T 03025d13f0e545cfa6288677a2715343 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:15.531157 19840 ts_tablet_manager.cc:1434] T 13f0ace4d9de43abab08eca32cdb30c1 P 78c2c46c7520492db35d51c244fb4f74: Time spent starting tablet: real 0.020s user 0.004s sys 0.001s
I20260130 02:58:15.530278 19807 raft_consensus.cc:359] T 78e9bc3e55fe4e4a98e8eb01da59d365 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:15.531889 19807 raft_consensus.cc:385] T 78e9bc3e55fe4e4a98e8eb01da59d365 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:15.532131 19807 raft_consensus.cc:740] T 78e9bc3e55fe4e4a98e8eb01da59d365 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 371a36d219164974abdf848841528152, State: Initialized, Role: FOLLOWER
I20260130 02:58:15.532718 19807 consensus_queue.cc:260] T 78e9bc3e55fe4e4a98e8eb01da59d365 P 371a36d219164974abdf848841528152 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:15.534324 19840 tablet_bootstrap.cc:492] T af0f90071a334016aa17fae2e3a97fc8 P 78c2c46c7520492db35d51c244fb4f74: Bootstrap starting.
I20260130 02:58:15.535521 20679 raft_consensus.cc:493] T 7783db3568f4434e91e9e456799cc312 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:15.535948 20679 raft_consensus.cc:515] T 7783db3568f4434e91e9e456799cc312 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:15.537714 20679 leader_election.cc:290] T 7783db3568f4434e91e9e456799cc312 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767)
I20260130 02:58:15.538640 19380 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7783db3568f4434e91e9e456799cc312" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74" is_pre_election: true
I20260130 02:58:15.539287 19380 raft_consensus.cc:2393] T 7783db3568f4434e91e9e456799cc312 P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d870164263bb44b6b780a557daea3656 in current term 1: Already voted for candidate 371a36d219164974abdf848841528152 in this term.
I20260130 02:58:15.548611 20591 raft_consensus.cc:515] T 03025d13f0e545cfa6288677a2715343 P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:15.551115 19223 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7783db3568f4434e91e9e456799cc312" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152" is_pre_election: true
I20260130 02:58:15.551525 19224 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "03025d13f0e545cfa6288677a2715343" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152"
I20260130 02:58:15.551992 19224 raft_consensus.cc:3060] T 03025d13f0e545cfa6288677a2715343 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:15.557248 19463 leader_election.cc:304] T 7783db3568f4434e91e9e456799cc312 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d870164263bb44b6b780a557daea3656; no voters: 371a36d219164974abdf848841528152, 78c2c46c7520492db35d51c244fb4f74
I20260130 02:58:15.557936 20679 raft_consensus.cc:3060] T 7783db3568f4434e91e9e456799cc312 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:15.559652 19807 ts_tablet_manager.cc:1434] T 78e9bc3e55fe4e4a98e8eb01da59d365 P 371a36d219164974abdf848841528152: Time spent starting tablet: real 0.032s user 0.006s sys 0.000s
I20260130 02:58:15.564349 19807 tablet_bootstrap.cc:492] T a90712651a604b83ba187f893ab17711 P 371a36d219164974abdf848841528152: Bootstrap starting.
I20260130 02:58:15.565006 20679 raft_consensus.cc:2749] T 7783db3568f4434e91e9e456799cc312 P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260130 02:58:15.565239 19380 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "03025d13f0e545cfa6288677a2715343" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74"
I20260130 02:58:15.565482 19224 raft_consensus.cc:2468] T 03025d13f0e545cfa6288677a2715343 P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d870164263bb44b6b780a557daea3656 in term 1.
I20260130 02:58:15.565737 19380 raft_consensus.cc:3060] T 03025d13f0e545cfa6288677a2715343 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:15.522653 18860 catalog_manager.cc:4985] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): tablet 7d96e93d95f14af79cfb268626559b1e (table default.Table_6eb0a92b1926487cbf0b978d26d6a7b1 [id=de7a420d40864f55b156e149308ba12f]) successfully deleted
W20260130 02:58:15.568826 18860 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.18.100.67:39571, user_credentials={real_user=slave}} blocked reactor thread for 46196.7us
I20260130 02:58:15.569020 18875 catalog_manager.cc:5746] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021) does not have the latest schema for tablet 4c139aa8cd184f488cec4e05ef7dcada (table default.Table_e4d07634d84a4678b1c94374e3adc476 [id=fcbd7e3a7fc64d89aa740494d7e1a0f7]). Expected version 1 got 0
I20260130 02:58:15.569975 18867 catalog_manager.cc:5941] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4: Sending DeleteTablet for 3 replicas of tablet 38a7a10e36874183bf9fe55a38d829f6
I20260130 02:58:15.571233 18867 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:50284 (request call id 237) took 2226 ms. Trace:
I20260130 02:58:15.572506 19350 tablet_service.cc:1552] Processing DeleteTablet for tablet 38a7a10e36874183bf9fe55a38d829f6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:58:14 UTC) from {username='slave'} at 127.0.0.1:60110
I20260130 02:58:15.573135 19496 tablet_service.cc:1552] Processing DeleteTablet for tablet 38a7a10e36874183bf9fe55a38d829f6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:58:14 UTC) from {username='slave'} at 127.0.0.1:59068
I20260130 02:58:15.573827 20570 tablet_replica.cc:333] T 38a7a10e36874183bf9fe55a38d829f6 P 78c2c46c7520492db35d51c244fb4f74: stopping tablet replica
I20260130 02:58:15.573977 19380 raft_consensus.cc:2468] T 03025d13f0e545cfa6288677a2715343 P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d870164263bb44b6b780a557daea3656 in term 1.
I20260130 02:58:15.571517 18867 rpcz_store.cc:276] 0130 02:58:13.344879 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:58:14.486916 (+1142037us) service_pool.cc:225] Handling call
0130 02:58:14.556484 (+ 69568us) spinlock_profiling.cc:243] Waited 68 ms on lock 0x7b5400030554. stack: 00007fb0bd3f7373 00007fb0bd3f7270 00007fb0bcf6c574 0000000000508744 00000000005086d9 00007fb0c44490c0 00007fb0c44e7bba 00007fb0c61e6eac 00007fb0c62e2966 00007fb0c3f0a024 00007fb0c3f09fb2 00007fb0c3f09ee1 00007fb0c3f09e5c 00007fb0c3f09112 00007fb0c21dd34c 00007fb0c21dc786
0130 02:58:14.615173 (+ 58689us) catalog_manager.cc:2393] Looking up, locking, and authorizing table
0130 02:58:14.616666 (+ 1493us) catalog_manager.cc:2815] Modifying in-memory table state
0130 02:58:14.616796 (+ 130us) catalog_manager.cc:2824] Locking tablets
0130 02:58:14.860366 (+243570us) catalog_manager.cc:2838] Removing table and tablets from system table
0130 02:58:14.878399 (+ 18033us) catalog_manager.cc:2856] Removing table from by-name map
0130 02:58:14.878620 (+ 221us) catalog_manager.cc:2872] Committing in-memory state
0130 02:58:15.569889 (+691269us) catalog_manager.cc:2876] Aborting table tasks
0130 02:58:15.571179 (+ 1290us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0130 02:58:14.862145 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0130 02:58:14.862625 (+ 480us) write_op.cc:432] Acquiring schema lock in shared mode
0130 02:58:14.862654 (+ 29us) write_op.cc:435] Acquired schema lock
0130 02:58:14.862666 (+ 12us) tablet.cc:660] Decoding operations
0130 02:58:14.863575 (+ 909us) write_op.cc:620] Acquiring the partition lock for write op
0130 02:58:14.863626 (+ 51us) write_op.cc:641] Partition lock acquired for write op
0130 02:58:14.863652 (+ 26us) tablet.cc:683] Acquiring locks for 2 operations
0130 02:58:14.863956 (+ 304us) tablet.cc:699] Row locks acquired
0130 02:58:14.863969 (+ 13us) write_op.cc:260] PREPARE: finished
0130 02:58:14.864101 (+ 132us) write_op.cc:270] Start()
0130 02:58:14.864188 (+ 87us) write_op.cc:276] Timestamp: P: 1769741894864031 usec, L: 0
0130 02:58:14.864206 (+ 18us) op_driver.cc:348] REPLICATION: starting
0130 02:58:14.864731 (+ 525us) log.cc:844] Serialized 771 byte log entry
0130 02:58:14.876285 (+ 11554us) op_driver.cc:464] REPLICATION: finished
0130 02:58:14.876789 (+ 504us) write_op.cc:301] APPLY: starting
0130 02:58:14.876852 (+ 63us) tablet.cc:1365] starting BulkCheckPresence
0130 02:58:14.876951 (+ 99us) tablet.cc:1368] finished BulkCheckPresence
0130 02:58:14.877046 (+ 95us) tablet.cc:1370] starting ApplyRowOperation cycle
0130 02:58:14.877293 (+ 247us) tablet.cc:1381] finished ApplyRowOperation cycle
0130 02:58:14.877426 (+ 133us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0130 02:58:14.877469 (+ 43us) write_op.cc:312] APPLY: finished
0130 02:58:14.877645 (+ 176us) log.cc:844] Serialized 29 byte log entry
0130 02:58:14.877782 (+ 137us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:58:14.877850 (+ 68us) write_op.cc:454] Released schema lock
0130 02:58:14.877879 (+ 29us) write_op.cc:341] FINISH: Updating metrics
0130 02:58:14.878091 (+ 212us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:58:14.878113 (+ 22us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":156307712,"child_traces":[["op",{"apply.queue_time_us":153,"apply.run_cpu_time_us":1479,"apply.run_wall_time_us":1479,"num_ops":2,"prepare.queue_time_us":116,"prepare.run_cpu_time_us":2870,"prepare.run_wall_time_us":2870,"raft.queue_time_us":4337,"raft.run_cpu_time_us":562,"raft.run_wall_time_us":562,"replication_time_us":11995}]]}
I20260130 02:58:15.574693 20570 raft_consensus.cc:2243] T 38a7a10e36874183bf9fe55a38d829f6 P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260130 02:58:15.575253 20570 raft_consensus.cc:2272] T 38a7a10e36874183bf9fe55a38d829f6 P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260130 02:58:15.577440 20570 ts_tablet_manager.cc:1916] T 38a7a10e36874183bf9fe55a38d829f6 P 78c2c46c7520492db35d51c244fb4f74: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260130 02:58:15.583511 19807 tablet_bootstrap.cc:654] T a90712651a604b83ba187f893ab17711 P 371a36d219164974abdf848841528152: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:15.585589 19840 tablet_bootstrap.cc:654] T af0f90071a334016aa17fae2e3a97fc8 P 78c2c46c7520492db35d51c244fb4f74: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:15.587357 19197 tablet_service.cc:1552] Processing DeleteTablet for tablet 38a7a10e36874183bf9fe55a38d829f6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:58:14 UTC) from {username='slave'} at 127.0.0.1:42870
I20260130 02:58:15.588047 19462 leader_election.cc:304] T 03025d13f0e545cfa6288677a2715343 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78c2c46c7520492db35d51c244fb4f74, d870164263bb44b6b780a557daea3656; no voters:
I20260130 02:58:15.591300 20679 raft_consensus.cc:2804] T 03025d13f0e545cfa6288677a2715343 P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Leader election won for term 1
I20260130 02:58:15.592289 20629 raft_consensus.cc:493] T 25fd48e693424537b91ba7c3fcc96878 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:15.592736 20629 raft_consensus.cc:515] T 25fd48e693424537b91ba7c3fcc96878 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:15.604892 20616 raft_consensus.cc:493] T 5fbe92a2227d44d88b4531a7f8aa0e08 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:15.607952 20629 leader_election.cc:290] T 25fd48e693424537b91ba7c3fcc96878 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), d870164263bb44b6b780a557daea3656 (127.18.100.67:39571)
I20260130 02:58:15.612267 19525 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "25fd48e693424537b91ba7c3fcc96878" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
W20260130 02:58:15.613947 19313 leader_election.cc:343] T 25fd48e693424537b91ba7c3fcc96878 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:15.614941 20679 raft_consensus.cc:697] T 03025d13f0e545cfa6288677a2715343 P d870164263bb44b6b780a557daea3656 [term 1 LEADER]: Becoming Leader. State: Replica: d870164263bb44b6b780a557daea3656, State: Running, Role: LEADER
I20260130 02:58:15.607161 19225 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "25fd48e693424537b91ba7c3fcc96878" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152" is_pre_election: true
I20260130 02:58:15.616820 20616 raft_consensus.cc:515] T 5fbe92a2227d44d88b4531a7f8aa0e08 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
W20260130 02:58:15.617105 19314 leader_election.cc:343] T 25fd48e693424537b91ba7c3fcc96878 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:15.617462 19314 leader_election.cc:304] T 25fd48e693424537b91ba7c3fcc96878 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78c2c46c7520492db35d51c244fb4f74; no voters: 371a36d219164974abdf848841528152, d870164263bb44b6b780a557daea3656
I20260130 02:58:15.619879 20616 leader_election.cc:290] T 5fbe92a2227d44d88b4531a7f8aa0e08 P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d870164263bb44b6b780a557daea3656 (127.18.100.67:39571), 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767)
I20260130 02:58:15.620976 19525 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5fbe92a2227d44d88b4531a7f8aa0e08" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
I20260130 02:58:15.621618 19525 raft_consensus.cc:2468] T 5fbe92a2227d44d88b4531a7f8aa0e08 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 371a36d219164974abdf848841528152 in term 0.
I20260130 02:58:15.621539 19380 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5fbe92a2227d44d88b4531a7f8aa0e08" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74" is_pre_election: true
I20260130 02:58:15.622623 19159 leader_election.cc:304] T 5fbe92a2227d44d88b4531a7f8aa0e08 P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 371a36d219164974abdf848841528152, d870164263bb44b6b780a557daea3656; no voters:
I20260130 02:58:15.622897 19380 raft_consensus.cc:2468] T 5fbe92a2227d44d88b4531a7f8aa0e08 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 371a36d219164974abdf848841528152 in term 0.
I20260130 02:58:15.623840 20616 raft_consensus.cc:2804] T 5fbe92a2227d44d88b4531a7f8aa0e08 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260130 02:58:15.624194 20616 raft_consensus.cc:493] T 5fbe92a2227d44d88b4531a7f8aa0e08 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260130 02:58:15.624536 20616 raft_consensus.cc:3060] T 5fbe92a2227d44d88b4531a7f8aa0e08 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:15.626391 18870 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:50284 (request call id 231) took 3925 ms. Trace:
I20260130 02:58:15.626729 18870 rpcz_store.cc:276] 0130 02:58:11.700722 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:58:13.904866 (+2204144us) service_pool.cc:225] Handling call
0130 02:58:13.985834 (+ 80968us) spinlock_profiling.cc:243] Waited 79.3 ms on lock 0x7b5400030554. stack: 00007fb0bd3f7373 00007fb0bd3f7270 00007fb0bcf6c574 0000000000508744 00000000005086d9 00007fb0c44490c0 00007fb0c44e7bba 00007fb0c61e6eac 00007fb0c62e2cd6 00007fb0c3f0eb24 00007fb0c3f0eab2 00007fb0c3f0e9e1 00007fb0c3f0e95c 00007fb0c3f0dc12 00007fb0c21dd34c 00007fb0c21dc786
0130 02:58:13.998399 (+ 12565us) catalog_manager.cc:2393] Looking up, locking, and authorizing table
0130 02:58:13.999050 (+ 651us) catalog_manager.cc:3708] Apply alter schema
0130 02:58:13.999623 (+ 573us) catalog_manager.cc:3740] Acquired catalog manager lock
0130 02:58:13.999811 (+ 188us) catalog_manager.cc:3898] Updating metadata on disk
0130 02:58:14.167334 (+167523us) catalog_manager.cc:3929] Committing alterations to in-memory state
0130 02:58:15.626328 (+1458994us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0130 02:58:14.013084 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0130 02:58:14.014532 (+ 1448us) write_op.cc:432] Acquiring schema lock in shared mode
0130 02:58:14.014559 (+ 27us) write_op.cc:435] Acquired schema lock
0130 02:58:14.014602 (+ 43us) tablet.cc:660] Decoding operations
0130 02:58:14.015268 (+ 666us) write_op.cc:620] Acquiring the partition lock for write op
0130 02:58:14.015314 (+ 46us) write_op.cc:641] Partition lock acquired for write op
0130 02:58:14.015339 (+ 25us) tablet.cc:683] Acquiring locks for 1 operations
0130 02:58:14.015509 (+ 170us) tablet.cc:699] Row locks acquired
0130 02:58:14.015525 (+ 16us) write_op.cc:260] PREPARE: finished
0130 02:58:14.015648 (+ 123us) write_op.cc:270] Start()
0130 02:58:14.015735 (+ 87us) write_op.cc:276] Timestamp: P: 1769741894015567 usec, L: 0
0130 02:58:14.015751 (+ 16us) op_driver.cc:348] REPLICATION: starting
0130 02:58:14.016303 (+ 552us) log.cc:844] Serialized 405 byte log entry
0130 02:58:14.164014 (+147711us) op_driver.cc:464] REPLICATION: finished
0130 02:58:14.165770 (+ 1756us) write_op.cc:301] APPLY: starting
0130 02:58:14.165827 (+ 57us) tablet.cc:1365] starting BulkCheckPresence
0130 02:58:14.165915 (+ 88us) tablet.cc:1368] finished BulkCheckPresence
0130 02:58:14.166050 (+ 135us) tablet.cc:1370] starting ApplyRowOperation cycle
0130 02:58:14.166212 (+ 162us) tablet.cc:1381] finished ApplyRowOperation cycle
0130 02:58:14.166338 (+ 126us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0130 02:58:14.166379 (+ 41us) write_op.cc:312] APPLY: finished
0130 02:58:14.166560 (+ 181us) log.cc:844] Serialized 23 byte log entry
0130 02:58:14.166723 (+ 163us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:58:14.166778 (+ 55us) write_op.cc:454] Released schema lock
0130 02:58:14.166809 (+ 31us) write_op.cc:341] FINISH: Updating metrics
0130 02:58:14.167037 (+ 228us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:58:14.167064 (+ 27us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":182395264,"child_traces":[["op",{"apply.queue_time_us":1589,"apply.run_cpu_time_us":1255,"apply.run_wall_time_us":1255,"num_ops":1,"prepare.queue_time_us":12220,"prepare.run_cpu_time_us":2489,"prepare.run_wall_time_us":6545,"raft.queue_time_us":157,"raft.run_cpu_time_us":700,"raft.run_wall_time_us":702,"replication_time_us":148158}]]}
I20260130 02:58:15.633850 20690 tablet_replica.cc:333] T 38a7a10e36874183bf9fe55a38d829f6 P 371a36d219164974abdf848841528152: stopping tablet replica
I20260130 02:58:15.634930 20690 raft_consensus.cc:2243] T 38a7a10e36874183bf9fe55a38d829f6 P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260130 02:58:15.635792 20690 raft_consensus.cc:2272] T 38a7a10e36874183bf9fe55a38d829f6 P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260130 02:58:15.637878 20690 ts_tablet_manager.cc:1916] T 38a7a10e36874183bf9fe55a38d829f6 P 371a36d219164974abdf848841528152: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260130 02:58:15.640201 20562 raft_consensus.cc:493] T 6e12a8eeb3204325a9a8b7779f72ab29 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:15.640717 20562 raft_consensus.cc:515] T 6e12a8eeb3204325a9a8b7779f72ab29 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:15.661010 18869 catalog_manager.cc:6199] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4: default.Table_3a858a3cb886427cb8268f42c953ad4c [id=17150cb73e374ff9a1e92c43e0c3f2c1] alter complete (version 1)
I20260130 02:58:15.663604 19840 tablet_bootstrap.cc:492] T af0f90071a334016aa17fae2e3a97fc8 P 78c2c46c7520492db35d51c244fb4f74: No bootstrap required, opened a new log
I20260130 02:58:15.665710 19840 ts_tablet_manager.cc:1403] T af0f90071a334016aa17fae2e3a97fc8 P 78c2c46c7520492db35d51c244fb4f74: Time spent bootstrapping tablet: real 0.132s user 0.007s sys 0.006s
I20260130 02:58:15.638444 20497 consensus_queue.cc:1048] T 7783db3568f4434e91e9e456799cc312 P 371a36d219164974abdf848841528152 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260130 02:58:15.648521 19807 tablet_bootstrap.cc:492] T a90712651a604b83ba187f893ab17711 P 371a36d219164974abdf848841528152: No bootstrap required, opened a new log
I20260130 02:58:15.667248 19807 ts_tablet_manager.cc:1403] T a90712651a604b83ba187f893ab17711 P 371a36d219164974abdf848841528152: Time spent bootstrapping tablet: real 0.103s user 0.010s sys 0.000s
I20260130 02:58:15.658231 20571 raft_consensus.cc:2243] T d55291d7cea64a0d9a345ce712fbfa80 P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260130 02:58:15.642536 20562 leader_election.cc:290] T 6e12a8eeb3204325a9a8b7779f72ab29 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767)
I20260130 02:58:15.648926 20655 raft_consensus.cc:493] T 78e9bc3e55fe4e4a98e8eb01da59d365 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:15.668792 20571 raft_consensus.cc:2272] T d55291d7cea64a0d9a345ce712fbfa80 P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260130 02:58:15.669047 20655 raft_consensus.cc:515] T 78e9bc3e55fe4e4a98e8eb01da59d365 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:15.669687 19807 raft_consensus.cc:359] T a90712651a604b83ba187f893ab17711 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:15.670257 19807 raft_consensus.cc:385] T a90712651a604b83ba187f893ab17711 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:15.670497 19807 raft_consensus.cc:740] T a90712651a604b83ba187f893ab17711 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 371a36d219164974abdf848841528152, State: Initialized, Role: FOLLOWER
I20260130 02:58:15.670899 20655 leader_election.cc:290] T 78e9bc3e55fe4e4a98e8eb01da59d365 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767)
I20260130 02:58:15.671167 20571 ts_tablet_manager.cc:1916] T d55291d7cea64a0d9a345ce712fbfa80 P d870164263bb44b6b780a557daea3656: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260130 02:58:15.671178 20616 raft_consensus.cc:515] T 5fbe92a2227d44d88b4531a7f8aa0e08 P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:15.672987 20616 leader_election.cc:290] T 5fbe92a2227d44d88b4531a7f8aa0e08 P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 election: Requested vote from peers d870164263bb44b6b780a557daea3656 (127.18.100.67:39571), 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767)
I20260130 02:58:15.677102 19225 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6e12a8eeb3204325a9a8b7779f72ab29" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152" is_pre_election: true
I20260130 02:58:15.677654 19225 raft_consensus.cc:2468] T 6e12a8eeb3204325a9a8b7779f72ab29 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d870164263bb44b6b780a557daea3656 in term 0.
I20260130 02:58:15.678457 19223 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "78e9bc3e55fe4e4a98e8eb01da59d365" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152" is_pre_election: true
I20260130 02:58:15.678987 19223 raft_consensus.cc:2468] T 78e9bc3e55fe4e4a98e8eb01da59d365 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d870164263bb44b6b780a557daea3656 in term 0.
I20260130 02:58:15.679133 19525 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5fbe92a2227d44d88b4531a7f8aa0e08" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656"
I20260130 02:58:15.679641 19525 raft_consensus.cc:3060] T 5fbe92a2227d44d88b4531a7f8aa0e08 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:15.685082 19807 consensus_queue.cc:260] T a90712651a604b83ba187f893ab17711 P 371a36d219164974abdf848841528152 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:15.686535 19378 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "78e9bc3e55fe4e4a98e8eb01da59d365" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74" is_pre_election: true
I20260130 02:58:15.687139 19378 raft_consensus.cc:2468] T 78e9bc3e55fe4e4a98e8eb01da59d365 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d870164263bb44b6b780a557daea3656 in term 0.
I20260130 02:58:15.687911 19380 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6e12a8eeb3204325a9a8b7779f72ab29" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74" is_pre_election: true
I20260130 02:58:15.679133 19463 leader_election.cc:304] T 6e12a8eeb3204325a9a8b7779f72ab29 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 371a36d219164974abdf848841528152, d870164263bb44b6b780a557daea3656; no voters:
I20260130 02:58:15.688422 19380 raft_consensus.cc:2468] T 6e12a8eeb3204325a9a8b7779f72ab29 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d870164263bb44b6b780a557daea3656 in term 0.
I20260130 02:58:15.688900 20655 raft_consensus.cc:2804] T 6e12a8eeb3204325a9a8b7779f72ab29 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260130 02:58:15.689213 20655 raft_consensus.cc:493] T 6e12a8eeb3204325a9a8b7779f72ab29 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260130 02:58:15.689522 20655 raft_consensus.cc:3060] T 6e12a8eeb3204325a9a8b7779f72ab29 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:15.690975 19463 leader_election.cc:304] T 78e9bc3e55fe4e4a98e8eb01da59d365 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 371a36d219164974abdf848841528152, d870164263bb44b6b780a557daea3656; no voters:
I20260130 02:58:15.691674 20562 raft_consensus.cc:2804] T 78e9bc3e55fe4e4a98e8eb01da59d365 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260130 02:58:15.692018 20562 raft_consensus.cc:493] T 78e9bc3e55fe4e4a98e8eb01da59d365 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260130 02:58:15.692382 20562 raft_consensus.cc:3060] T 78e9bc3e55fe4e4a98e8eb01da59d365 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:15.692286 19840 raft_consensus.cc:359] T af0f90071a334016aa17fae2e3a97fc8 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:15.692821 19807 ts_tablet_manager.cc:1434] T a90712651a604b83ba187f893ab17711 P 371a36d219164974abdf848841528152: Time spent starting tablet: real 0.025s user 0.005s sys 0.000s
I20260130 02:58:15.692888 19840 raft_consensus.cc:385] T af0f90071a334016aa17fae2e3a97fc8 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:15.693133 19840 raft_consensus.cc:740] T af0f90071a334016aa17fae2e3a97fc8 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78c2c46c7520492db35d51c244fb4f74, State: Initialized, Role: FOLLOWER
I20260130 02:58:15.693666 19840 consensus_queue.cc:260] T af0f90071a334016aa17fae2e3a97fc8 P 78c2c46c7520492db35d51c244fb4f74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:15.699637 19807 tablet_bootstrap.cc:492] T e2aaaf00f53d418b87d5c7af286e4d84 P 371a36d219164974abdf848841528152: Bootstrap starting.
I20260130 02:58:15.705025 20497 raft_consensus.cc:493] T 274530359b174ff79f046c2d4a6afa76 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:15.705549 20497 raft_consensus.cc:515] T 274530359b174ff79f046c2d4a6afa76 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:15.686164 19525 raft_consensus.cc:2468] T 5fbe92a2227d44d88b4531a7f8aa0e08 P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 371a36d219164974abdf848841528152 in term 1.
I20260130 02:58:15.707433 20497 leader_election.cc:290] T 274530359b174ff79f046c2d4a6afa76 P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d870164263bb44b6b780a557daea3656 (127.18.100.67:39571), 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767)
I20260130 02:58:15.709026 19159 leader_election.cc:304] T 5fbe92a2227d44d88b4531a7f8aa0e08 P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 371a36d219164974abdf848841528152, d870164263bb44b6b780a557daea3656; no voters:
I20260130 02:58:15.709707 20497 raft_consensus.cc:2804] T 5fbe92a2227d44d88b4531a7f8aa0e08 P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Leader election won for term 1
I20260130 02:58:15.710140 20497 raft_consensus.cc:697] T 5fbe92a2227d44d88b4531a7f8aa0e08 P 371a36d219164974abdf848841528152 [term 1 LEADER]: Becoming Leader. State: Replica: 371a36d219164974abdf848841528152, State: Running, Role: LEADER
I20260130 02:58:15.711004 19526 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "274530359b174ff79f046c2d4a6afa76" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
I20260130 02:58:15.710898 20497 consensus_queue.cc:237] T 5fbe92a2227d44d88b4531a7f8aa0e08 P 371a36d219164974abdf848841528152 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:15.711512 19526 raft_consensus.cc:2468] T 274530359b174ff79f046c2d4a6afa76 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 371a36d219164974abdf848841528152 in term 0.
I20260130 02:58:15.714386 19159 leader_election.cc:304] T 274530359b174ff79f046c2d4a6afa76 P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 371a36d219164974abdf848841528152, d870164263bb44b6b780a557daea3656; no voters:
I20260130 02:58:15.665019 20693 raft_consensus.cc:2749] T 25fd48e693424537b91ba7c3fcc96878 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260130 02:58:15.715126 20608 raft_consensus.cc:2804] T 274530359b174ff79f046c2d4a6afa76 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260130 02:58:15.715447 20608 raft_consensus.cc:493] T 274530359b174ff79f046c2d4a6afa76 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260130 02:58:15.715737 20608 raft_consensus.cc:3060] T 274530359b174ff79f046c2d4a6afa76 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:15.717895 18868 catalog_manager.cc:6199] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4: default.Table_1703c69112744c0e951c1ac9667bbac6 [id=8d76f4328edc491291491c78ba205d30] alter complete (version 1)
I20260130 02:58:15.722476 19378 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5fbe92a2227d44d88b4531a7f8aa0e08" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74"
I20260130 02:58:15.723016 19378 raft_consensus.cc:3060] T 5fbe92a2227d44d88b4531a7f8aa0e08 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:15.723284 20608 raft_consensus.cc:515] T 274530359b174ff79f046c2d4a6afa76 P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:15.726243 20608 leader_election.cc:290] T 274530359b174ff79f046c2d4a6afa76 P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 election: Requested vote from peers d870164263bb44b6b780a557daea3656 (127.18.100.67:39571), 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767)
I20260130 02:58:15.728009 19526 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "274530359b174ff79f046c2d4a6afa76" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656"
I20260130 02:58:15.728531 19526 raft_consensus.cc:3060] T 274530359b174ff79f046c2d4a6afa76 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:15.728914 19378 raft_consensus.cc:2468] T 5fbe92a2227d44d88b4531a7f8aa0e08 P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 371a36d219164974abdf848841528152 in term 1.
I20260130 02:58:15.734989 19526 raft_consensus.cc:2468] T 274530359b174ff79f046c2d4a6afa76 P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 371a36d219164974abdf848841528152 in term 1.
I20260130 02:58:15.739214 19159 leader_election.cc:304] T 274530359b174ff79f046c2d4a6afa76 P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 371a36d219164974abdf848841528152, d870164263bb44b6b780a557daea3656; no voters:
I20260130 02:58:15.740137 20497 raft_consensus.cc:2804] T 274530359b174ff79f046c2d4a6afa76 P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Leader election won for term 1
I20260130 02:58:15.740724 20497 raft_consensus.cc:697] T 274530359b174ff79f046c2d4a6afa76 P 371a36d219164974abdf848841528152 [term 1 LEADER]: Becoming Leader. State: Replica: 371a36d219164974abdf848841528152, State: Running, Role: LEADER
I20260130 02:58:15.741533 20497 consensus_queue.cc:237] T 274530359b174ff79f046c2d4a6afa76 P 371a36d219164974abdf848841528152 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:15.747627 20655 raft_consensus.cc:515] T 6e12a8eeb3204325a9a8b7779f72ab29 P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:15.748247 19197 tablet_service.cc:1552] Processing DeleteTablet for tablet e364f4e70e5749e083e14e7fa7416ff0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:58:10 UTC) from {username='slave'} at 127.0.0.1:42870
I20260130 02:58:15.749610 20655 leader_election.cc:290] T 6e12a8eeb3204325a9a8b7779f72ab29 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 election: Requested vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767)
I20260130 02:58:15.751888 19197 tablet_service.cc:1552] Processing DeleteTablet for tablet 3b148691090e4ee581d46f9845a6e49a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:58:10 UTC) from {username='slave'} at 127.0.0.1:42870
I20260130 02:58:15.752689 19197 tablet_service.cc:1552] Processing DeleteTablet for tablet d27ca7dd87f44102b7680e1f4a88954a with delete_type TABLET_DATA_DELETED (replaced by tablet d480bf18140245e3870a62582d5a8318) from {username='slave'} at 127.0.0.1:42870
W20260130 02:58:15.753978 18871 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.18.100.65:51021 (request call id 34) took 3882 ms (client timeout 1000 ms). Trace:
I20260130 02:58:15.754089 19840 ts_tablet_manager.cc:1434] T af0f90071a334016aa17fae2e3a97fc8 P 78c2c46c7520492db35d51c244fb4f74: Time spent starting tablet: real 0.079s user 0.004s sys 0.001s
I20260130 02:58:15.754158 20562 raft_consensus.cc:515] T 78e9bc3e55fe4e4a98e8eb01da59d365 P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:15.754968 19840 tablet_bootstrap.cc:492] T 3aa76993beb94b12bc706a76db26a03b P 78c2c46c7520492db35d51c244fb4f74: Bootstrap starting.
I20260130 02:58:15.755959 20562 leader_election.cc:290] T 78e9bc3e55fe4e4a98e8eb01da59d365 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 election: Requested vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767)
I20260130 02:58:15.756400 19380 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "274530359b174ff79f046c2d4a6afa76" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74" is_pre_election: true
I20260130 02:58:15.756915 19380 raft_consensus.cc:2468] T 274530359b174ff79f046c2d4a6afa76 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 371a36d219164974abdf848841528152 in term 0.
I20260130 02:58:15.758277 19625 tablet_bootstrap.cc:492] T a90712651a604b83ba187f893ab17711 P d870164263bb44b6b780a557daea3656: No bootstrap required, opened a new log
I20260130 02:58:15.758682 19625 ts_tablet_manager.cc:1403] T a90712651a604b83ba187f893ab17711 P d870164263bb44b6b780a557daea3656: Time spent bootstrapping tablet: real 0.376s user 0.016s sys 0.019s
I20260130 02:58:15.761798 19378 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "274530359b174ff79f046c2d4a6afa76" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74"
I20260130 02:58:15.762303 19378 raft_consensus.cc:3060] T 274530359b174ff79f046c2d4a6afa76 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:15.769680 20570 ts_tablet_manager.cc:1929] T 38a7a10e36874183bf9fe55a38d829f6 P 78c2c46c7520492db35d51c244fb4f74: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260130 02:58:15.770109 20570 log.cc:1199] T 38a7a10e36874183bf9fe55a38d829f6 P 78c2c46c7520492db35d51c244fb4f74: Deleting WAL directory at /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-1/wal/wals/38a7a10e36874183bf9fe55a38d829f6
I20260130 02:58:15.772240 20570 ts_tablet_manager.cc:1950] T 38a7a10e36874183bf9fe55a38d829f6 P 78c2c46c7520492db35d51c244fb4f74: Deleting consensus metadata
I20260130 02:58:15.757874 19225 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6e12a8eeb3204325a9a8b7779f72ab29" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152"
I20260130 02:58:15.773123 19225 raft_consensus.cc:3060] T 6e12a8eeb3204325a9a8b7779f72ab29 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:15.763183 19807 tablet_bootstrap.cc:654] T e2aaaf00f53d418b87d5c7af286e4d84 P 371a36d219164974abdf848841528152: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:15.758188 19222 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "78e9bc3e55fe4e4a98e8eb01da59d365" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152"
I20260130 02:58:15.779629 19222 raft_consensus.cc:3060] T 78e9bc3e55fe4e4a98e8eb01da59d365 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:15.779793 19225 raft_consensus.cc:2468] T 6e12a8eeb3204325a9a8b7779f72ab29 P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d870164263bb44b6b780a557daea3656 in term 1.
I20260130 02:58:15.754537 19204 tablet_service.cc:1552] Processing DeleteTablet for tablet 7d96e93d95f14af79cfb268626559b1e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:58:04 UTC) from {username='slave'} at 127.0.0.1:42870
I20260130 02:58:15.781925 19463 leader_election.cc:304] T 6e12a8eeb3204325a9a8b7779f72ab29 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 371a36d219164974abdf848841528152, d870164263bb44b6b780a557daea3656; no voters:
I20260130 02:58:15.783706 20571 ts_tablet_manager.cc:1929] T d55291d7cea64a0d9a345ce712fbfa80 P d870164263bb44b6b780a557daea3656: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260130 02:58:15.784066 20571 log.cc:1199] T d55291d7cea64a0d9a345ce712fbfa80 P d870164263bb44b6b780a557daea3656: Deleting WAL directory at /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-2/wal/wals/d55291d7cea64a0d9a345ce712fbfa80
I20260130 02:58:15.784610 20562 raft_consensus.cc:2804] T 6e12a8eeb3204325a9a8b7779f72ab29 P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Leader election won for term 1
I20260130 02:58:15.785123 20562 raft_consensus.cc:697] T 6e12a8eeb3204325a9a8b7779f72ab29 P d870164263bb44b6b780a557daea3656 [term 1 LEADER]: Becoming Leader. State: Replica: d870164263bb44b6b780a557daea3656, State: Running, Role: LEADER
I20260130 02:58:15.786437 19222 raft_consensus.cc:2468] T 78e9bc3e55fe4e4a98e8eb01da59d365 P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d870164263bb44b6b780a557daea3656 in term 1.
I20260130 02:58:15.789175 19625 raft_consensus.cc:359] T a90712651a604b83ba187f893ab17711 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:15.789784 19625 raft_consensus.cc:385] T a90712651a604b83ba187f893ab17711 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:15.789556 20690 ts_tablet_manager.cc:1929] T 38a7a10e36874183bf9fe55a38d829f6 P 371a36d219164974abdf848841528152: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260130 02:58:15.790027 19625 raft_consensus.cc:740] T a90712651a604b83ba187f893ab17711 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d870164263bb44b6b780a557daea3656, State: Initialized, Role: FOLLOWER
I20260130 02:58:15.790256 20690 log.cc:1199] T 38a7a10e36874183bf9fe55a38d829f6 P 371a36d219164974abdf848841528152: Deleting WAL directory at /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-0/wal/wals/38a7a10e36874183bf9fe55a38d829f6
I20260130 02:58:15.794097 19463 leader_election.cc:304] T 78e9bc3e55fe4e4a98e8eb01da59d365 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 371a36d219164974abdf848841528152, d870164263bb44b6b780a557daea3656; no voters:
I20260130 02:58:15.794811 20655 raft_consensus.cc:2804] T 78e9bc3e55fe4e4a98e8eb01da59d365 P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Leader election won for term 1
W20260130 02:58:15.794850 18871 rpcz_store.cc:269] 0130 02:58:11.871328 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:58:12.043184 (+171856us) service_pool.cc:225] Handling call
0130 02:58:15.753908 (+3710724us) inbound_call.cc:173] Queueing success response
Metrics: {"reported_tablets":39}
I20260130 02:58:15.795245 20655 raft_consensus.cc:697] T 78e9bc3e55fe4e4a98e8eb01da59d365 P d870164263bb44b6b780a557daea3656 [term 1 LEADER]: Becoming Leader. State: Replica: d870164263bb44b6b780a557daea3656, State: Running, Role: LEADER
I20260130 02:58:15.796278 20655 consensus_queue.cc:237] T 78e9bc3e55fe4e4a98e8eb01da59d365 P d870164263bb44b6b780a557daea3656 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:15.793524 19204 tablet_service.cc:1552] Processing DeleteTablet for tablet d55291d7cea64a0d9a345ce712fbfa80 with delete_type TABLET_DATA_DELETED (replaced by tablet cd30d7adcc4545848e3501a205a10293) from {username='slave'} at 127.0.0.1:42870
W20260130 02:58:15.781055 18868 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.18.100.66:59691 (request call id 30) took 4508 ms (client timeout 1000 ms). Trace:
I20260130 02:58:15.798307 20690 ts_tablet_manager.cc:1950] T 38a7a10e36874183bf9fe55a38d829f6 P 371a36d219164974abdf848841528152: Deleting consensus metadata
W20260130 02:58:15.797245 18868 rpcz_store.cc:269] 0130 02:58:11.273003 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:58:11.432133 (+159130us) service_pool.cc:225] Handling call
0130 02:58:12.418735 (+986602us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:58:12.418761 (+ 26us) write_op.cc:454] Released schema lock
0130 02:58:15.780972 (+3362211us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0130 02:58:12.348568 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0130 02:58:12.348947 (+ 379us) write_op.cc:432] Acquiring schema lock in shared mode
0130 02:58:12.348973 (+ 26us) write_op.cc:435] Acquired schema lock
0130 02:58:12.348985 (+ 12us) tablet.cc:660] Decoding operations
0130 02:58:12.349404 (+ 419us) write_op.cc:620] Acquiring the partition lock for write op
0130 02:58:12.349443 (+ 39us) write_op.cc:641] Partition lock acquired for write op
0130 02:58:12.349463 (+ 20us) tablet.cc:683] Acquiring locks for 3 operations
0130 02:58:12.349709 (+ 246us) tablet.cc:699] Row locks acquired
0130 02:58:12.349723 (+ 14us) write_op.cc:260] PREPARE: finished
0130 02:58:12.349827 (+ 104us) write_op.cc:270] Start()
0130 02:58:12.349903 (+ 76us) write_op.cc:276] Timestamp: P: 1769741892349762 usec, L: 0
0130 02:58:12.349918 (+ 15us) op_driver.cc:348] REPLICATION: starting
0130 02:58:12.350375 (+ 457us) log.cc:844] Serialized 1293 byte log entry
0130 02:58:12.369978 (+ 19603us) op_driver.cc:464] REPLICATION: finished
0130 02:58:12.370365 (+ 387us) write_op.cc:301] APPLY: starting
0130 02:58:12.370417 (+ 52us) tablet.cc:1365] starting BulkCheckPresence
0130 02:58:12.370534 (+ 117us) tablet.cc:1368] finished BulkCheckPresence
0130 02:58:12.370800 (+ 266us) tablet.cc:1370] starting ApplyRowOperation cycle
0130 02:58:12.371188 (+ 388us) tablet.cc:1381] finished ApplyRowOperation cycle
0130 02:58:12.371330 (+ 142us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=3
0130 02:58:12.371371 (+ 41us) write_op.cc:312] APPLY: finished
0130 02:58:12.371552 (+ 181us) log.cc:844] Serialized 35 byte log entry
0130 02:58:12.371698 (+ 146us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:58:12.371768 (+ 70us) write_op.cc:454] Released schema lock
0130 02:58:12.371797 (+ 29us) write_op.cc:341] FINISH: Updating metrics
Related trace 'op':
0130 02:58:13.447106 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0130 02:58:13.447799 (+ 693us) write_op.cc:432] Acquiring schema lock in shared mode
0130 02:58:13.447828 (+ 29us) write_op.cc:435] Acquired schema lock
0130 02:58:13.447844 (+ 16us) tablet.cc:660] Decoding operations
0130 02:58:13.448136 (+ 292us) write_op.cc:620] Acquiring the partition lock for write op
0130 02:58:13.448193 (+ 57us) write_op.cc:641] Partition lock acquired for write op
0130 02:58:13.448220 (+ 27us) tablet.cc:683] Acquiring locks for 1 operations
0130 02:58:13.448377 (+ 157us) tablet.cc:699] Row locks acquired
0130 02:58:13.448392 (+ 15us) write_op.cc:260] PREPARE: finished
0130 02:58:13.449781 (+ 1389us) write_op.cc:270] Start()
0130 02:58:13.449886 (+ 105us) write_op.cc:276] Timestamp: P: 1769741893449685 usec, L: 0
0130 02:58:13.449905 (+ 19us) op_driver.cc:348] REPLICATION: starting
0130 02:58:13.450598 (+ 693us) log.cc:844] Serialized 356 byte log entry
0130 02:58:13.718390 (+267792us) op_driver.cc:464] REPLICATION: finished
0130 02:58:13.745676 (+ 27286us) write_op.cc:301] APPLY: starting
0130 02:58:13.745735 (+ 59us) tablet.cc:1365] starting BulkCheckPresence
0130 02:58:13.745821 (+ 86us) tablet.cc:1368] finished BulkCheckPresence
0130 02:58:13.745924 (+ 103us) tablet.cc:1370] starting ApplyRowOperation cycle
0130 02:58:13.746121 (+ 197us) tablet.cc:1381] finished ApplyRowOperation cycle
0130 02:58:13.746267 (+ 146us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0130 02:58:13.746308 (+ 41us) write_op.cc:312] APPLY: finished
0130 02:58:13.746488 (+ 180us) log.cc:844] Serialized 23 byte log entry
0130 02:58:13.792303 (+ 45815us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:58:13.792366 (+ 63us) write_op.cc:454] Released schema lock
0130 02:58:13.792398 (+ 32us) write_op.cc:341] FINISH: Updating metrics
0130 02:58:13.792762 (+ 364us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:58:13.792788 (+ 26us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":2389,"reported_tablets":40,"child_traces":[["op",{"apply.queue_time_us":248,"apply.run_cpu_time_us":1451,"apply.run_wall_time_us":1593,"num_ops":3,"prepare.queue_time_us":93,"prepare.run_cpu_time_us":2138,"prepare.run_wall_time_us":2138,"raft.queue_time_us":3247,"raft.run_cpu_time_us":565,"raft.run_wall_time_us":566,"replication_time_us":19961}],["op",{"apply.queue_time_us":27107,"apply.run_cpu_time_us":1320,"apply.run_wall_time_us":47059,"num_ops":1,"prepare.queue_time_us":165243,"prepare.run_cpu_time_us":2461,"prepare.run_wall_time_us":83255,"raft.queue_time_us":78740,"raft.run_cpu_time_us":629,"raft.run_wall_time_us":628,"replication_time_us":268371,"thread_start_us":229091,"threads_started":2}]]}
I20260130 02:58:15.770171 19378 raft_consensus.cc:2468] T 274530359b174ff79f046c2d4a6afa76 P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 371a36d219164974abdf848841528152 in term 1.
I20260130 02:58:15.802588 20562 consensus_queue.cc:237] T 6e12a8eeb3204325a9a8b7779f72ab29 P d870164263bb44b6b780a557daea3656 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:15.805498 19625 consensus_queue.cc:260] T a90712651a604b83ba187f893ab17711 P d870164263bb44b6b780a557daea3656 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:15.809211 19840 tablet_bootstrap.cc:654] T 3aa76993beb94b12bc706a76db26a03b P 78c2c46c7520492db35d51c244fb4f74: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:15.784720 20571 ts_tablet_manager.cc:1950] T d55291d7cea64a0d9a345ce712fbfa80 P d870164263bb44b6b780a557daea3656: Deleting consensus metadata
I20260130 02:58:15.811820 20571 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59068 (request call id 269) took 1859 ms. Trace:
I20260130 02:58:15.812116 20571 rpcz_store.cc:276] 0130 02:58:13.951941 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:58:13.952044 (+ 103us) service_pool.cc:225] Handling call
0130 02:58:15.521988 (+1569944us) ts_tablet_manager.cc:1151] Deleting tablet d55291d7cea64a0d9a345ce712fbfa80
0130 02:58:15.522025 (+ 37us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0130 02:58:15.682085 (+160060us) tablet_metadata.cc:641] Metadata flushed
0130 02:58:15.783482 (+101397us) tablet_metadata.cc:641] Metadata flushed
0130 02:58:15.811769 (+ 28287us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":135309,"tablet-delete.queue_time_us":1569353}
I20260130 02:58:15.615808 20679 consensus_queue.cc:237] T 03025d13f0e545cfa6288677a2715343 P d870164263bb44b6b780a557daea3656 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:15.815965 19807 tablet_bootstrap.cc:492] T e2aaaf00f53d418b87d5c7af286e4d84 P 371a36d219164974abdf848841528152: No bootstrap required, opened a new log
I20260130 02:58:15.816350 19807 ts_tablet_manager.cc:1403] T e2aaaf00f53d418b87d5c7af286e4d84 P 371a36d219164974abdf848841528152: Time spent bootstrapping tablet: real 0.117s user 0.009s sys 0.001s
I20260130 02:58:15.816349 19625 ts_tablet_manager.cc:1434] T a90712651a604b83ba187f893ab17711 P d870164263bb44b6b780a557daea3656: Time spent starting tablet: real 0.057s user 0.005s sys 0.000s
I20260130 02:58:15.814363 20591 leader_election.cc:290] T 03025d13f0e545cfa6288677a2715343 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 election: Requested vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767)
I20260130 02:58:15.818640 20703 raft_consensus.cc:493] T 6218f2f7d73a42428b57a45858f53afd P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:15.818423 19807 raft_consensus.cc:359] T e2aaaf00f53d418b87d5c7af286e4d84 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:15.819028 19807 raft_consensus.cc:385] T e2aaaf00f53d418b87d5c7af286e4d84 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:15.819275 19807 raft_consensus.cc:740] T e2aaaf00f53d418b87d5c7af286e4d84 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 371a36d219164974abdf848841528152, State: Initialized, Role: FOLLOWER
I20260130 02:58:15.819079 20703 raft_consensus.cc:515] T 6218f2f7d73a42428b57a45858f53afd P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:15.819824 19807 consensus_queue.cc:260] T e2aaaf00f53d418b87d5c7af286e4d84 P 371a36d219164974abdf848841528152 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:15.821202 20703 leader_election.cc:290] T 6218f2f7d73a42428b57a45858f53afd P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), d870164263bb44b6b780a557daea3656 (127.18.100.67:39571)
I20260130 02:58:15.822924 19807 ts_tablet_manager.cc:1434] T e2aaaf00f53d418b87d5c7af286e4d84 P 371a36d219164974abdf848841528152: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260130 02:58:15.823717 19807 tablet_bootstrap.cc:492] T dcba6b534f9d406dad4fb502be40921c P 371a36d219164974abdf848841528152: Bootstrap starting.
I20260130 02:58:15.825862 19222 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6218f2f7d73a42428b57a45858f53afd" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152" is_pre_election: true
I20260130 02:58:15.828052 20616 consensus_queue.cc:1048] T 7783db3568f4434e91e9e456799cc312 P 371a36d219164974abdf848841528152 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260130 02:58:15.829741 19314 leader_election.cc:343] T 6218f2f7d73a42428b57a45858f53afd P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:15.835033 20690 tablet_replica.cc:333] T d27ca7dd87f44102b7680e1f4a88954a P 371a36d219164974abdf848841528152: stopping tablet replica
I20260130 02:58:15.837891 20690 raft_consensus.cc:2243] T d27ca7dd87f44102b7680e1f4a88954a P 371a36d219164974abdf848841528152 [term 1 LEADER]: Raft consensus shutting down.
I20260130 02:58:15.843402 20690 raft_consensus.cc:2272] T d27ca7dd87f44102b7680e1f4a88954a P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260130 02:58:15.847131 20571 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59068 (request call id 271) took 1395 ms. Trace:
I20260130 02:58:15.860666 20690 ts_tablet_manager.cc:1916] T d27ca7dd87f44102b7680e1f4a88954a P 371a36d219164974abdf848841528152: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260130 02:58:15.871538 20717 tablet.cc:1722] T 0e8018c094024c4788c484e8f5466e00 P 78c2c46c7520492db35d51c244fb4f74: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260130 02:58:15.879226 19355 tablet_service.cc:1552] Processing DeleteTablet for tablet f54addb1a4304ce69c711ae0cc3b452c with delete_type TABLET_DATA_DELETED (replaced by tablet 7783db3568f4434e91e9e456799cc312) from {username='slave'} at 127.0.0.1:60110
I20260130 02:58:15.880275 20716 tablet.cc:1722] T 16691c9315e94253bb9b3fb5058e5636 P 78c2c46c7520492db35d51c244fb4f74: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260130 02:58:15.883059 19378 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6e12a8eeb3204325a9a8b7779f72ab29" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74"
I20260130 02:58:15.884639 19378 raft_consensus.cc:3060] T 6e12a8eeb3204325a9a8b7779f72ab29 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:15.847445 20571 rpcz_store.cc:276] 0130 02:58:14.451534 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:58:14.451661 (+ 127us) service_pool.cc:225] Handling call
0130 02:58:15.846780 (+1395119us) ts_tablet_manager.cc:1151] Deleting tablet 2c79a3aecab844698e70834999e1caac
0130 02:58:15.846825 (+ 45us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0130 02:58:15.847077 (+ 252us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1350117}
I20260130 02:58:15.886013 20571 tablet_replica.cc:333] T 38a7a10e36874183bf9fe55a38d829f6 P d870164263bb44b6b780a557daea3656: stopping tablet replica
I20260130 02:58:15.886835 20571 raft_consensus.cc:2243] T 38a7a10e36874183bf9fe55a38d829f6 P d870164263bb44b6b780a557daea3656 [term 1 LEADER]: Raft consensus shutting down.
I20260130 02:58:15.889429 19625 tablet_bootstrap.cc:492] T e2aaaf00f53d418b87d5c7af286e4d84 P d870164263bb44b6b780a557daea3656: Bootstrap starting.
I20260130 02:58:15.890268 19378 raft_consensus.cc:2468] T 6e12a8eeb3204325a9a8b7779f72ab29 P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d870164263bb44b6b780a557daea3656 in term 1.
I20260130 02:58:15.893704 20719 tablet.cc:1722] T 0e8018c094024c4788c484e8f5466e00 P 371a36d219164974abdf848841528152: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260130 02:58:15.883841 19807 tablet_bootstrap.cc:654] T dcba6b534f9d406dad4fb502be40921c P 371a36d219164974abdf848841528152: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:15.887822 20571 raft_consensus.cc:2272] T 38a7a10e36874183bf9fe55a38d829f6 P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260130 02:58:15.911484 20616 consensus_queue.cc:1048] T ba91a508b1cb4430a4d8ca26946aecc8 P 371a36d219164974abdf848841528152 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260130 02:58:15.916215 19807 tablet_bootstrap.cc:492] T dcba6b534f9d406dad4fb502be40921c P 371a36d219164974abdf848841528152: No bootstrap required, opened a new log
W20260130 02:58:15.785804 18860 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet 2b0edaad6877460cb7982efc62b00b66 (table default.Table_d95381c590c94802bdc83acef0f0df96 [id=64a8e0e079f34806aa68795f3d548c5e], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 2b0edaad6877460cb7982efc62b00b66 consensus state has no leader
I20260130 02:58:15.924921 18860 catalog_manager.cc:4985] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767): tablet 38a7a10e36874183bf9fe55a38d829f6 (table default.Table_f359d249f8274f319f5ff4e4ea89dd22 [id=27adf356fd004597a54c7a9887dd3ca8]) successfully deleted
W20260130 02:58:15.925577 18860 catalog_manager.cc:4952] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767): delete failed for tablet f54addb1a4304ce69c711ae0cc3b452c because the tablet was not found. No further retry: Not found: Tablet not found: f54addb1a4304ce69c711ae0cc3b452c
I20260130 02:58:15.804467 18861 catalog_manager.cc:4985] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021): tablet 38a7a10e36874183bf9fe55a38d829f6 (table default.Table_f359d249f8274f319f5ff4e4ea89dd22 [id=27adf356fd004597a54c7a9887dd3ca8]) successfully deleted
W20260130 02:58:15.926884 18861 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.18.100.65:42021, user_credentials={real_user=slave}} blocked reactor thread for 122433us
I20260130 02:58:15.927333 19807 ts_tablet_manager.cc:1403] T dcba6b534f9d406dad4fb502be40921c P 371a36d219164974abdf848841528152: Time spent bootstrapping tablet: real 0.093s user 0.009s sys 0.000s
I20260130 02:58:15.929841 19496 tablet_service.cc:1552] Processing DeleteTablet for tablet e364f4e70e5749e083e14e7fa7416ff0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:58:10 UTC) from {username='slave'} at 127.0.0.1:59068
I20260130 02:58:15.930397 19807 raft_consensus.cc:359] T dcba6b534f9d406dad4fb502be40921c P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:15.931016 19807 raft_consensus.cc:385] T dcba6b534f9d406dad4fb502be40921c P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:15.931253 19807 raft_consensus.cc:740] T dcba6b534f9d406dad4fb502be40921c P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 371a36d219164974abdf848841528152, State: Initialized, Role: FOLLOWER
I20260130 02:58:15.934841 18860 catalog_manager.cc:4985] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): tablet d55291d7cea64a0d9a345ce712fbfa80 (table default.Table_b85cb876f25740bfaaab01949ac26b38 [id=839e292e040c4aeb832f3fc0258c32ec]) successfully deleted
I20260130 02:58:15.935021 20571 ts_tablet_manager.cc:1916] T 38a7a10e36874183bf9fe55a38d829f6 P d870164263bb44b6b780a557daea3656: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260130 02:58:15.935619 18860 catalog_manager.cc:4965] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): delete failed for tablet 2c79a3aecab844698e70834999e1caac because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2c79a3aecab844698e70834999e1caac already in progress: creating tablet
I20260130 02:58:15.937144 19501 tablet_service.cc:1552] Processing DeleteTablet for tablet b2ac91da01954c2390b6b7dcc06c394f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:58:04 UTC) from {username='slave'} at 127.0.0.1:59068
I20260130 02:58:15.937443 19503 tablet_service.cc:1552] Processing DeleteTablet for tablet 8bd3e1c06b89426da57e2b259e7983af with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:58:08 UTC) from {username='slave'} at 127.0.0.1:59068
I20260130 02:58:15.937940 19502 tablet_service.cc:1552] Processing DeleteTablet for tablet d27ca7dd87f44102b7680e1f4a88954a with delete_type TABLET_DATA_DELETED (replaced by tablet d480bf18140245e3870a62582d5a8318) from {username='slave'} at 127.0.0.1:59068
I20260130 02:58:15.938498 19508 tablet_service.cc:1552] Processing DeleteTablet for tablet 3389db58493448aea3cf00aca411047c with delete_type TABLET_DATA_DELETED (replaced by tablet af0f90071a334016aa17fae2e3a97fc8) from {username='slave'} at 127.0.0.1:59068
I20260130 02:58:15.939155 19505 tablet_service.cc:1552] Processing DeleteTablet for tablet 7d96e93d95f14af79cfb268626559b1e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:58:04 UTC) from {username='slave'} at 127.0.0.1:59068
I20260130 02:58:15.939723 19501 tablet_service.cc:1552] Processing DeleteTablet for tablet 3b148691090e4ee581d46f9845a6e49a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:58:10 UTC) from {username='slave'} at 127.0.0.1:59068
I20260130 02:58:15.940671 19496 tablet_service.cc:1552] Processing DeleteTablet for tablet f54addb1a4304ce69c711ae0cc3b452c with delete_type TABLET_DATA_DELETED (replaced by tablet 7783db3568f4434e91e9e456799cc312) from {username='slave'} at 127.0.0.1:59068
W20260130 02:58:15.868777 18866 catalog_manager.cc:5846] Failed to send AlterTable RPC for tablet 7caf7cbad5054396b5617094e4d536a0 (table default.Table_007e969f70d345a7bcb3dfbaabbc025d [id=77d4fa8705a3419cb0a84d3ca1bb4ce3], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet 7caf7cbad5054396b5617094e4d536a0 consensus state has no leader
I20260130 02:58:15.945348 20616 consensus_queue.cc:1048] T ba91a508b1cb4430a4d8ca26946aecc8 P 371a36d219164974abdf848841528152 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260130 02:58:15.954813 19807 consensus_queue.cc:260] T dcba6b534f9d406dad4fb502be40921c P 371a36d219164974abdf848841528152 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:15.956697 19807 ts_tablet_manager.cc:1434] T dcba6b534f9d406dad4fb502be40921c P 371a36d219164974abdf848841528152: Time spent starting tablet: real 0.029s user 0.004s sys 0.000s
W20260130 02:58:15.960186 18874 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.18.100.67:47213 (request call id 28) took 3879 ms (client timeout 1000 ms). Trace:
I20260130 02:58:15.961118 20497 raft_consensus.cc:493] T a90712651a604b83ba187f893ab17711 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260130 02:58:15.960629 18874 rpcz_store.cc:269] 0130 02:58:12.080533 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:58:13.043444 (+962911us) service_pool.cc:225] Handling call
0130 02:58:15.960115 (+2916671us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":270346,"reported_tablets":36}
I20260130 02:58:15.961535 20497 raft_consensus.cc:515] T a90712651a604b83ba187f893ab17711 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:15.962750 18874 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50284:
name: "default.Table_5e70ab0e32bc4048bd19388f9d7cd302"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260130 02:58:15.964540 20690 ts_tablet_manager.cc:1929] T d27ca7dd87f44102b7680e1f4a88954a P 371a36d219164974abdf848841528152: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260130 02:58:15.967171 19525 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a90712651a604b83ba187f893ab17711" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
I20260130 02:58:15.967784 19525 raft_consensus.cc:2468] T a90712651a604b83ba187f893ab17711 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 371a36d219164974abdf848841528152 in term 0.
I20260130 02:58:15.968643 19159 leader_election.cc:304] T a90712651a604b83ba187f893ab17711 P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 371a36d219164974abdf848841528152, d870164263bb44b6b780a557daea3656; no voters:
I20260130 02:58:15.970752 20690 log.cc:1199] T d27ca7dd87f44102b7680e1f4a88954a P 371a36d219164974abdf848841528152: Deleting WAL directory at /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-0/wal/wals/d27ca7dd87f44102b7680e1f4a88954a
I20260130 02:58:15.971091 20616 raft_consensus.cc:2804] T a90712651a604b83ba187f893ab17711 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260130 02:58:15.978746 20616 raft_consensus.cc:493] T a90712651a604b83ba187f893ab17711 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260130 02:58:15.979071 20616 raft_consensus.cc:3060] T a90712651a604b83ba187f893ab17711 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:15.980850 20719 tablet.cc:1722] T 16691c9315e94253bb9b3fb5058e5636 P 371a36d219164974abdf848841528152: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260130 02:58:15.985472 20616 raft_consensus.cc:515] T a90712651a604b83ba187f893ab17711 P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:15.987581 20616 leader_election.cc:290] T a90712651a604b83ba187f893ab17711 P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 election: Requested vote from peers 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767), d870164263bb44b6b780a557daea3656 (127.18.100.67:39571)
I20260130 02:58:15.989851 19525 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a90712651a604b83ba187f893ab17711" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656"
I20260130 02:58:15.990320 19525 raft_consensus.cc:3060] T a90712651a604b83ba187f893ab17711 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:15.944559 18868 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50284:
table { table_name: "default.Table_748cb9c252084fb48ceac74812b82a64" } modify_external_catalogs: true
I20260130 02:58:15.991068 18868 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50284:
table { table_name: "default.Table_748cb9c252084fb48ceac74812b82a64" } modify_external_catalogs: true
I20260130 02:58:15.971395 20497 leader_election.cc:290] T a90712651a604b83ba187f893ab17711 P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767), d870164263bb44b6b780a557daea3656 (127.18.100.67:39571)
I20260130 02:58:15.988447 20690 ts_tablet_manager.cc:1950] T d27ca7dd87f44102b7680e1f4a88954a P 371a36d219164974abdf848841528152: Deleting consensus metadata
W20260130 02:58:15.993371 18866 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.18.100.67:47213 (request call id 29) took 2461 ms (client timeout 1000 ms). Trace:
W20260130 02:58:15.993706 18866 rpcz_store.cc:269] 0130 02:58:13.532354 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:58:13.560862 (+ 28508us) service_pool.cc:225] Handling call
0130 02:58:13.827645 (+266783us) spinlock_profiling.cc:243] Waited 101 ms on lock 0x7b5400030554. stack: 00007fb0bd3f7373 00007fb0bd3f7270 00007fb0bcf6c574 0000000000508744 00000000005086d9 00007fb0c44490c0 00007fb0c44e7bba 00007fb0c61e6eac 00007fb0c62e110e 00007fb0c3efe4a4 00007fb0c3efe432 00007fb0c3efe361 00007fb0c3efe2dc 00007fb0c3efd592 00007fb0c21dd34c 00007fb0c21dc786
0130 02:58:15.993288 (+2165643us) inbound_call.cc:173] Queueing success response
Metrics: {"reported_tablets":43,"spinlock_wait_cycles":231239424}
I20260130 02:58:15.997104 19525 raft_consensus.cc:2468] T a90712651a604b83ba187f893ab17711 P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 371a36d219164974abdf848841528152 in term 1.
W20260130 02:58:15.876305 18872 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.18.100.65:51021 (request call id 35) took 2898 ms (client timeout 1000 ms). Trace:
I20260130 02:58:15.999430 19625 tablet_bootstrap.cc:654] T e2aaaf00f53d418b87d5c7af286e4d84 P d870164263bb44b6b780a557daea3656: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:15.999301 19159 leader_election.cc:304] T a90712651a604b83ba187f893ab17711 P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 371a36d219164974abdf848841528152, d870164263bb44b6b780a557daea3656; no voters:
I20260130 02:58:16.000073 20616 raft_consensus.cc:2804] T a90712651a604b83ba187f893ab17711 P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Leader election won for term 1
I20260130 02:58:16.000689 20616 raft_consensus.cc:697] T a90712651a604b83ba187f893ab17711 P 371a36d219164974abdf848841528152 [term 1 LEADER]: Becoming Leader. State: Replica: 371a36d219164974abdf848841528152, State: Running, Role: LEADER
W20260130 02:58:15.999522 18872 rpcz_store.cc:269] 0130 02:58:12.977832 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:58:13.235544 (+257712us) service_pool.cc:225] Handling call
0130 02:58:14.851847 (+1616303us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:58:14.851873 (+ 26us) write_op.cc:454] Released schema lock
0130 02:58:15.876234 (+1024361us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0130 02:58:14.806091 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0130 02:58:14.806613 (+ 522us) write_op.cc:432] Acquiring schema lock in shared mode
0130 02:58:14.806638 (+ 25us) write_op.cc:435] Acquired schema lock
0130 02:58:14.806655 (+ 17us) tablet.cc:660] Decoding operations
0130 02:58:14.807215 (+ 560us) write_op.cc:620] Acquiring the partition lock for write op
0130 02:58:14.807261 (+ 46us) write_op.cc:641] Partition lock acquired for write op
0130 02:58:14.807286 (+ 25us) tablet.cc:683] Acquiring locks for 5 operations
0130 02:58:14.807823 (+ 537us) tablet.cc:699] Row locks acquired
0130 02:58:14.807837 (+ 14us) write_op.cc:260] PREPARE: finished
0130 02:58:14.807962 (+ 125us) write_op.cc:270] Start()
0130 02:58:14.808052 (+ 90us) write_op.cc:276] Timestamp: P: 1769741894807904 usec, L: 0
0130 02:58:14.808070 (+ 18us) op_driver.cc:348] REPLICATION: starting
0130 02:58:14.808606 (+ 536us) log.cc:844] Serialized 2054 byte log entry
0130 02:58:14.818193 (+ 9587us) op_driver.cc:464] REPLICATION: finished
0130 02:58:14.818522 (+ 329us) write_op.cc:301] APPLY: starting
0130 02:58:14.818594 (+ 72us) tablet.cc:1365] starting BulkCheckPresence
0130 02:58:14.818735 (+ 141us) tablet.cc:1368] finished BulkCheckPresence
0130 02:58:14.818833 (+ 98us) tablet.cc:1370] starting ApplyRowOperation cycle
0130 02:58:14.819342 (+ 509us) tablet.cc:1381] finished ApplyRowOperation cycle
0130 02:58:14.819517 (+ 175us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=5
0130 02:58:14.819559 (+ 42us) write_op.cc:312] APPLY: finished
0130 02:58:14.819761 (+ 202us) log.cc:844] Serialized 47 byte log entry
0130 02:58:14.819895 (+ 134us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:58:14.819977 (+ 82us) write_op.cc:454] Released schema lock
0130 02:58:14.820009 (+ 32us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"mutex_wait_us":191345,"reported_tablets":42,"child_traces":[["op",{"apply.queue_time_us":197,"apply.run_cpu_time_us":1648,"apply.run_wall_time_us":1651,"num_ops":5,"prepare.queue_time_us":108,"prepare.run_cpu_time_us":2862,"prepare.run_wall_time_us":6144,"raft.queue_time_us":2072,"raft.run_cpu_time_us":704,"raft.run_wall_time_us":704,"replication_time_us":10031}]]}
I20260130 02:58:16.007936 20616 consensus_queue.cc:237] T a90712651a604b83ba187f893ab17711 P 371a36d219164974abdf848841528152 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:16.012602 20571 ts_tablet_manager.cc:1929] T 38a7a10e36874183bf9fe55a38d829f6 P d870164263bb44b6b780a557daea3656: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260130 02:58:16.012961 20571 log.cc:1199] T 38a7a10e36874183bf9fe55a38d829f6 P d870164263bb44b6b780a557daea3656: Deleting WAL directory at /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-2/wal/wals/38a7a10e36874183bf9fe55a38d829f6
I20260130 02:58:16.013622 20571 ts_tablet_manager.cc:1950] T 38a7a10e36874183bf9fe55a38d829f6 P d870164263bb44b6b780a557daea3656: Deleting consensus metadata
I20260130 02:58:16.016366 19523 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6218f2f7d73a42428b57a45858f53afd" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
I20260130 02:58:15.914865 19355 tablet_service.cc:1552] Processing DeleteTablet for tablet e364f4e70e5749e083e14e7fa7416ff0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:58:10 UTC) from {username='slave'} at 127.0.0.1:60110
I20260130 02:58:15.915724 19350 tablet_service.cc:1552] Processing DeleteTablet for tablet d27ca7dd87f44102b7680e1f4a88954a with delete_type TABLET_DATA_DELETED (replaced by tablet d480bf18140245e3870a62582d5a8318) from {username='slave'} at 127.0.0.1:60110
I20260130 02:58:16.027588 19807 tablet_bootstrap.cc:492] T cf1b66a20fce4c4d96bc0bc32a7bafa8 P 371a36d219164974abdf848841528152: Bootstrap starting.
I20260130 02:58:15.916520 19351 tablet_service.cc:1552] Processing DeleteTablet for tablet 7b80b22bd44c4233837c951aa97e6996 with delete_type TABLET_DATA_DELETED (replaced by tablet 1fc8f09d6329407c8078542b4123ef4d) from {username='slave'} at 127.0.0.1:60110
I20260130 02:58:16.028354 20571 tablet_replica.cc:333] T d27ca7dd87f44102b7680e1f4a88954a P d870164263bb44b6b780a557daea3656: stopping tablet replica
I20260130 02:58:16.028956 20570 tablet_replica.cc:333] T d27ca7dd87f44102b7680e1f4a88954a P 78c2c46c7520492db35d51c244fb4f74: stopping tablet replica
I20260130 02:58:16.029012 20571 raft_consensus.cc:2243] T d27ca7dd87f44102b7680e1f4a88954a P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260130 02:58:16.029455 20571 raft_consensus.cc:2272] T d27ca7dd87f44102b7680e1f4a88954a P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260130 02:58:15.916817 19354 tablet_service.cc:1552] Processing DeleteTablet for tablet b2ac91da01954c2390b6b7dcc06c394f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:58:04 UTC) from {username='slave'} at 127.0.0.1:60110
I20260130 02:58:15.917308 19358 tablet_service.cc:1552] Processing DeleteTablet for tablet 7d96e93d95f14af79cfb268626559b1e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:58:04 UTC) from {username='slave'} at 127.0.0.1:60110
I20260130 02:58:15.918318 19353 tablet_service.cc:1552] Processing DeleteTablet for tablet 3b148691090e4ee581d46f9845a6e49a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:58:10 UTC) from {username='slave'} at 127.0.0.1:60110
I20260130 02:58:15.920055 19380 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "78e9bc3e55fe4e4a98e8eb01da59d365" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74"
I20260130 02:58:16.031553 19380 raft_consensus.cc:3060] T 78e9bc3e55fe4e4a98e8eb01da59d365 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:16.034240 20571 ts_tablet_manager.cc:1916] T d27ca7dd87f44102b7680e1f4a88954a P d870164263bb44b6b780a557daea3656: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260130 02:58:15.929611 19840 tablet_bootstrap.cc:492] T 3aa76993beb94b12bc706a76db26a03b P 78c2c46c7520492db35d51c244fb4f74: No bootstrap required, opened a new log
I20260130 02:58:16.034873 19840 ts_tablet_manager.cc:1403] T 3aa76993beb94b12bc706a76db26a03b P 78c2c46c7520492db35d51c244fb4f74: Time spent bootstrapping tablet: real 0.280s user 0.010s sys 0.002s
I20260130 02:58:16.037166 19840 raft_consensus.cc:359] T 3aa76993beb94b12bc706a76db26a03b P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:16.037783 19840 raft_consensus.cc:385] T 3aa76993beb94b12bc706a76db26a03b P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:16.038009 19840 raft_consensus.cc:740] T 3aa76993beb94b12bc706a76db26a03b P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78c2c46c7520492db35d51c244fb4f74, State: Initialized, Role: FOLLOWER
I20260130 02:58:15.964563 19376 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a90712651a604b83ba187f893ab17711" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74" is_pre_election: true
I20260130 02:58:16.038681 19376 raft_consensus.cc:2468] T a90712651a604b83ba187f893ab17711 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 371a36d219164974abdf848841528152 in term 0.
I20260130 02:58:15.988868 19375 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a90712651a604b83ba187f893ab17711" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74"
I20260130 02:58:16.007166 20703 raft_consensus.cc:493] T af0f90071a334016aa17fae2e3a97fc8 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:16.039535 19375 raft_consensus.cc:3060] T a90712651a604b83ba187f893ab17711 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:16.020773 19356 tablet_service.cc:1552] Processing DeleteTablet for tablet 38a7a10e36874183bf9fe55a38d829f6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:58:14 UTC) from {username='slave'} at 127.0.0.1:60110
I20260130 02:58:16.039583 20703 raft_consensus.cc:515] T af0f90071a334016aa17fae2e3a97fc8 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:16.021384 19360 tablet_service.cc:1552] Processing DeleteTablet for tablet d55291d7cea64a0d9a345ce712fbfa80 with delete_type TABLET_DATA_DELETED (replaced by tablet cd30d7adcc4545848e3501a205a10293) from {username='slave'} at 127.0.0.1:60110
I20260130 02:58:16.041445 20703 leader_election.cc:290] T af0f90071a334016aa17fae2e3a97fc8 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), d870164263bb44b6b780a557daea3656 (127.18.100.67:39571)
I20260130 02:58:16.043666 19840 consensus_queue.cc:260] T 3aa76993beb94b12bc706a76db26a03b P 78c2c46c7520492db35d51c244fb4f74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:16.046131 19840 ts_tablet_manager.cc:1434] T 3aa76993beb94b12bc706a76db26a03b P 78c2c46c7520492db35d51c244fb4f74: Time spent starting tablet: real 0.011s user 0.003s sys 0.001s
I20260130 02:58:16.046962 19840 tablet_bootstrap.cc:492] T 12c3f3ec523040e7b7cd356e3da57e0d P 78c2c46c7520492db35d51c244fb4f74: Bootstrap starting.
I20260130 02:58:16.046830 19225 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "af0f90071a334016aa17fae2e3a97fc8" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152" is_pre_election: true
W20260130 02:58:16.047937 19314 leader_election.cc:343] T af0f90071a334016aa17fae2e3a97fc8 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:16.053766 19807 tablet_bootstrap.cc:654] T cf1b66a20fce4c4d96bc0bc32a7bafa8 P 371a36d219164974abdf848841528152: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:16.058723 19375 raft_consensus.cc:2468] T a90712651a604b83ba187f893ab17711 P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 371a36d219164974abdf848841528152 in term 1.
I20260130 02:58:16.054337 19380 raft_consensus.cc:2468] T 78e9bc3e55fe4e4a98e8eb01da59d365 P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d870164263bb44b6b780a557daea3656 in term 1.
I20260130 02:58:16.061862 20571 ts_tablet_manager.cc:1929] T d27ca7dd87f44102b7680e1f4a88954a P d870164263bb44b6b780a557daea3656: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260130 02:58:16.062180 20571 log.cc:1199] T d27ca7dd87f44102b7680e1f4a88954a P d870164263bb44b6b780a557daea3656: Deleting WAL directory at /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-2/wal/wals/d27ca7dd87f44102b7680e1f4a88954a
I20260130 02:58:16.062178 19840 tablet_bootstrap.cc:654] T 12c3f3ec523040e7b7cd356e3da57e0d P 78c2c46c7520492db35d51c244fb4f74: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:16.063169 20571 ts_tablet_manager.cc:1950] T d27ca7dd87f44102b7680e1f4a88954a P d870164263bb44b6b780a557daea3656: Deleting consensus metadata
I20260130 02:58:16.064570 20571 tablet_replica.cc:333] T 3389db58493448aea3cf00aca411047c P d870164263bb44b6b780a557daea3656: stopping tablet replica
I20260130 02:58:16.065267 20571 raft_consensus.cc:2243] T 3389db58493448aea3cf00aca411047c P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260130 02:58:16.065697 20571 raft_consensus.cc:2272] T 3389db58493448aea3cf00aca411047c P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260130 02:58:16.067795 20571 ts_tablet_manager.cc:1916] T 3389db58493448aea3cf00aca411047c P d870164263bb44b6b780a557daea3656: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260130 02:58:16.076917 19625 tablet_bootstrap.cc:492] T e2aaaf00f53d418b87d5c7af286e4d84 P d870164263bb44b6b780a557daea3656: No bootstrap required, opened a new log
I20260130 02:58:16.077382 19625 ts_tablet_manager.cc:1403] T e2aaaf00f53d418b87d5c7af286e4d84 P d870164263bb44b6b780a557daea3656: Time spent bootstrapping tablet: real 0.229s user 0.009s sys 0.004s
I20260130 02:58:16.080473 19625 raft_consensus.cc:359] T e2aaaf00f53d418b87d5c7af286e4d84 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:16.084162 19807 tablet_bootstrap.cc:492] T cf1b66a20fce4c4d96bc0bc32a7bafa8 P 371a36d219164974abdf848841528152: No bootstrap required, opened a new log
I20260130 02:58:16.084545 19807 ts_tablet_manager.cc:1403] T cf1b66a20fce4c4d96bc0bc32a7bafa8 P 371a36d219164974abdf848841528152: Time spent bootstrapping tablet: real 0.057s user 0.001s sys 0.009s
I20260130 02:58:16.103067 19807 raft_consensus.cc:359] T cf1b66a20fce4c4d96bc0bc32a7bafa8 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:16.103758 19807 raft_consensus.cc:385] T cf1b66a20fce4c4d96bc0bc32a7bafa8 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:16.104005 19807 raft_consensus.cc:740] T cf1b66a20fce4c4d96bc0bc32a7bafa8 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 371a36d219164974abdf848841528152, State: Initialized, Role: FOLLOWER
I20260130 02:58:16.104537 19807 consensus_queue.cc:260] T cf1b66a20fce4c4d96bc0bc32a7bafa8 P 371a36d219164974abdf848841528152 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:16.106647 19807 ts_tablet_manager.cc:1434] T cf1b66a20fce4c4d96bc0bc32a7bafa8 P 371a36d219164974abdf848841528152: Time spent starting tablet: real 0.022s user 0.006s sys 0.001s
I20260130 02:58:16.107460 19807 tablet_bootstrap.cc:492] T 6218f2f7d73a42428b57a45858f53afd P 371a36d219164974abdf848841528152: Bootstrap starting.
I20260130 02:58:16.113471 19807 tablet_bootstrap.cc:654] T 6218f2f7d73a42428b57a45858f53afd P 371a36d219164974abdf848841528152: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:16.113440 19625 raft_consensus.cc:385] T e2aaaf00f53d418b87d5c7af286e4d84 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:16.113869 19625 raft_consensus.cc:740] T e2aaaf00f53d418b87d5c7af286e4d84 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d870164263bb44b6b780a557daea3656, State: Initialized, Role: FOLLOWER
I20260130 02:58:16.097661 19526 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "af0f90071a334016aa17fae2e3a97fc8" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
I20260130 02:58:16.100322 20718 consensus_queue.cc:1048] T f41f4cff7dd4489f948700a45e9a7057 P d870164263bb44b6b780a557daea3656 [LEADER]: Connected to new peer: Peer: permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260130 02:58:16.136628 19625 consensus_queue.cc:260] T e2aaaf00f53d418b87d5c7af286e4d84 P d870164263bb44b6b780a557daea3656 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
W20260130 02:58:16.079031 20720 log.cc:927] Time spent T 16691c9315e94253bb9b3fb5058e5636 P d870164263bb44b6b780a557daea3656: Append to log took a long time: real 0.064s user 0.001s sys 0.000s
I20260130 02:58:16.014890 18860 catalog_manager.cc:4985] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): tablet 38a7a10e36874183bf9fe55a38d829f6 (table default.Table_f359d249f8274f319f5ff4e4ea89dd22 [id=27adf356fd004597a54c7a9887dd3ca8]) successfully deleted
W20260130 02:58:16.142342 18860 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.18.100.67:39571, user_credentials={real_user=slave}} blocked reactor thread for 127465us
I20260130 02:58:16.142995 18872 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50284:
name: "default.Table_4dde1730dacb42728dd8375fdaa37615"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260130 02:58:16.144131 18860 catalog_manager.cc:4952] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767): delete failed for tablet e364f4e70e5749e083e14e7fa7416ff0 because the tablet was not found. No further retry: Not found: Tablet not found: e364f4e70e5749e083e14e7fa7416ff0
I20260130 02:58:16.019197 18873 catalog_manager.cc:6199] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4: default.Table_dd22e9dc63f84d8688c0071f427f0b25 [id=77979d498aa040828de415129cb15a06] alter complete (version 1)
W20260130 02:58:16.144703 18860 catalog_manager.cc:4952] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): delete failed for tablet e364f4e70e5749e083e14e7fa7416ff0 because the tablet was not found. No further retry: Not found: Tablet not found: e364f4e70e5749e083e14e7fa7416ff0
W20260130 02:58:16.145253 18860 catalog_manager.cc:4952] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): delete failed for tablet b2ac91da01954c2390b6b7dcc06c394f because the tablet was not found. No further retry: Not found: Tablet not found: b2ac91da01954c2390b6b7dcc06c394f
W20260130 02:58:16.145776 18860 catalog_manager.cc:4952] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): delete failed for tablet 8bd3e1c06b89426da57e2b259e7983af because the tablet was not found. No further retry: Not found: Tablet not found: 8bd3e1c06b89426da57e2b259e7983af
I20260130 02:58:16.146255 18860 catalog_manager.cc:4985] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): tablet d27ca7dd87f44102b7680e1f4a88954a (table default.Table_109c8b6c17d4442b939bde9441d0476f [id=dcf82ea15d6044c2b023f81f0a579ade]) successfully deleted
I20260130 02:58:16.148927 19625 ts_tablet_manager.cc:1434] T e2aaaf00f53d418b87d5c7af286e4d84 P d870164263bb44b6b780a557daea3656: Time spent starting tablet: real 0.071s user 0.004s sys 0.000s
I20260130 02:58:16.079510 18866 master_service.cc:954] ReplaceTablet: received request to replace tablet 1ca34772ef484c1f809a99d126e28eff from {username='slave'} at 127.0.0.1:50284
I20260130 02:58:16.158433 19360 tablet_service.cc:1552] Processing DeleteTablet for tablet 3389db58493448aea3cf00aca411047c with delete_type TABLET_DATA_DELETED (replaced by tablet af0f90071a334016aa17fae2e3a97fc8) from {username='slave'} at 127.0.0.1:60110
W20260130 02:58:16.165405 18869 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.18.100.66:59691 (request call id 31) took 3835 ms (client timeout 1000 ms). Trace:
I20260130 02:58:16.167174 19625 tablet_bootstrap.cc:492] T 6218f2f7d73a42428b57a45858f53afd P d870164263bb44b6b780a557daea3656: Bootstrap starting.
W20260130 02:58:16.141597 18874 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5e70ab0e32bc4048bd19388f9d7cd302 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260130 02:58:15.928021 18861 catalog_manager.cc:4952] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021): delete failed for tablet e364f4e70e5749e083e14e7fa7416ff0 because the tablet was not found. No further retry: Not found: Tablet not found: e364f4e70e5749e083e14e7fa7416ff0
W20260130 02:58:16.180020 18861 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.18.100.65:42021, user_credentials={real_user=slave}} blocked reactor thread for 252003us
W20260130 02:58:16.180688 18861 catalog_manager.cc:4952] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021): delete failed for tablet 3b148691090e4ee581d46f9845a6e49a because the tablet was not found. No further retry: Not found: Tablet not found: 3b148691090e4ee581d46f9845a6e49a
I20260130 02:58:16.181599 18861 catalog_manager.cc:4985] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021): tablet d27ca7dd87f44102b7680e1f4a88954a (table default.Table_109c8b6c17d4442b939bde9441d0476f [id=dcf82ea15d6044c2b023f81f0a579ade]) successfully deleted
W20260130 02:58:16.182190 18861 catalog_manager.cc:4952] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021): delete failed for tablet 7d96e93d95f14af79cfb268626559b1e because the tablet was not found. No further retry: Not found: Tablet not found: 7d96e93d95f14af79cfb268626559b1e
W20260130 02:58:16.182811 18861 catalog_manager.cc:4952] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021): delete failed for tablet d55291d7cea64a0d9a345ce712fbfa80 because the tablet was not found. No further retry: Not found: Tablet not found: d55291d7cea64a0d9a345ce712fbfa80
W20260130 02:58:16.191537 18872 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4dde1730dacb42728dd8375fdaa37615 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260130 02:58:16.218894 20570 raft_consensus.cc:2243] T d27ca7dd87f44102b7680e1f4a88954a P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260130 02:58:16.219612 20570 raft_consensus.cc:2272] T d27ca7dd87f44102b7680e1f4a88954a P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260130 02:58:16.221660 20570 ts_tablet_manager.cc:1916] T d27ca7dd87f44102b7680e1f4a88954a P 78c2c46c7520492db35d51c244fb4f74: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260130 02:58:16.229408 19840 tablet_bootstrap.cc:492] T 12c3f3ec523040e7b7cd356e3da57e0d P 78c2c46c7520492db35d51c244fb4f74: No bootstrap required, opened a new log
I20260130 02:58:16.229807 19840 ts_tablet_manager.cc:1403] T 12c3f3ec523040e7b7cd356e3da57e0d P 78c2c46c7520492db35d51c244fb4f74: Time spent bootstrapping tablet: real 0.183s user 0.008s sys 0.000s
I20260130 02:58:16.237928 20697 raft_consensus.cc:493] T 13f0ace4d9de43abab08eca32cdb30c1 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:16.238376 20697 raft_consensus.cc:515] T 13f0ace4d9de43abab08eca32cdb30c1 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:16.241027 20697 leader_election.cc:290] T 13f0ace4d9de43abab08eca32cdb30c1 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), d870164263bb44b6b780a557daea3656 (127.18.100.67:39571)
I20260130 02:58:16.249565 19840 raft_consensus.cc:359] T 12c3f3ec523040e7b7cd356e3da57e0d P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:16.251890 20608 raft_consensus.cc:493] T e2aaaf00f53d418b87d5c7af286e4d84 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:16.252321 20608 raft_consensus.cc:515] T e2aaaf00f53d418b87d5c7af286e4d84 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:16.254179 20608 leader_election.cc:290] T e2aaaf00f53d418b87d5c7af286e4d84 P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767), d870164263bb44b6b780a557daea3656 (127.18.100.67:39571)
I20260130 02:58:16.255702 19222 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "13f0ace4d9de43abab08eca32cdb30c1" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152" is_pre_election: true
I20260130 02:58:16.262038 19625 tablet_bootstrap.cc:654] T 6218f2f7d73a42428b57a45858f53afd P d870164263bb44b6b780a557daea3656: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:16.262337 19524 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e2aaaf00f53d418b87d5c7af286e4d84" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
I20260130 02:58:16.262931 19524 raft_consensus.cc:2468] T e2aaaf00f53d418b87d5c7af286e4d84 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 371a36d219164974abdf848841528152 in term 0.
I20260130 02:58:16.266733 19159 leader_election.cc:304] T e2aaaf00f53d418b87d5c7af286e4d84 P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 371a36d219164974abdf848841528152, d870164263bb44b6b780a557daea3656; no voters:
I20260130 02:58:16.267364 20608 raft_consensus.cc:2804] T e2aaaf00f53d418b87d5c7af286e4d84 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260130 02:58:16.267730 20608 raft_consensus.cc:493] T e2aaaf00f53d418b87d5c7af286e4d84 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260130 02:58:16.268036 20608 raft_consensus.cc:3060] T e2aaaf00f53d418b87d5c7af286e4d84 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:16.275839 20571 ts_tablet_manager.cc:1929] T 3389db58493448aea3cf00aca411047c P d870164263bb44b6b780a557daea3656: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260130 02:58:16.276221 20571 log.cc:1199] T 3389db58493448aea3cf00aca411047c P d870164263bb44b6b780a557daea3656: Deleting WAL directory at /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-2/wal/wals/3389db58493448aea3cf00aca411047c
I20260130 02:58:16.274724 20608 raft_consensus.cc:515] T e2aaaf00f53d418b87d5c7af286e4d84 P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:16.276865 20571 ts_tablet_manager.cc:1950] T 3389db58493448aea3cf00aca411047c P d870164263bb44b6b780a557daea3656: Deleting consensus metadata
I20260130 02:58:16.277657 20608 leader_election.cc:290] T e2aaaf00f53d418b87d5c7af286e4d84 P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 election: Requested vote from peers 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767), d870164263bb44b6b780a557daea3656 (127.18.100.67:39571)
I20260130 02:58:16.283752 19526 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e2aaaf00f53d418b87d5c7af286e4d84" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656"
I20260130 02:58:16.284332 19526 raft_consensus.cc:3060] T e2aaaf00f53d418b87d5c7af286e4d84 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:16.293627 19526 raft_consensus.cc:2468] T e2aaaf00f53d418b87d5c7af286e4d84 P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 371a36d219164974abdf848841528152 in term 1.
I20260130 02:58:16.294723 19159 leader_election.cc:304] T e2aaaf00f53d418b87d5c7af286e4d84 P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 371a36d219164974abdf848841528152, d870164263bb44b6b780a557daea3656; no voters:
I20260130 02:58:16.295642 20736 raft_consensus.cc:2804] T e2aaaf00f53d418b87d5c7af286e4d84 P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Leader election won for term 1
I20260130 02:58:16.296073 20736 raft_consensus.cc:697] T e2aaaf00f53d418b87d5c7af286e4d84 P 371a36d219164974abdf848841528152 [term 1 LEADER]: Becoming Leader. State: Replica: 371a36d219164974abdf848841528152, State: Running, Role: LEADER
I20260130 02:58:16.296897 20736 consensus_queue.cc:237] T e2aaaf00f53d418b87d5c7af286e4d84 P 371a36d219164974abdf848841528152 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:16.304010 19807 tablet_bootstrap.cc:492] T 6218f2f7d73a42428b57a45858f53afd P 371a36d219164974abdf848841528152: No bootstrap required, opened a new log
I20260130 02:58:16.304452 19807 ts_tablet_manager.cc:1403] T 6218f2f7d73a42428b57a45858f53afd P 371a36d219164974abdf848841528152: Time spent bootstrapping tablet: real 0.197s user 0.039s sys 0.051s
I20260130 02:58:16.306958 19807 raft_consensus.cc:359] T 6218f2f7d73a42428b57a45858f53afd P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:16.307678 19807 raft_consensus.cc:385] T 6218f2f7d73a42428b57a45858f53afd P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:16.307986 19807 raft_consensus.cc:740] T 6218f2f7d73a42428b57a45858f53afd P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 371a36d219164974abdf848841528152, State: Initialized, Role: FOLLOWER
I20260130 02:58:16.308645 19807 consensus_queue.cc:260] T 6218f2f7d73a42428b57a45858f53afd P 371a36d219164974abdf848841528152 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:16.310688 19807 ts_tablet_manager.cc:1434] T 6218f2f7d73a42428b57a45858f53afd P 371a36d219164974abdf848841528152: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260130 02:58:16.311650 19807 tablet_bootstrap.cc:492] T f9e5f01e69254d658c369f3e21cef523 P 371a36d219164974abdf848841528152: Bootstrap starting.
I20260130 02:58:16.325318 19840 raft_consensus.cc:385] T 12c3f3ec523040e7b7cd356e3da57e0d P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20260130 02:58:16.243780 19313 leader_election.cc:343] T 6218f2f7d73a42428b57a45858f53afd P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:16.259598 19380 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e2aaaf00f53d418b87d5c7af286e4d84" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74" is_pre_election: true
I20260130 02:58:16.326798 19313 leader_election.cc:304] T 6218f2f7d73a42428b57a45858f53afd P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78c2c46c7520492db35d51c244fb4f74; no voters: 371a36d219164974abdf848841528152, d870164263bb44b6b780a557daea3656
I20260130 02:58:16.326930 19380 raft_consensus.cc:2468] T e2aaaf00f53d418b87d5c7af286e4d84 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 371a36d219164974abdf848841528152 in term 0.
I20260130 02:58:16.327497 20697 raft_consensus.cc:2749] T 6218f2f7d73a42428b57a45858f53afd P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260130 02:58:16.265605 19314 leader_election.cc:343] T 13f0ace4d9de43abab08eca32cdb30c1 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Illegal state: must be running to vote when last-logged opid is not known
W20260130 02:58:16.328218 19313 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.100.67:39571, user_credentials={real_user=slave}} blocked reactor thread for 84465.2us
W20260130 02:58:16.328450 19314 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.100.65:42021, user_credentials={real_user=slave}} blocked reactor thread for 62882.8us
I20260130 02:58:16.280279 19378 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e2aaaf00f53d418b87d5c7af286e4d84" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74"
I20260130 02:58:16.296919 20570 ts_tablet_manager.cc:1929] T d27ca7dd87f44102b7680e1f4a88954a P 78c2c46c7520492db35d51c244fb4f74: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260130 02:58:16.329495 20570 log.cc:1199] T d27ca7dd87f44102b7680e1f4a88954a P 78c2c46c7520492db35d51c244fb4f74: Deleting WAL directory at /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-1/wal/wals/d27ca7dd87f44102b7680e1f4a88954a
I20260130 02:58:16.330322 20570 ts_tablet_manager.cc:1950] T d27ca7dd87f44102b7680e1f4a88954a P 78c2c46c7520492db35d51c244fb4f74: Deleting consensus metadata
I20260130 02:58:16.329486 19378 raft_consensus.cc:3060] T e2aaaf00f53d418b87d5c7af286e4d84 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Advancing to term 1
W20260130 02:58:16.332641 19313 leader_election.cc:343] T af0f90071a334016aa17fae2e3a97fc8 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:16.326267 19840 raft_consensus.cc:740] T 12c3f3ec523040e7b7cd356e3da57e0d P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78c2c46c7520492db35d51c244fb4f74, State: Initialized, Role: FOLLOWER
I20260130 02:58:16.333060 19313 leader_election.cc:304] T af0f90071a334016aa17fae2e3a97fc8 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78c2c46c7520492db35d51c244fb4f74; no voters: 371a36d219164974abdf848841528152, d870164263bb44b6b780a557daea3656
I20260130 02:58:16.339264 20697 raft_consensus.cc:2749] T af0f90071a334016aa17fae2e3a97fc8 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260130 02:58:16.342310 19378 raft_consensus.cc:2468] T e2aaaf00f53d418b87d5c7af286e4d84 P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 371a36d219164974abdf848841528152 in term 1.
I20260130 02:58:16.345247 20570 tablet_replica.cc:333] T 3389db58493448aea3cf00aca411047c P 78c2c46c7520492db35d51c244fb4f74: stopping tablet replica
I20260130 02:58:16.348429 18860 catalog_manager.cc:4985] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): tablet 3389db58493448aea3cf00aca411047c (table default.Table_22d10332a83f4700ac21892bab67f46b [id=da3a9cb1f6d5419abc1afc93659b5324]) successfully deleted
I20260130 02:58:16.349495 18860 catalog_manager.cc:4985] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767): tablet d27ca7dd87f44102b7680e1f4a88954a (table default.Table_109c8b6c17d4442b939bde9441d0476f [id=dcf82ea15d6044c2b023f81f0a579ade]) successfully deleted
W20260130 02:58:16.350176 18860 catalog_manager.cc:4952] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767): delete failed for tablet 7b80b22bd44c4233837c951aa97e6996 because the tablet was not found. No further retry: Not found: Tablet not found: 7b80b22bd44c4233837c951aa97e6996
I20260130 02:58:16.350920 20570 raft_consensus.cc:2243] T 3389db58493448aea3cf00aca411047c P 78c2c46c7520492db35d51c244fb4f74 [term 1 LEADER]: Raft consensus shutting down.
W20260130 02:58:16.351368 18860 catalog_manager.cc:4952] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767): delete failed for tablet b2ac91da01954c2390b6b7dcc06c394f because the tablet was not found. No further retry: Not found: Tablet not found: b2ac91da01954c2390b6b7dcc06c394f
I20260130 02:58:16.352430 20570 raft_consensus.cc:2272] T 3389db58493448aea3cf00aca411047c P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260130 02:58:16.355513 19360 tablet_service.cc:1552] Processing DeleteTablet for tablet e364f4e70e5749e083e14e7fa7416ff0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:58:10 UTC) from {username='slave'} at 127.0.0.1:60110
W20260130 02:58:16.165733 18869 rpcz_store.cc:269] 0130 02:58:12.329910 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:58:13.143806 (+813896us) service_pool.cc:225] Handling call
0130 02:58:16.165328 (+3021522us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0130 02:58:14.947323 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0130 02:58:14.947760 (+ 437us) write_op.cc:432] Acquiring schema lock in shared mode
0130 02:58:14.947786 (+ 26us) write_op.cc:435] Acquired schema lock
0130 02:58:14.947800 (+ 14us) tablet.cc:660] Decoding operations
0130 02:58:14.948746 (+ 946us) write_op.cc:620] Acquiring the partition lock for write op
0130 02:58:14.948794 (+ 48us) write_op.cc:641] Partition lock acquired for write op
0130 02:58:14.948819 (+ 25us) tablet.cc:683] Acquiring locks for 1 operations
0130 02:58:14.948971 (+ 152us) tablet.cc:699] Row locks acquired
0130 02:58:14.948984 (+ 13us) write_op.cc:260] PREPARE: finished
0130 02:58:14.949107 (+ 123us) write_op.cc:270] Start()
0130 02:58:14.949194 (+ 87us) write_op.cc:276] Timestamp: P: 1769741894949033 usec, L: 0
0130 02:58:14.949209 (+ 15us) op_driver.cc:348] REPLICATION: starting
0130 02:58:14.949721 (+ 512us) log.cc:844] Serialized 533 byte log entry
0130 02:58:14.956706 (+ 6985us) op_driver.cc:464] REPLICATION: finished
0130 02:58:14.957072 (+ 366us) write_op.cc:301] APPLY: starting
0130 02:58:14.957122 (+ 50us) tablet.cc:1365] starting BulkCheckPresence
0130 02:58:14.957205 (+ 83us) tablet.cc:1368] finished BulkCheckPresence
0130 02:58:14.957287 (+ 82us) tablet.cc:1370] starting ApplyRowOperation cycle
0130 02:58:14.957467 (+ 180us) tablet.cc:1381] finished ApplyRowOperation cycle
0130 02:58:14.957586 (+ 119us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0130 02:58:14.957631 (+ 45us) write_op.cc:312] APPLY: finished
0130 02:58:14.957800 (+ 169us) log.cc:844] Serialized 23 byte log entry
0130 02:58:14.957934 (+ 134us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:58:14.957997 (+ 63us) write_op.cc:454] Released schema lock
0130 02:58:14.958024 (+ 27us) write_op.cc:341] FINISH: Updating metrics
0130 02:58:14.958222 (+ 198us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:58:14.958245 (+ 23us) write_op.cc:454] Released schema lock
Related trace 'op':
0130 02:58:15.630854 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0130 02:58:15.631324 (+ 470us) write_op.cc:432] Acquiring schema lock in shared mode
0130 02:58:15.631350 (+ 26us) write_op.cc:435] Acquired schema lock
0130 02:58:15.631377 (+ 27us) tablet.cc:660] Decoding operations
0130 02:58:15.631680 (+ 303us) write_op.cc:620] Acquiring the partition lock for write op
0130 02:58:15.631725 (+ 45us) write_op.cc:641] Partition lock acquired for write op
0130 02:58:15.631750 (+ 25us) tablet.cc:683] Acquiring locks for 1 operations
0130 02:58:15.631896 (+ 146us) tablet.cc:699] Row locks acquired
0130 02:58:15.631912 (+ 16us) write_op.cc:260] PREPARE: finished
0130 02:58:15.632051 (+ 139us) write_op.cc:270] Start()
0130 02:58:15.632150 (+ 99us) write_op.cc:276] Timestamp: P: 1769741895631962 usec, L: 0
0130 02:58:15.632164 (+ 14us) op_driver.cc:348] REPLICATION: starting
0130 02:58:15.632710 (+ 546us) log.cc:844] Serialized 356 byte log entry
0130 02:58:15.656618 (+ 23908us) op_driver.cc:464] REPLICATION: finished
0130 02:58:15.656969 (+ 351us) write_op.cc:301] APPLY: starting
0130 02:58:15.657019 (+ 50us) tablet.cc:1365] starting BulkCheckPresence
0130 02:58:15.657137 (+ 118us) tablet.cc:1368] finished BulkCheckPresence
0130 02:58:15.657222 (+ 85us) tablet.cc:1370] starting ApplyRowOperation cycle
0130 02:58:15.657395 (+ 173us) tablet.cc:1381] finished ApplyRowOperation cycle
0130 02:58:15.657513 (+ 118us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0130 02:58:15.657555 (+ 42us) write_op.cc:312] APPLY: finished
0130 02:58:15.657721 (+ 166us) log.cc:844] Serialized 23 byte log entry
0130 02:58:15.657867 (+ 146us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:58:15.657924 (+ 57us) write_op.cc:454] Released schema lock
0130 02:58:15.657952 (+ 28us) write_op.cc:341] FINISH: Updating metrics
0130 02:58:15.658163 (+ 211us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:58:15.658186 (+ 23us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":97102,"reported_tablets":43,"child_traces":[["op",{"apply.queue_time_us":237,"apply.run_cpu_time_us":1126,"apply.run_wall_time_us":1125,"num_ops":1,"prepare.queue_time_us":113,"prepare.run_cpu_time_us":2747,"prepare.run_wall_time_us":2747,"raft.queue_time_us":517,"raft.run_cpu_time_us":589,"raft.run_wall_time_us":589,"replication_time_us":7410}],["op",{"apply.queue_time_us":222,"apply.run_cpu_time_us":1163,"apply.run_wall_time_us":1163,"num_ops":1,"prepare.queue_time_us":942,"prepare.run_cpu_time_us":2179,"prepare.run_wall_time_us":2179,"raft.queue_time_us":2246,"raft.run_cpu_time_us":588,"raft.run_wall_time_us":3474,"replication_time_us":24359}]]}
I20260130 02:58:16.359443 20570 ts_tablet_manager.cc:1916] T 3389db58493448aea3cf00aca411047c P 78c2c46c7520492db35d51c244fb4f74: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260130 02:58:16.367524 18860 catalog_manager.cc:4952] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767): delete failed for tablet 7d96e93d95f14af79cfb268626559b1e because the tablet was not found. No further retry: Not found: Tablet not found: 7d96e93d95f14af79cfb268626559b1e
W20260130 02:58:16.368214 18860 catalog_manager.cc:4952] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767): delete failed for tablet 3b148691090e4ee581d46f9845a6e49a because the tablet was not found. No further retry: Not found: Tablet not found: 3b148691090e4ee581d46f9845a6e49a
W20260130 02:58:16.368772 18860 catalog_manager.cc:4952] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767): delete failed for tablet 38a7a10e36874183bf9fe55a38d829f6 because the tablet was not found. No further retry: Not found: Tablet not found: 38a7a10e36874183bf9fe55a38d829f6
I20260130 02:58:16.372049 20608 consensus_queue.cc:1048] T b1e8b78eff0d4a0b92059adb2de4a3ce P 371a36d219164974abdf848841528152 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260130 02:58:16.387703 18860 catalog_manager.cc:4952] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767): delete failed for tablet d55291d7cea64a0d9a345ce712fbfa80 because the tablet was not found. No further retry: Not found: Tablet not found: d55291d7cea64a0d9a345ce712fbfa80
I20260130 02:58:16.387938 20497 raft_consensus.cc:493] T cf1b66a20fce4c4d96bc0bc32a7bafa8 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260130 02:58:16.388653 18860 catalog_manager.cc:4952] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): delete failed for tablet 7d96e93d95f14af79cfb268626559b1e because the tablet was not found. No further retry: Not found: Tablet not found: 7d96e93d95f14af79cfb268626559b1e
I20260130 02:58:16.388386 20497 raft_consensus.cc:515] T cf1b66a20fce4c4d96bc0bc32a7bafa8 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
W20260130 02:58:16.389290 18860 catalog_manager.cc:4952] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): delete failed for tablet 3b148691090e4ee581d46f9845a6e49a because the tablet was not found. No further retry: Not found: Tablet not found: 3b148691090e4ee581d46f9845a6e49a
I20260130 02:58:16.390228 20497 leader_election.cc:290] T cf1b66a20fce4c4d96bc0bc32a7bafa8 P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d870164263bb44b6b780a557daea3656 (127.18.100.67:39571), 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767)
I20260130 02:58:16.392426 19380 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cf1b66a20fce4c4d96bc0bc32a7bafa8" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74" is_pre_election: true
I20260130 02:58:16.392994 19380 raft_consensus.cc:2468] T cf1b66a20fce4c4d96bc0bc32a7bafa8 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 371a36d219164974abdf848841528152 in term 0.
I20260130 02:58:16.394199 19159 leader_election.cc:304] T cf1b66a20fce4c4d96bc0bc32a7bafa8 P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 371a36d219164974abdf848841528152, 78c2c46c7520492db35d51c244fb4f74; no voters:
I20260130 02:58:16.395076 20497 raft_consensus.cc:2804] T cf1b66a20fce4c4d96bc0bc32a7bafa8 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260130 02:58:16.395457 20497 raft_consensus.cc:493] T cf1b66a20fce4c4d96bc0bc32a7bafa8 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260130 02:58:16.395829 20497 raft_consensus.cc:3060] T cf1b66a20fce4c4d96bc0bc32a7bafa8 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:16.404170 19526 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cf1b66a20fce4c4d96bc0bc32a7bafa8" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
W20260130 02:58:16.406137 19159 leader_election.cc:343] T cf1b66a20fce4c4d96bc0bc32a7bafa8 P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:16.411552 20497 raft_consensus.cc:515] T cf1b66a20fce4c4d96bc0bc32a7bafa8 P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:16.421223 19526 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cf1b66a20fce4c4d96bc0bc32a7bafa8" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656"
W20260130 02:58:16.422323 19159 leader_election.cc:343] T cf1b66a20fce4c4d96bc0bc32a7bafa8 P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:16.429708 20570 ts_tablet_manager.cc:1929] T 3389db58493448aea3cf00aca411047c P 78c2c46c7520492db35d51c244fb4f74: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260130 02:58:16.431830 20616 consensus_queue.cc:1048] T b1e8b78eff0d4a0b92059adb2de4a3ce P 371a36d219164974abdf848841528152 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260130 02:58:16.436033 20497 leader_election.cc:290] T cf1b66a20fce4c4d96bc0bc32a7bafa8 P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 election: Requested vote from peers d870164263bb44b6b780a557daea3656 (127.18.100.67:39571), 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767)
I20260130 02:58:16.435547 19380 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cf1b66a20fce4c4d96bc0bc32a7bafa8" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74"
I20260130 02:58:16.436578 19380 raft_consensus.cc:3060] T cf1b66a20fce4c4d96bc0bc32a7bafa8 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:16.442893 19380 raft_consensus.cc:2468] T cf1b66a20fce4c4d96bc0bc32a7bafa8 P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 371a36d219164974abdf848841528152 in term 1.
I20260130 02:58:16.443907 19159 leader_election.cc:304] T cf1b66a20fce4c4d96bc0bc32a7bafa8 P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 371a36d219164974abdf848841528152, 78c2c46c7520492db35d51c244fb4f74; no voters: d870164263bb44b6b780a557daea3656
I20260130 02:58:16.444603 20497 raft_consensus.cc:2804] T cf1b66a20fce4c4d96bc0bc32a7bafa8 P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Leader election won for term 1
I20260130 02:58:16.445266 20497 raft_consensus.cc:697] T cf1b66a20fce4c4d96bc0bc32a7bafa8 P 371a36d219164974abdf848841528152 [term 1 LEADER]: Becoming Leader. State: Replica: 371a36d219164974abdf848841528152, State: Running, Role: LEADER
I20260130 02:58:16.445071 20570 log.cc:1199] T 3389db58493448aea3cf00aca411047c P 78c2c46c7520492db35d51c244fb4f74: Deleting WAL directory at /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-1/wal/wals/3389db58493448aea3cf00aca411047c
I20260130 02:58:16.446938 20570 ts_tablet_manager.cc:1950] T 3389db58493448aea3cf00aca411047c P 78c2c46c7520492db35d51c244fb4f74: Deleting consensus metadata
I20260130 02:58:16.448180 20497 consensus_queue.cc:237] T cf1b66a20fce4c4d96bc0bc32a7bafa8 P 371a36d219164974abdf848841528152 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:16.479018 18866 catalog_manager.cc:5941] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4: Sending DeleteTablet for 3 replicas of tablet 1ca34772ef484c1f809a99d126e28eff
I20260130 02:58:16.480566 18866 catalog_manager.cc:6724] ReplaceTablet: tablet 1ca34772ef484c1f809a99d126e28eff deleted and replaced by tablet 00ddbfe6218c44d788fbf65245a31dc6
I20260130 02:58:16.481022 18866 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:50284 (request call id 240) took 2177 ms. Trace:
I20260130 02:58:16.494213 19197 tablet_service.cc:1505] Processing CreateTablet for tablet 2b8034a92b2043e5b57cf02f1ee6296d (DEFAULT_TABLE table=default.Table_e2480022cdae4429b710395eebae7fec [id=fa72411018324474aa9a7265d9f1a2e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:16.494946 19202 tablet_service.cc:1552] Processing DeleteTablet for tablet 1ca34772ef484c1f809a99d126e28eff with delete_type TABLET_DATA_DELETED (replaced by tablet 00ddbfe6218c44d788fbf65245a31dc6) from {username='slave'} at 127.0.0.1:42870
I20260130 02:58:16.495194 19197 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2b8034a92b2043e5b57cf02f1ee6296d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:16.494993 19204 tablet_service.cc:1505] Processing CreateTablet for tablet 94355fc1338f479e8798070aff04fcba (DEFAULT_TABLE table=default.Table_12c2fe20cc3e45d29b7be37f900f68a9 [id=67ac4f4eda324e799b5867b2b011a16b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:16.496042 19204 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 94355fc1338f479e8798070aff04fcba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:16.497507 19356 tablet_service.cc:1552] Processing DeleteTablet for tablet 1ca34772ef484c1f809a99d126e28eff with delete_type TABLET_DATA_DELETED (replaced by tablet 00ddbfe6218c44d788fbf65245a31dc6) from {username='slave'} at 127.0.0.1:60110
I20260130 02:58:16.498252 20690 tablet_replica.cc:333] T 1ca34772ef484c1f809a99d126e28eff P 371a36d219164974abdf848841528152: stopping tablet replica
I20260130 02:58:16.498430 18860 catalog_manager.cc:4985] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767): tablet 3389db58493448aea3cf00aca411047c (table default.Table_22d10332a83f4700ac21892bab67f46b [id=da3a9cb1f6d5419abc1afc93659b5324]) successfully deleted
W20260130 02:58:16.499038 18860 catalog_manager.cc:4952] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767): delete failed for tablet e364f4e70e5749e083e14e7fa7416ff0 because the tablet was not found. No further retry: Not found: Tablet not found: e364f4e70e5749e083e14e7fa7416ff0
I20260130 02:58:16.499197 20690 raft_consensus.cc:2243] T 1ca34772ef484c1f809a99d126e28eff P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260130 02:58:16.499709 20690 raft_consensus.cc:2272] T 1ca34772ef484c1f809a99d126e28eff P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260130 02:58:16.501681 20690 ts_tablet_manager.cc:1916] T 1ca34772ef484c1f809a99d126e28eff P 371a36d219164974abdf848841528152: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260130 02:58:16.501873 20570 tablet_replica.cc:333] T 1ca34772ef484c1f809a99d126e28eff P 78c2c46c7520492db35d51c244fb4f74: stopping tablet replica
I20260130 02:58:16.502643 20570 raft_consensus.cc:2243] T 1ca34772ef484c1f809a99d126e28eff P 78c2c46c7520492db35d51c244fb4f74 [term 1 LEADER]: Raft consensus shutting down.
I20260130 02:58:16.338002 19840 consensus_queue.cc:260] T 12c3f3ec523040e7b7cd356e3da57e0d P 78c2c46c7520492db35d51c244fb4f74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:16.503402 20570 raft_consensus.cc:2272] T 1ca34772ef484c1f809a99d126e28eff P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260130 02:58:16.504334 19360 tablet_service.cc:1505] Processing CreateTablet for tablet 2b8034a92b2043e5b57cf02f1ee6296d (DEFAULT_TABLE table=default.Table_e2480022cdae4429b710395eebae7fec [id=fa72411018324474aa9a7265d9f1a2e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:16.505340 19360 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2b8034a92b2043e5b57cf02f1ee6296d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:16.505419 20570 ts_tablet_manager.cc:1916] T 1ca34772ef484c1f809a99d126e28eff P 78c2c46c7520492db35d51c244fb4f74: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260130 02:58:16.507604 19496 tablet_service.cc:1505] Processing CreateTablet for tablet 2b8034a92b2043e5b57cf02f1ee6296d (DEFAULT_TABLE table=default.Table_e2480022cdae4429b710395eebae7fec [id=fa72411018324474aa9a7265d9f1a2e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:16.508349 19501 tablet_service.cc:1552] Processing DeleteTablet for tablet 1ca34772ef484c1f809a99d126e28eff with delete_type TABLET_DATA_DELETED (replaced by tablet 00ddbfe6218c44d788fbf65245a31dc6) from {username='slave'} at 127.0.0.1:59068
I20260130 02:58:16.510073 19496 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2b8034a92b2043e5b57cf02f1ee6296d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:16.510178 20571 tablet_replica.cc:333] T 1ca34772ef484c1f809a99d126e28eff P d870164263bb44b6b780a557daea3656: stopping tablet replica
I20260130 02:58:16.510723 20697 raft_consensus.cc:493] T 12c3f3ec523040e7b7cd356e3da57e0d P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:16.510938 20571 raft_consensus.cc:2243] T 1ca34772ef484c1f809a99d126e28eff P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260130 02:58:16.511114 20697 raft_consensus.cc:515] T 12c3f3ec523040e7b7cd356e3da57e0d P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:16.511451 20571 raft_consensus.cc:2272] T 1ca34772ef484c1f809a99d126e28eff P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260130 02:58:16.513291 20571 ts_tablet_manager.cc:1916] T 1ca34772ef484c1f809a99d126e28eff P d870164263bb44b6b780a557daea3656: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260130 02:58:16.515641 19224 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "12c3f3ec523040e7b7cd356e3da57e0d" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152" is_pre_election: true
W20260130 02:58:16.516784 19314 leader_election.cc:343] T 12c3f3ec523040e7b7cd356e3da57e0d P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:16.517230 20697 leader_election.cc:290] T 12c3f3ec523040e7b7cd356e3da57e0d P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d870164263bb44b6b780a557daea3656 (127.18.100.67:39571), 371a36d219164974abdf848841528152 (127.18.100.65:42021)
I20260130 02:58:16.481405 18866 rpcz_store.cc:276] 0130 02:58:14.303753 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:58:16.079373 (+1775620us) service_pool.cc:225] Handling call
0130 02:58:16.480961 (+401588us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0130 02:58:16.201099 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0130 02:58:16.201578 (+ 479us) write_op.cc:432] Acquiring schema lock in shared mode
0130 02:58:16.202112 (+ 534us) write_op.cc:435] Acquired schema lock
0130 02:58:16.202130 (+ 18us) tablet.cc:660] Decoding operations
0130 02:58:16.202488 (+ 358us) write_op.cc:620] Acquiring the partition lock for write op
0130 02:58:16.202534 (+ 46us) write_op.cc:641] Partition lock acquired for write op
0130 02:58:16.202558 (+ 24us) tablet.cc:683] Acquiring locks for 2 operations
0130 02:58:16.202807 (+ 249us) tablet.cc:699] Row locks acquired
0130 02:58:16.202820 (+ 13us) write_op.cc:260] PREPARE: finished
0130 02:58:16.202938 (+ 118us) write_op.cc:270] Start()
0130 02:58:16.203042 (+ 104us) write_op.cc:276] Timestamp: P: 1769741896202892 usec, L: 0
0130 02:58:16.203060 (+ 18us) op_driver.cc:348] REPLICATION: starting
0130 02:58:16.203580 (+ 520us) log.cc:844] Serialized 662 byte log entry
0130 02:58:16.369809 (+166229us) op_driver.cc:464] REPLICATION: finished
0130 02:58:16.370492 (+ 683us) write_op.cc:301] APPLY: starting
0130 02:58:16.370559 (+ 67us) tablet.cc:1365] starting BulkCheckPresence
0130 02:58:16.370692 (+ 133us) tablet.cc:1368] finished BulkCheckPresence
0130 02:58:16.370788 (+ 96us) tablet.cc:1370] starting ApplyRowOperation cycle
0130 02:58:16.371159 (+ 371us) tablet.cc:1381] finished ApplyRowOperation cycle
0130 02:58:16.371414 (+ 255us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0130 02:58:16.371472 (+ 58us) write_op.cc:312] APPLY: finished
0130 02:58:16.371669 (+ 197us) log.cc:844] Serialized 29 byte log entry
0130 02:58:16.371820 (+ 151us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:58:16.371890 (+ 70us) write_op.cc:454] Released schema lock
0130 02:58:16.371923 (+ 33us) write_op.cc:341] FINISH: Updating metrics
0130 02:58:16.372173 (+ 250us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:58:16.372199 (+ 26us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":512,"apply.run_cpu_time_us":1639,"apply.run_wall_time_us":1639,"num_ops":2,"prepare.queue_time_us":48506,"prepare.run_cpu_time_us":2780,"prepare.run_wall_time_us":2791,"raft.queue_time_us":1213,"raft.run_cpu_time_us":531,"raft.run_wall_time_us":532,"replication_time_us":166614}]]}
I20260130 02:58:16.529680 19840 ts_tablet_manager.cc:1434] T 12c3f3ec523040e7b7cd356e3da57e0d P 78c2c46c7520492db35d51c244fb4f74: Time spent starting tablet: real 0.300s user 0.006s sys 0.001s
I20260130 02:58:16.532491 19840 tablet_bootstrap.cc:492] T 320dbe98a32a43b9899bbeed80b20f55 P 78c2c46c7520492db35d51c244fb4f74: Bootstrap starting.
W20260130 02:58:16.533056 18860 catalog_manager.cc:4952] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): delete failed for tablet f54addb1a4304ce69c711ae0cc3b452c because the tablet was not found. No further retry: Not found: Tablet not found: f54addb1a4304ce69c711ae0cc3b452c
I20260130 02:58:16.532439 19356 tablet_service.cc:1505] Processing CreateTablet for tablet 94355fc1338f479e8798070aff04fcba (DEFAULT_TABLE table=default.Table_12c2fe20cc3e45d29b7be37f900f68a9 [id=67ac4f4eda324e799b5867b2b011a16b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:16.534269 20736 raft_consensus.cc:493] T dcba6b534f9d406dad4fb502be40921c P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:16.534428 19356 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 94355fc1338f479e8798070aff04fcba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:16.534729 20736 raft_consensus.cc:515] T dcba6b534f9d406dad4fb502be40921c P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:16.536476 20736 leader_election.cc:290] T dcba6b534f9d406dad4fb502be40921c P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d870164263bb44b6b780a557daea3656 (127.18.100.67:39571), 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767)
I20260130 02:58:16.539722 19380 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dcba6b534f9d406dad4fb502be40921c" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74" is_pre_election: true
I20260130 02:58:16.540272 19380 raft_consensus.cc:2468] T dcba6b534f9d406dad4fb502be40921c P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 371a36d219164974abdf848841528152 in term 0.
I20260130 02:58:16.540231 19524 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dcba6b534f9d406dad4fb502be40921c" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
W20260130 02:58:16.541481 19159 leader_election.cc:343] T dcba6b534f9d406dad4fb502be40921c P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:16.542076 19159 leader_election.cc:304] T dcba6b534f9d406dad4fb502be40921c P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 371a36d219164974abdf848841528152, 78c2c46c7520492db35d51c244fb4f74; no voters: d870164263bb44b6b780a557daea3656
I20260130 02:58:16.548372 20736 raft_consensus.cc:2804] T dcba6b534f9d406dad4fb502be40921c P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260130 02:58:16.548702 20736 raft_consensus.cc:493] T dcba6b534f9d406dad4fb502be40921c P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260130 02:58:16.549062 20736 raft_consensus.cc:3060] T dcba6b534f9d406dad4fb502be40921c P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:16.549294 19807 tablet_bootstrap.cc:654] T f9e5f01e69254d658c369f3e21cef523 P 371a36d219164974abdf848841528152: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:16.551854 20571 ts_tablet_manager.cc:1929] T 1ca34772ef484c1f809a99d126e28eff P d870164263bb44b6b780a557daea3656: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260130 02:58:16.531526 20690 ts_tablet_manager.cc:1929] T 1ca34772ef484c1f809a99d126e28eff P 371a36d219164974abdf848841528152: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260130 02:58:16.552384 20690 log.cc:1199] T 1ca34772ef484c1f809a99d126e28eff P 371a36d219164974abdf848841528152: Deleting WAL directory at /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-0/wal/wals/1ca34772ef484c1f809a99d126e28eff
I20260130 02:58:16.553750 20690 ts_tablet_manager.cc:1950] T 1ca34772ef484c1f809a99d126e28eff P 371a36d219164974abdf848841528152: Deleting consensus metadata
I20260130 02:58:16.558420 20570 ts_tablet_manager.cc:1929] T 1ca34772ef484c1f809a99d126e28eff P 78c2c46c7520492db35d51c244fb4f74: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260130 02:58:16.558996 20570 log.cc:1199] T 1ca34772ef484c1f809a99d126e28eff P 78c2c46c7520492db35d51c244fb4f74: Deleting WAL directory at /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-1/wal/wals/1ca34772ef484c1f809a99d126e28eff
I20260130 02:58:16.559656 20570 ts_tablet_manager.cc:1950] T 1ca34772ef484c1f809a99d126e28eff P 78c2c46c7520492db35d51c244fb4f74: Deleting consensus metadata
I20260130 02:58:16.559620 20571 log.cc:1199] T 1ca34772ef484c1f809a99d126e28eff P d870164263bb44b6b780a557daea3656: Deleting WAL directory at /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-2/wal/wals/1ca34772ef484c1f809a99d126e28eff
I20260130 02:58:16.560426 20571 ts_tablet_manager.cc:1950] T 1ca34772ef484c1f809a99d126e28eff P d870164263bb44b6b780a557daea3656: Deleting consensus metadata
I20260130 02:58:16.561478 19840 tablet_bootstrap.cc:654] T 320dbe98a32a43b9899bbeed80b20f55 P 78c2c46c7520492db35d51c244fb4f74: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:16.563463 18861 catalog_manager.cc:4985] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021): tablet 1ca34772ef484c1f809a99d126e28eff (table default.Table_9cb9a9f9e5d145a6a42d7ccbc42aced9 [id=5f1f179fadc247da9382dd7796dcf2c2]) successfully deleted
I20260130 02:58:16.564401 18860 catalog_manager.cc:4985] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): tablet 1ca34772ef484c1f809a99d126e28eff (table default.Table_9cb9a9f9e5d145a6a42d7ccbc42aced9 [id=5f1f179fadc247da9382dd7796dcf2c2]) successfully deleted
I20260130 02:58:16.568534 19524 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "13f0ace4d9de43abab08eca32cdb30c1" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
I20260130 02:58:16.574749 18870 catalog_manager.cc:5746] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571) does not have the latest schema for tablet a8a5038e9def44eb8b3b1e4aeae25652 (table default.Table_dee034aec0b24982b8dc2a3fa1b326f1 [id=66033e75b3814a8584311cf7e4a07ad3]). Expected version 1 got 0
I20260130 02:58:16.577962 18860 catalog_manager.cc:4985] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767): tablet 1ca34772ef484c1f809a99d126e28eff (table default.Table_9cb9a9f9e5d145a6a42d7ccbc42aced9 [id=5f1f179fadc247da9382dd7796dcf2c2]) successfully deleted
I20260130 02:58:16.579334 20616 consensus_queue.cc:1048] T 5fbe92a2227d44d88b4531a7f8aa0e08 P 371a36d219164974abdf848841528152 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260130 02:58:16.579499 20746 tablet.cc:1722] T 0e8018c094024c4788c484e8f5466e00 P d870164263bb44b6b780a557daea3656: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260130 02:58:16.581694 20745 tablet.cc:1722] T 16691c9315e94253bb9b3fb5058e5636 P d870164263bb44b6b780a557daea3656: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260130 02:58:16.587446 20718 consensus_queue.cc:1048] T 6e12a8eeb3204325a9a8b7779f72ab29 P d870164263bb44b6b780a557daea3656 [LEADER]: Connected to new peer: Peer: permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260130 02:58:16.587706 20736 raft_consensus.cc:515] T dcba6b534f9d406dad4fb502be40921c P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:16.589758 20736 leader_election.cc:290] T dcba6b534f9d406dad4fb502be40921c P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 election: Requested vote from peers d870164263bb44b6b780a557daea3656 (127.18.100.67:39571), 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767)
I20260130 02:58:16.596107 19380 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dcba6b534f9d406dad4fb502be40921c" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74"
I20260130 02:58:16.596611 19380 raft_consensus.cc:3060] T dcba6b534f9d406dad4fb502be40921c P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:16.603842 19522 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dcba6b534f9d406dad4fb502be40921c" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656"
I20260130 02:58:16.604594 19380 raft_consensus.cc:2468] T dcba6b534f9d406dad4fb502be40921c P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 371a36d219164974abdf848841528152 in term 1.
I20260130 02:58:16.621439 19522 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "12c3f3ec523040e7b7cd356e3da57e0d" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
I20260130 02:58:16.614676 18870 catalog_manager.cc:5654] T 345eaaa5625c48d099280b7fbb2e514b P d870164263bb44b6b780a557daea3656 reported cstate change: term changed from 0 to 1, leader changed from <none> to d870164263bb44b6b780a557daea3656 (127.18.100.67). New cstate: current_term: 1 leader_uuid: "d870164263bb44b6b780a557daea3656" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } health_report { overall_health: HEALTHY } } }
I20260130 02:58:16.633324 18870 catalog_manager.cc:5746] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571) does not have the latest schema for tablet 345eaaa5625c48d099280b7fbb2e514b (table default.Table_c1886dd9a7934e7797e0e7e0b065d692 [id=7a2d5367224b492b844199088b01b149]). Expected version 1 got 0
I20260130 02:58:16.634325 19501 tablet_service.cc:1505] Processing CreateTablet for tablet 94355fc1338f479e8798070aff04fcba (DEFAULT_TABLE table=default.Table_12c2fe20cc3e45d29b7be37f900f68a9 [id=67ac4f4eda324e799b5867b2b011a16b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:16.634641 20718 consensus_queue.cc:1048] T 03025d13f0e545cfa6288677a2715343 P d870164263bb44b6b780a557daea3656 [LEADER]: Connected to new peer: Peer: permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260130 02:58:16.635381 19501 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 94355fc1338f479e8798070aff04fcba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:16.635788 20703 raft_consensus.cc:493] T 3aa76993beb94b12bc706a76db26a03b P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:16.636170 20703 raft_consensus.cc:515] T 3aa76993beb94b12bc706a76db26a03b P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:16.637849 20703 leader_election.cc:290] T 3aa76993beb94b12bc706a76db26a03b P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d870164263bb44b6b780a557daea3656 (127.18.100.67:39571), 371a36d219164974abdf848841528152 (127.18.100.65:42021)
W20260130 02:58:16.650257 18873 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.18.100.66:59691 (request call id 32) took 3244 ms (client timeout 1000 ms). Trace:
I20260130 02:58:16.664072 18868 catalog_manager.cc:5941] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4: Sending DeleteTablet for 3 replicas of tablet ab6afa40408a4f458495abf20290e55b
I20260130 02:58:16.665948 19501 tablet_service.cc:1552] Processing DeleteTablet for tablet ab6afa40408a4f458495abf20290e55b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:58:15 UTC) from {username='slave'} at 127.0.0.1:59068
W20260130 02:58:16.667634 18860 catalog_manager.cc:4965] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): delete failed for tablet ab6afa40408a4f458495abf20290e55b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ab6afa40408a4f458495abf20290e55b already in progress: creating tablet
I20260130 02:58:16.653926 18870 catalog_manager.cc:5654] T 7caf7cbad5054396b5617094e4d536a0 P d870164263bb44b6b780a557daea3656 reported cstate change: term changed from 0 to 1, leader changed from <none> to d870164263bb44b6b780a557daea3656 (127.18.100.67). New cstate: current_term: 1 leader_uuid: "d870164263bb44b6b780a557daea3656" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } health_report { overall_health: HEALTHY } } }
I20260130 02:58:16.673028 19360 tablet_service.cc:1505] Processing CreateTablet for tablet 39bcc41f10ae4241b370529364454586 (DEFAULT_TABLE table=default.Table_a132cf962cfc4ed8a6a8c177e088daeb [id=1c4937ca6e014fec8596a3dc25779600]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:16.674000 19360 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 39bcc41f10ae4241b370529364454586. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:16.674309 18868 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:50284 (request call id 238) took 3196 ms. Trace:
I20260130 02:58:16.674646 18868 rpcz_store.cc:276] 0130 02:58:13.477787 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:58:15.944033 (+2466246us) service_pool.cc:225] Handling call
0130 02:58:15.991485 (+ 47452us) catalog_manager.cc:2393] Looking up, locking, and authorizing table
0130 02:58:15.992046 (+ 561us) catalog_manager.cc:2815] Modifying in-memory table state
0130 02:58:15.992172 (+ 126us) catalog_manager.cc:2824] Locking tablets
0130 02:58:16.431943 (+439771us) catalog_manager.cc:2838] Removing table and tablets from system table
0130 02:58:16.663668 (+231725us) catalog_manager.cc:2856] Removing table from by-name map
0130 02:58:16.663871 (+ 203us) catalog_manager.cc:2872] Committing in-memory state
0130 02:58:16.663982 (+ 111us) catalog_manager.cc:2876] Aborting table tasks
0130 02:58:16.674250 (+ 10268us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0130 02:58:16.442363 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0130 02:58:16.442885 (+ 522us) write_op.cc:432] Acquiring schema lock in shared mode
0130 02:58:16.442913 (+ 28us) write_op.cc:435] Acquired schema lock
0130 02:58:16.442926 (+ 13us) tablet.cc:660] Decoding operations
0130 02:58:16.443324 (+ 398us) write_op.cc:620] Acquiring the partition lock for write op
0130 02:58:16.443372 (+ 48us) write_op.cc:641] Partition lock acquired for write op
0130 02:58:16.443398 (+ 26us) tablet.cc:683] Acquiring locks for 2 operations
0130 02:58:16.444023 (+ 625us) tablet.cc:699] Row locks acquired
0130 02:58:16.444043 (+ 20us) write_op.cc:260] PREPARE: finished
0130 02:58:16.444327 (+ 284us) write_op.cc:270] Start()
0130 02:58:16.444744 (+ 417us) write_op.cc:276] Timestamp: P: 1769741896444210 usec, L: 0
0130 02:58:16.444805 (+ 61us) op_driver.cc:348] REPLICATION: starting
0130 02:58:16.448232 (+ 3427us) log.cc:844] Serialized 731 byte log entry
0130 02:58:16.658709 (+210477us) op_driver.cc:464] REPLICATION: finished
0130 02:58:16.661705 (+ 2996us) write_op.cc:301] APPLY: starting
0130 02:58:16.661763 (+ 58us) tablet.cc:1365] starting BulkCheckPresence
0130 02:58:16.661859 (+ 96us) tablet.cc:1368] finished BulkCheckPresence
0130 02:58:16.661943 (+ 84us) tablet.cc:1370] starting ApplyRowOperation cycle
0130 02:58:16.662191 (+ 248us) tablet.cc:1381] finished ApplyRowOperation cycle
0130 02:58:16.662319 (+ 128us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0130 02:58:16.662367 (+ 48us) write_op.cc:312] APPLY: finished
0130 02:58:16.662543 (+ 176us) log.cc:844] Serialized 29 byte log entry
0130 02:58:16.662729 (+ 186us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:58:16.662788 (+ 59us) write_op.cc:454] Released schema lock
0130 02:58:16.662817 (+ 29us) write_op.cc:341] FINISH: Updating metrics
0130 02:58:16.663014 (+ 197us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:58:16.667283 (+ 4269us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":430,"apply.run_cpu_time_us":1286,"apply.run_wall_time_us":1286,"num_ops":2,"prepare.queue_time_us":8553,"prepare.run_cpu_time_us":3557,"prepare.run_wall_time_us":7797,"replication_time_us":187302,"spinlock_wait_cycles":12672}]]}
I20260130 02:58:16.681694 19356 tablet_service.cc:1552] Processing DeleteTablet for tablet ab6afa40408a4f458495abf20290e55b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:58:15 UTC) from {username='slave'} at 127.0.0.1:60110
I20260130 02:58:16.683323 19501 tablet_service.cc:1505] Processing CreateTablet for tablet 39bcc41f10ae4241b370529364454586 (DEFAULT_TABLE table=default.Table_a132cf962cfc4ed8a6a8c177e088daeb [id=1c4937ca6e014fec8596a3dc25779600]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:16.684401 19501 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 39bcc41f10ae4241b370529364454586. 1 dirs total, 0 dirs full, 0 dirs failed
W20260130 02:58:16.650653 18873 rpcz_store.cc:269] 0130 02:58:13.406267 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:58:13.470887 (+ 64620us) service_pool.cc:225] Handling call
0130 02:58:13.774400 (+303513us) spinlock_profiling.cc:243] Waited 258 ms on lock 0x7b5400030554. stack: 00007fb0bd3f7373 00007fb0bd3f7270 00007fb0bcf6c574 0000000000508744 00000000005086d9 00007fb0c44490c0 00007fb0c44e7bba 00007fb0c61e6eac 00007fb0c62e110e 00007fb0c3efe4a4 00007fb0c3efe432 00007fb0c3efe361 00007fb0c3efe2dc 00007fb0c3efd592 00007fb0c21dd34c 00007fb0c21dc786
0130 02:58:15.215093 (+1440693us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:58:15.215121 (+ 28us) write_op.cc:454] Released schema lock
0130 02:58:16.018837 (+803716us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:58:16.018876 (+ 39us) write_op.cc:454] Released schema lock
0130 02:58:16.650175 (+631299us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0130 02:58:15.191616 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0130 02:58:15.192079 (+ 463us) write_op.cc:432] Acquiring schema lock in shared mode
0130 02:58:15.192103 (+ 24us) write_op.cc:435] Acquired schema lock
0130 02:58:15.192117 (+ 14us) tablet.cc:660] Decoding operations
0130 02:58:15.192477 (+ 360us) write_op.cc:620] Acquiring the partition lock for write op
0130 02:58:15.192524 (+ 47us) write_op.cc:641] Partition lock acquired for write op
0130 02:58:15.192551 (+ 27us) tablet.cc:683] Acquiring locks for 2 operations
0130 02:58:15.192916 (+ 365us) tablet.cc:699] Row locks acquired
0130 02:58:15.192934 (+ 18us) write_op.cc:260] PREPARE: finished
0130 02:58:15.193062 (+ 128us) write_op.cc:270] Start()
0130 02:58:15.193154 (+ 92us) write_op.cc:276] Timestamp: P: 1769741895193019 usec, L: 0
0130 02:58:15.193172 (+ 18us) op_driver.cc:348] REPLICATION: starting
0130 02:58:15.193711 (+ 539us) log.cc:844] Serialized 913 byte log entry
0130 02:58:15.202103 (+ 8392us) op_driver.cc:464] REPLICATION: finished
0130 02:58:15.202528 (+ 425us) write_op.cc:301] APPLY: starting
0130 02:58:15.202614 (+ 86us) tablet.cc:1365] starting BulkCheckPresence
0130 02:58:15.202716 (+ 102us) tablet.cc:1368] finished BulkCheckPresence
0130 02:58:15.202808 (+ 92us) tablet.cc:1370] starting ApplyRowOperation cycle
0130 02:58:15.203217 (+ 409us) tablet.cc:1381] finished ApplyRowOperation cycle
0130 02:58:15.203362 (+ 145us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0130 02:58:15.203407 (+ 45us) write_op.cc:312] APPLY: finished
0130 02:58:15.203600 (+ 193us) log.cc:844] Serialized 29 byte log entry
0130 02:58:15.203734 (+ 134us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:58:15.203802 (+ 68us) write_op.cc:454] Released schema lock
0130 02:58:15.203834 (+ 32us) write_op.cc:341] FINISH: Updating metrics
Related trace 'op':
0130 02:58:15.965979 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0130 02:58:15.966483 (+ 504us) write_op.cc:432] Acquiring schema lock in shared mode
0130 02:58:15.966512 (+ 29us) write_op.cc:435] Acquired schema lock
0130 02:58:15.966528 (+ 16us) tablet.cc:660] Decoding operations
0130 02:58:15.966840 (+ 312us) write_op.cc:620] Acquiring the partition lock for write op
0130 02:58:15.966885 (+ 45us) write_op.cc:641] Partition lock acquired for write op
0130 02:58:15.966911 (+ 26us) tablet.cc:683] Acquiring locks for 1 operations
0130 02:58:15.967071 (+ 160us) tablet.cc:699] Row locks acquired
0130 02:58:15.967088 (+ 17us) write_op.cc:260] PREPARE: finished
0130 02:58:15.967210 (+ 122us) write_op.cc:270] Start()
0130 02:58:15.967300 (+ 90us) write_op.cc:276] Timestamp: P: 1769741895967139 usec, L: 0
0130 02:58:15.967327 (+ 27us) op_driver.cc:348] REPLICATION: starting
0130 02:58:15.967848 (+ 521us) log.cc:844] Serialized 356 byte log entry
0130 02:58:15.982274 (+ 14426us) spinlock_profiling.cc:243] Waited 11.5 ms on lock 0x7b580009fed0. stack: 00007fb0bd3f7373 00007fb0bd3f7270 00007fb0bcf6c574 0000000000508744 00000000005086d9 00007fb0c44490c0 00007fb0c443946c 00007fb0c443fd83 00007fb0c443fd09 00007fb0c443fc99 00007fb0c443fc61 00007fb0c443ee2d 00000000005aa2b4 00000000005aa0e9 00007fb0bd429355 00007fb0bd42b6a1
0130 02:58:15.982548 (+ 274us) op_driver.cc:464] REPLICATION: finished
0130 02:58:15.983539 (+ 991us) write_op.cc:301] APPLY: starting
0130 02:58:15.983595 (+ 56us) tablet.cc:1365] starting BulkCheckPresence
0130 02:58:15.983693 (+ 98us) tablet.cc:1368] finished BulkCheckPresence
0130 02:58:15.983713 (+ 20us) tablet.cc:1370] starting ApplyRowOperation cycle
0130 02:58:15.983890 (+ 177us) tablet.cc:1381] finished ApplyRowOperation cycle
0130 02:58:15.984021 (+ 131us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0130 02:58:15.984067 (+ 46us) write_op.cc:312] APPLY: finished
0130 02:58:15.984242 (+ 175us) log.cc:844] Serialized 23 byte log entry
0130 02:58:15.984391 (+ 149us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:58:15.984456 (+ 65us) write_op.cc:454] Released schema lock
0130 02:58:15.984489 (+ 33us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"reported_tablets":54,"spinlock_wait_cycles":592482560,"child_traces":[["op",{"apply.queue_time_us":281,"apply.run_cpu_time_us":1478,"apply.run_wall_time_us":1478,"num_ops":2,"prepare.queue_time_us":326,"prepare.run_cpu_time_us":2426,"prepare.run_wall_time_us":2426,"raft.queue_time_us":2453,"raft.run_cpu_time_us":610,"raft.run_wall_time_us":611,"replication_time_us":8843}],["op",{"apply.queue_time_us":815,"apply.run_cpu_time_us":1125,"apply.run_wall_time_us":1126,"num_ops":1,"prepare.queue_time_us":146,"prepare.run_cpu_time_us":2201,"prepare.run_wall_time_us":2201,"raft.queue_time_us":2817,"raft.run_cpu_time_us":2035,"raft.run_wall_time_us":13442,"replication_time_us":15137,"spinlock_wait_cycles":26438400}]]}
I20260130 02:58:16.669819 18870 catalog_manager.cc:5746] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571) does not have the latest schema for tablet 7caf7cbad5054396b5617094e4d536a0 (table default.Table_007e969f70d345a7bcb3dfbaabbc025d [id=77d4fa8705a3419cb0a84d3ca1bb4ce3]). Expected version 1 got 0
W20260130 02:58:16.692062 18860 catalog_manager.cc:4965] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767): delete failed for tablet ab6afa40408a4f458495abf20290e55b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ab6afa40408a4f458495abf20290e55b already in progress: creating tablet
I20260130 02:58:16.702641 19840 tablet_bootstrap.cc:492] T 320dbe98a32a43b9899bbeed80b20f55 P 78c2c46c7520492db35d51c244fb4f74: No bootstrap required, opened a new log
I20260130 02:58:16.702637 18870 catalog_manager.cc:5746] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571) does not have the latest schema for tablet 16691c9315e94253bb9b3fb5058e5636 (table default.Table_8f311c5878744ef692b496ecfdfc3ede [id=4f499357303d464aafc17679ffcaaa07]). Expected version 1 got 0
I20260130 02:58:16.705063 18870 catalog_manager.cc:5746] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571) does not have the latest schema for tablet 9fe0448bcc4a4346b3c67fc3f626ed3c (table default.Table_5c1c447e85524ac5a61ffd8a48509d36 [id=70aaf21ad9bd4d959f46939c6b4a4880]). Expected version 1 got 0
I20260130 02:58:16.711082 18870 catalog_manager.cc:5746] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571) does not have the latest schema for tablet 4c139aa8cd184f488cec4e05ef7dcada (table default.Table_e4d07634d84a4678b1c94374e3adc476 [id=fcbd7e3a7fc64d89aa740494d7e1a0f7]). Expected version 1 got 0
I20260130 02:58:16.710076 18868 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50284:
name: "default.Table_5d5e26f93a5a4fb084831add317d90f9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260130 02:58:16.716606 18868 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5d5e26f93a5a4fb084831add317d90f9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260130 02:58:16.709028 18873 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50284:
name: "default.Table_c238448062ad4cef8ddd0bd199d5e7f2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260130 02:58:16.718150 18873 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c238448062ad4cef8ddd0bd199d5e7f2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260130 02:58:16.719420 18870 catalog_manager.cc:5746] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571) does not have the latest schema for tablet 0e8018c094024c4788c484e8f5466e00 (table default.Table_f17d8e4d9f25449b84aa751af93936b2 [id=43254e466e7b4a2fb59fbcec68cfde36]). Expected version 1 got 0
I20260130 02:58:16.752389 18874 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:50284 (request call id 239) took 2545 ms. Trace:
I20260130 02:58:16.752785 18874 rpcz_store.cc:276] 0130 02:58:14.206680 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:58:15.961920 (+1755240us) service_pool.cc:225] Handling call
0130 02:58:16.468712 (+506792us) catalog_manager.cc:2101] Acquired catalog manager lock
0130 02:58:16.471682 (+ 2970us) catalog_manager.cc:2147] Created new table and tablet info
0130 02:58:16.726161 (+254479us) catalog_manager.cc:2207] Wrote table and tablets to system table
0130 02:58:16.752066 (+ 25905us) catalog_manager.cc:2241] Inserted table and tablets into CatalogManager maps
0130 02:58:16.752317 (+ 251us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0130 02:58:16.649348 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0130 02:58:16.661741 (+ 12393us) write_op.cc:432] Acquiring schema lock in shared mode
0130 02:58:16.661767 (+ 26us) write_op.cc:435] Acquired schema lock
0130 02:58:16.661785 (+ 18us) tablet.cc:660] Decoding operations
0130 02:58:16.662105 (+ 320us) write_op.cc:620] Acquiring the partition lock for write op
0130 02:58:16.662143 (+ 38us) write_op.cc:641] Partition lock acquired for write op
0130 02:58:16.662181 (+ 38us) tablet.cc:683] Acquiring locks for 2 operations
0130 02:58:16.662394 (+ 213us) tablet.cc:699] Row locks acquired
0130 02:58:16.662410 (+ 16us) write_op.cc:260] PREPARE: finished
0130 02:58:16.662523 (+ 113us) write_op.cc:270] Start()
0130 02:58:16.662699 (+ 176us) write_op.cc:276] Timestamp: P: 1769741896662429 usec, L: 0
0130 02:58:16.662719 (+ 20us) op_driver.cc:348] REPLICATION: starting
0130 02:58:16.706693 (+ 43974us) log.cc:844] Serialized 433 byte log entry
0130 02:58:16.707375 (+ 682us) spinlock_profiling.cc:243] Waited 43.4 ms on lock 0x7b5400030554. stack: 00007fb0bd3f7373 00007fb0bd3f7270 00007fb0bcf6c574 0000000000508744 00000000005086d9 00007fb0c44485a1 00007fb0c44daaa8 00007fb0c494c4d1 00007fb0c494bca8 00007fb0c4950941 00007fb0c49508f9 00007fb0c4950889 00007fb0c4950851 00007fb0c494fa7d 00000000005aa2b4 00000000005aa0e9
0130 02:58:16.723990 (+ 16615us) op_driver.cc:464] REPLICATION: finished
0130 02:58:16.724230 (+ 240us) write_op.cc:301] APPLY: starting
0130 02:58:16.724285 (+ 55us) tablet.cc:1365] starting BulkCheckPresence
0130 02:58:16.724380 (+ 95us) tablet.cc:1368] finished BulkCheckPresence
0130 02:58:16.724400 (+ 20us) tablet.cc:1370] starting ApplyRowOperation cycle
0130 02:58:16.724759 (+ 359us) tablet.cc:1381] finished ApplyRowOperation cycle
0130 02:58:16.724892 (+ 133us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0130 02:58:16.724932 (+ 40us) write_op.cc:312] APPLY: finished
0130 02:58:16.725114 (+ 182us) log.cc:844] Serialized 29 byte log entry
0130 02:58:16.725256 (+ 142us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:58:16.725321 (+ 65us) write_op.cc:454] Released schema lock
0130 02:58:16.725350 (+ 29us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":91,"apply.run_cpu_time_us":1296,"apply.run_wall_time_us":1296,"mutex_wait_us":4168,"num_ops":2,"prepare.queue_time_us":91867,"prepare.run_cpu_time_us":3443,"prepare.run_wall_time_us":96096,"raft.queue_time_us":6722,"raft.run_cpu_time_us":808,"raft.run_wall_time_us":1117,"replication_time_us":61160,"spinlock_wait_cycles":99719936}]]}
I20260130 02:58:16.757531 18874 master_service.cc:954] ReplaceTablet: received request to replace tablet ec28ff12927d450ead16d26cf016138d from {username='slave'} at 127.0.0.1:50284
I20260130 02:58:16.756757 18872 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:50284 (request call id 241) took 2368 ms. Trace:
I20260130 02:58:16.775264 18872 rpcz_store.cc:276] 0130 02:58:14.388525 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:58:16.142474 (+1753949us) service_pool.cc:225] Handling call
0130 02:58:16.614268 (+471794us) catalog_manager.cc:2101] Acquired catalog manager lock
0130 02:58:16.674697 (+ 60429us) catalog_manager.cc:2147] Created new table and tablet info
0130 02:58:16.728001 (+ 53304us) catalog_manager.cc:2207] Wrote table and tablets to system table
0130 02:58:16.756469 (+ 28468us) catalog_manager.cc:2241] Inserted table and tablets into CatalogManager maps
0130 02:58:16.756705 (+ 236us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0130 02:58:16.711885 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0130 02:58:16.712353 (+ 468us) write_op.cc:432] Acquiring schema lock in shared mode
0130 02:58:16.712378 (+ 25us) write_op.cc:435] Acquired schema lock
0130 02:58:16.712394 (+ 16us) tablet.cc:660] Decoding operations
0130 02:58:16.712690 (+ 296us) write_op.cc:620] Acquiring the partition lock for write op
0130 02:58:16.712728 (+ 38us) write_op.cc:641] Partition lock acquired for write op
0130 02:58:16.712753 (+ 25us) tablet.cc:683] Acquiring locks for 2 operations
0130 02:58:16.712977 (+ 224us) tablet.cc:699] Row locks acquired
0130 02:58:16.712995 (+ 18us) write_op.cc:260] PREPARE: finished
0130 02:58:16.713118 (+ 123us) write_op.cc:270] Start()
0130 02:58:16.713218 (+ 100us) write_op.cc:276] Timestamp: P: 1769741896713065 usec, L: 0
0130 02:58:16.713236 (+ 18us) op_driver.cc:348] REPLICATION: starting
0130 02:58:16.713731 (+ 495us) log.cc:844] Serialized 433 byte log entry
0130 02:58:16.726434 (+ 12703us) op_driver.cc:464] REPLICATION: finished
0130 02:58:16.726646 (+ 212us) write_op.cc:301] APPLY: starting
0130 02:58:16.726696 (+ 50us) tablet.cc:1365] starting BulkCheckPresence
0130 02:58:16.726785 (+ 89us) tablet.cc:1368] finished BulkCheckPresence
0130 02:58:16.726870 (+ 85us) tablet.cc:1370] starting ApplyRowOperation cycle
0130 02:58:16.727245 (+ 375us) tablet.cc:1381] finished ApplyRowOperation cycle
0130 02:58:16.727363 (+ 118us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0130 02:58:16.727400 (+ 37us) write_op.cc:312] APPLY: finished
0130 02:58:16.727549 (+ 149us) log.cc:844] Serialized 29 byte log entry
0130 02:58:16.727677 (+ 128us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:58:16.727739 (+ 62us) write_op.cc:454] Released schema lock
0130 02:58:16.727769 (+ 30us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":73,"apply.run_cpu_time_us":1275,"apply.run_wall_time_us":1275,"num_ops":2,"prepare.queue_time_us":36099,"prepare.run_cpu_time_us":2105,"prepare.run_wall_time_us":2105,"raft.queue_time_us":1505,"raft.run_cpu_time_us":10,"raft.run_wall_time_us":10,"replication_time_us":13115}]]}
I20260130 02:58:16.779929 18872 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50284:
table { table_name: "default.Table_09baf25d18fa489e9af0d7e289070082" } new_table_name: "default.Table_1ae68ae3f0404843aca3c72f02678ac6" modify_external_catalogs: true
I20260130 02:58:16.789583 18868 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:50284 (request call id 242) took 2095 ms. Trace:
I20260130 02:58:16.791980 18868 rpcz_store.cc:276] 0130 02:58:14.693701 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:58:16.682648 (+1988947us) service_pool.cc:225] Handling call
0130 02:58:16.709991 (+ 27343us) spinlock_profiling.cc:243] Waited 6.26 ms on lock 0x7b5400030554. stack: 00007fb0bd3f7373 00007fb0bd3f7270 00007fb0bcf6c574 0000000000508744 00000000005086d9 00007fb0c44490c0 00007fb0c44e7bba 00007fb0c61e6eac 00007fb0c62e2656 00007fb0c3f05524 00007fb0c3f054b2 00007fb0c3f053e1 00007fb0c3f0535c 00007fb0c3f04612 00007fb0c21dd34c 00007fb0c21dc786
0130 02:58:16.746054 (+ 36063us) catalog_manager.cc:2101] Acquired catalog manager lock
0130 02:58:16.747523 (+ 1469us) catalog_manager.cc:2147] Created new table and tablet info
0130 02:58:16.788995 (+ 41472us) catalog_manager.cc:2207] Wrote table and tablets to system table
0130 02:58:16.789274 (+ 279us) catalog_manager.cc:2241] Inserted table and tablets into CatalogManager maps
0130 02:58:16.789526 (+ 252us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0130 02:58:16.748791 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0130 02:58:16.749242 (+ 451us) write_op.cc:432] Acquiring schema lock in shared mode
0130 02:58:16.749269 (+ 27us) write_op.cc:435] Acquired schema lock
0130 02:58:16.749280 (+ 11us) tablet.cc:660] Decoding operations
0130 02:58:16.749582 (+ 302us) write_op.cc:620] Acquiring the partition lock for write op
0130 02:58:16.749624 (+ 42us) write_op.cc:641] Partition lock acquired for write op
0130 02:58:16.749647 (+ 23us) tablet.cc:683] Acquiring locks for 2 operations
0130 02:58:16.749905 (+ 258us) tablet.cc:699] Row locks acquired
0130 02:58:16.749922 (+ 17us) write_op.cc:260] PREPARE: finished
0130 02:58:16.750033 (+ 111us) write_op.cc:270] Start()
0130 02:58:16.750129 (+ 96us) write_op.cc:276] Timestamp: P: 1769741896749987 usec, L: 0
0130 02:58:16.750147 (+ 18us) op_driver.cc:348] REPLICATION: starting
0130 02:58:16.750658 (+ 511us) log.cc:844] Serialized 433 byte log entry
0130 02:58:16.784991 (+ 34333us) op_driver.cc:464] REPLICATION: finished
0130 02:58:16.785285 (+ 294us) write_op.cc:301] APPLY: starting
0130 02:58:16.785340 (+ 55us) tablet.cc:1365] starting BulkCheckPresence
0130 02:58:16.785437 (+ 97us) tablet.cc:1368] finished BulkCheckPresence
0130 02:58:16.785521 (+ 84us) tablet.cc:1370] starting ApplyRowOperation cycle
0130 02:58:16.785886 (+ 365us) tablet.cc:1381] finished ApplyRowOperation cycle
0130 02:58:16.786013 (+ 127us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0130 02:58:16.786052 (+ 39us) write_op.cc:312] APPLY: finished
0130 02:58:16.786255 (+ 203us) log.cc:844] Serialized 29 byte log entry
0130 02:58:16.786396 (+ 141us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:58:16.786456 (+ 60us) write_op.cc:454] Released schema lock
0130 02:58:16.786489 (+ 33us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"spinlock_wait_cycles":14397824,"child_traces":[["op",{"apply.queue_time_us":73,"apply.run_cpu_time_us":1394,"apply.run_wall_time_us":1394,"num_ops":2,"prepare.queue_time_us":95,"prepare.run_cpu_time_us":2051,"prepare.run_wall_time_us":2939,"replication_time_us":34750}]]}
I20260130 02:58:16.807587 18868 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50284:
table { table_name: "default.Table_3413dae2efac41488448e6e5975cf710" } modify_external_catalogs: true
I20260130 02:58:16.808306 18868 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50284:
table { table_name: "default.Table_3413dae2efac41488448e6e5975cf710" } modify_external_catalogs: true
I20260130 02:58:16.816269 19625 tablet_bootstrap.cc:492] T 6218f2f7d73a42428b57a45858f53afd P d870164263bb44b6b780a557daea3656: No bootstrap required, opened a new log
I20260130 02:58:16.816730 19625 ts_tablet_manager.cc:1403] T 6218f2f7d73a42428b57a45858f53afd P d870164263bb44b6b780a557daea3656: Time spent bootstrapping tablet: real 0.650s user 0.111s sys 0.106s
I20260130 02:58:16.821225 19625 raft_consensus.cc:359] T 6218f2f7d73a42428b57a45858f53afd P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:16.822302 19625 raft_consensus.cc:385] T 6218f2f7d73a42428b57a45858f53afd P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:16.822616 19625 raft_consensus.cc:740] T 6218f2f7d73a42428b57a45858f53afd P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d870164263bb44b6b780a557daea3656, State: Initialized, Role: FOLLOWER
W20260130 02:58:16.830013 19313 leader_election.cc:343] T 13f0ace4d9de43abab08eca32cdb30c1 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:16.830468 19313 leader_election.cc:304] T 13f0ace4d9de43abab08eca32cdb30c1 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78c2c46c7520492db35d51c244fb4f74; no voters: 371a36d219164974abdf848841528152, d870164263bb44b6b780a557daea3656
I20260130 02:58:16.639189 19222 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3aa76993beb94b12bc706a76db26a03b" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152" is_pre_election: true
I20260130 02:58:16.671895 19204 tablet_service.cc:1552] Processing DeleteTablet for tablet ab6afa40408a4f458495abf20290e55b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:58:15 UTC) from {username='slave'} at 127.0.0.1:42870
I20260130 02:58:16.673043 19202 tablet_service.cc:1505] Processing CreateTablet for tablet 39bcc41f10ae4241b370529364454586 (DEFAULT_TABLE table=default.Table_a132cf962cfc4ed8a6a8c177e088daeb [id=1c4937ca6e014fec8596a3dc25779600]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:16.832937 19840 ts_tablet_manager.cc:1403] T 320dbe98a32a43b9899bbeed80b20f55 P 78c2c46c7520492db35d51c244fb4f74: Time spent bootstrapping tablet: real 0.301s user 0.008s sys 0.005s
I20260130 02:58:16.833271 19202 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 39bcc41f10ae4241b370529364454586. 1 dirs total, 0 dirs full, 0 dirs failed
W20260130 02:58:16.835790 19314 leader_election.cc:343] T 3aa76993beb94b12bc706a76db26a03b P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Illegal state: must be running to vote when last-logged opid is not known
W20260130 02:58:16.836692 18861 catalog_manager.cc:4965] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021): delete failed for tablet ab6afa40408a4f458495abf20290e55b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ab6afa40408a4f458495abf20290e55b already in progress: creating tablet
I20260130 02:58:16.836884 20736 raft_consensus.cc:493] T 6218f2f7d73a42428b57a45858f53afd P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:16.837409 20736 raft_consensus.cc:515] T 6218f2f7d73a42428b57a45858f53afd P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:16.839521 20736 leader_election.cc:290] T 6218f2f7d73a42428b57a45858f53afd P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d870164263bb44b6b780a557daea3656 (127.18.100.67:39571), 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767)
I20260130 02:58:16.843492 19625 consensus_queue.cc:260] T 6218f2f7d73a42428b57a45858f53afd P d870164263bb44b6b780a557daea3656 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:16.701717 19807 tablet_bootstrap.cc:492] T f9e5f01e69254d658c369f3e21cef523 P 371a36d219164974abdf848841528152: No bootstrap required, opened a new log
W20260130 02:58:16.605688 19159 leader_election.cc:343] T dcba6b534f9d406dad4fb502be40921c P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): Illegal state: must be running to vote when last-logged opid is not known
W20260130 02:58:16.853765 19159 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.100.67:39571, user_credentials={real_user=slave}} blocked reactor thread for 248086us
I20260130 02:58:16.854341 19159 leader_election.cc:304] T dcba6b534f9d406dad4fb502be40921c P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 371a36d219164974abdf848841528152, 78c2c46c7520492db35d51c244fb4f74; no voters: d870164263bb44b6b780a557daea3656
I20260130 02:58:16.867341 19625 ts_tablet_manager.cc:1434] T 6218f2f7d73a42428b57a45858f53afd P d870164263bb44b6b780a557daea3656: Time spent starting tablet: real 0.050s user 0.006s sys 0.001s
I20260130 02:58:16.869926 19625 tablet_bootstrap.cc:492] T f9e5f01e69254d658c369f3e21cef523 P d870164263bb44b6b780a557daea3656: Bootstrap starting.
I20260130 02:58:16.872730 19380 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6218f2f7d73a42428b57a45858f53afd" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74" is_pre_election: true
I20260130 02:58:16.873261 19380 raft_consensus.cc:2468] T 6218f2f7d73a42428b57a45858f53afd P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 371a36d219164974abdf848841528152 in term 0.
I20260130 02:58:16.874540 20608 raft_consensus.cc:2804] T dcba6b534f9d406dad4fb502be40921c P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Leader election won for term 1
W20260130 02:58:16.874598 18860 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet b1e8b78eff0d4a0b92059adb2de4a3ce (table default.Table_d0b4516d24384e5386c1debaf2543adb [id=f1342f9497b74e7a9ecad1de8082578d], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet b1e8b78eff0d4a0b92059adb2de4a3ce consensus state has no leader
I20260130 02:58:16.874964 20608 raft_consensus.cc:697] T dcba6b534f9d406dad4fb502be40921c P 371a36d219164974abdf848841528152 [term 1 LEADER]: Becoming Leader. State: Replica: 371a36d219164974abdf848841528152, State: Running, Role: LEADER
I20260130 02:58:16.875667 20608 consensus_queue.cc:237] T dcba6b534f9d406dad4fb502be40921c P 371a36d219164974abdf848841528152 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:16.884495 19523 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6218f2f7d73a42428b57a45858f53afd" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
I20260130 02:58:16.885052 19523 raft_consensus.cc:2468] T 6218f2f7d73a42428b57a45858f53afd P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 371a36d219164974abdf848841528152 in term 0.
I20260130 02:58:16.886042 20697 raft_consensus.cc:2749] T 13f0ace4d9de43abab08eca32cdb30c1 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260130 02:58:16.887023 19313 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.100.67:39571, user_credentials={real_user=slave}} blocked reactor thread for 57052.2us
I20260130 02:58:16.895071 19840 raft_consensus.cc:359] T 320dbe98a32a43b9899bbeed80b20f55 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:16.898326 19840 raft_consensus.cc:385] T 320dbe98a32a43b9899bbeed80b20f55 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:16.905447 20718 consensus_queue.cc:1048] T 78e9bc3e55fe4e4a98e8eb01da59d365 P d870164263bb44b6b780a557daea3656 [LEADER]: Connected to new peer: Peer: permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260130 02:58:16.908751 20497 consensus_queue.cc:1048] T 274530359b174ff79f046c2d4a6afa76 P 371a36d219164974abdf848841528152 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260130 02:58:16.910689 19840 raft_consensus.cc:740] T 320dbe98a32a43b9899bbeed80b20f55 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78c2c46c7520492db35d51c244fb4f74, State: Initialized, Role: FOLLOWER
I20260130 02:58:16.911341 19840 consensus_queue.cc:260] T 320dbe98a32a43b9899bbeed80b20f55 P 78c2c46c7520492db35d51c244fb4f74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:16.913931 19840 ts_tablet_manager.cc:1434] T 320dbe98a32a43b9899bbeed80b20f55 P 78c2c46c7520492db35d51c244fb4f74: Time spent starting tablet: real 0.081s user 0.001s sys 0.005s
I20260130 02:58:16.914834 19840 tablet_bootstrap.cc:492] T 2c79a3aecab844698e70834999e1caac P 78c2c46c7520492db35d51c244fb4f74: Bootstrap starting.
I20260130 02:58:16.917755 20763 tablet.cc:1722] T 4c139aa8cd184f488cec4e05ef7dcada P 371a36d219164974abdf848841528152: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260130 02:58:16.851446 19807 ts_tablet_manager.cc:1403] T f9e5f01e69254d658c369f3e21cef523 P 371a36d219164974abdf848841528152: Time spent bootstrapping tablet: real 0.540s user 0.009s sys 0.002s
I20260130 02:58:16.921269 19807 raft_consensus.cc:359] T f9e5f01e69254d658c369f3e21cef523 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:16.925089 18873 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:50284 (request call id 243) took 2231 ms. Trace:
W20260130 02:58:16.927386 19313 leader_election.cc:343] T 12c3f3ec523040e7b7cd356e3da57e0d P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:16.927769 19313 leader_election.cc:304] T 12c3f3ec523040e7b7cd356e3da57e0d P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78c2c46c7520492db35d51c244fb4f74; no voters: 371a36d219164974abdf848841528152, d870164263bb44b6b780a557daea3656
I20260130 02:58:16.925474 18873 rpcz_store.cc:276] 0130 02:58:14.693935 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:58:16.688420 (+1994485us) service_pool.cc:225] Handling call
0130 02:58:16.708948 (+ 20528us) spinlock_profiling.cc:243] Waited 2.78 ms on lock 0x7b5400030554. stack: 00007fb0bd3f7373 00007fb0bd3f7270 00007fb0bcf6c574 0000000000508744 00000000005086d9 00007fb0c44490c0 00007fb0c44e7bba 00007fb0c61e6eac 00007fb0c62e2656 00007fb0c3f05524 00007fb0c3f054b2 00007fb0c3f053e1 00007fb0c3f0535c 00007fb0c3f04612 00007fb0c21dd34c 00007fb0c21dc786
0130 02:58:16.752970 (+ 44022us) catalog_manager.cc:2101] Acquired catalog manager lock
0130 02:58:16.756209 (+ 3239us) catalog_manager.cc:2147] Created new table and tablet info
0130 02:58:16.880171 (+123962us) catalog_manager.cc:2207] Wrote table and tablets to system table
0130 02:58:16.924762 (+ 44591us) catalog_manager.cc:2241] Inserted table and tablets into CatalogManager maps
0130 02:58:16.925025 (+ 263us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0130 02:58:16.762931 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0130 02:58:16.763454 (+ 523us) write_op.cc:432] Acquiring schema lock in shared mode
0130 02:58:16.763478 (+ 24us) write_op.cc:435] Acquired schema lock
0130 02:58:16.763492 (+ 14us) tablet.cc:660] Decoding operations
0130 02:58:16.763781 (+ 289us) write_op.cc:620] Acquiring the partition lock for write op
0130 02:58:16.763816 (+ 35us) write_op.cc:641] Partition lock acquired for write op
0130 02:58:16.763843 (+ 27us) tablet.cc:683] Acquiring locks for 2 operations
0130 02:58:16.764049 (+ 206us) tablet.cc:699] Row locks acquired
0130 02:58:16.764065 (+ 16us) write_op.cc:260] PREPARE: finished
0130 02:58:16.764185 (+ 120us) write_op.cc:270] Start()
0130 02:58:16.764318 (+ 133us) write_op.cc:276] Timestamp: P: 1769741896764136 usec, L: 0
0130 02:58:16.764337 (+ 19us) op_driver.cc:348] REPLICATION: starting
0130 02:58:16.764819 (+ 482us) log.cc:844] Serialized 433 byte log entry
0130 02:58:16.875683 (+110864us) op_driver.cc:464] REPLICATION: finished
0130 02:58:16.876305 (+ 622us) write_op.cc:301] APPLY: starting
0130 02:58:16.876361 (+ 56us) tablet.cc:1365] starting BulkCheckPresence
0130 02:58:16.876462 (+ 101us) tablet.cc:1368] finished BulkCheckPresence
0130 02:58:16.876555 (+ 93us) tablet.cc:1370] starting ApplyRowOperation cycle
0130 02:58:16.876991 (+ 436us) tablet.cc:1381] finished ApplyRowOperation cycle
0130 02:58:16.877137 (+ 146us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0130 02:58:16.877177 (+ 40us) write_op.cc:312] APPLY: finished
0130 02:58:16.877359 (+ 182us) log.cc:844] Serialized 29 byte log entry
0130 02:58:16.877505 (+ 146us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:58:16.877570 (+ 65us) write_op.cc:454] Released schema lock
0130 02:58:16.877599 (+ 29us) write_op.cc:341] FINISH: Updating metrics
0130 02:58:16.877809 (+ 210us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:58:16.877833 (+ 24us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":6397696,"child_traces":[["op",{"apply.queue_time_us":477,"apply.run_cpu_time_us":1479,"apply.run_wall_time_us":1480,"num_ops":2,"prepare.queue_time_us":5638,"prepare.run_cpu_time_us":2102,"prepare.run_wall_time_us":3015,"replication_time_us":111248}]]}
I20260130 02:58:16.928399 20703 raft_consensus.cc:2749] T 12c3f3ec523040e7b7cd356e3da57e0d P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260130 02:58:16.900063 20760 tablet.cc:1722] T 4c139aa8cd184f488cec4e05ef7dcada P 78c2c46c7520492db35d51c244fb4f74: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260130 02:58:16.943017 20736 consensus_queue.cc:1048] T a90712651a604b83ba187f893ab17711 P 371a36d219164974abdf848841528152 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260130 02:58:16.951159 19807 raft_consensus.cc:385] T f9e5f01e69254d658c369f3e21cef523 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:16.951503 19807 raft_consensus.cc:740] T f9e5f01e69254d658c369f3e21cef523 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 371a36d219164974abdf848841528152, State: Initialized, Role: FOLLOWER
I20260130 02:58:16.952040 19807 consensus_queue.cc:260] T f9e5f01e69254d658c369f3e21cef523 P 371a36d219164974abdf848841528152 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:16.955299 19807 ts_tablet_manager.cc:1434] T f9e5f01e69254d658c369f3e21cef523 P 371a36d219164974abdf848841528152: Time spent starting tablet: real 0.036s user 0.004s sys 0.001s
I20260130 02:58:16.956099 19807 tablet_bootstrap.cc:492] T 838e49e5e8874db393bdc15a1c92f87e P 371a36d219164974abdf848841528152: Bootstrap starting.
I20260130 02:58:16.970913 19159 leader_election.cc:304] T 6218f2f7d73a42428b57a45858f53afd P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 371a36d219164974abdf848841528152, d870164263bb44b6b780a557daea3656; no voters:
I20260130 02:58:16.971963 20497 raft_consensus.cc:2804] T 6218f2f7d73a42428b57a45858f53afd P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260130 02:58:16.972311 20497 raft_consensus.cc:493] T 6218f2f7d73a42428b57a45858f53afd P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260130 02:58:16.972630 20497 raft_consensus.cc:3060] T 6218f2f7d73a42428b57a45858f53afd P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:16.980247 19807 tablet_bootstrap.cc:654] T 838e49e5e8874db393bdc15a1c92f87e P 371a36d219164974abdf848841528152: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:16.982191 20497 raft_consensus.cc:515] T 6218f2f7d73a42428b57a45858f53afd P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:16.998162 20497 leader_election.cc:290] T 6218f2f7d73a42428b57a45858f53afd P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 election: Requested vote from peers d870164263bb44b6b780a557daea3656 (127.18.100.67:39571), 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767)
I20260130 02:58:17.010766 20679 consensus_queue.cc:1048] T 6e12a8eeb3204325a9a8b7779f72ab29 P d870164263bb44b6b780a557daea3656 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260130 02:58:17.011933 20679 consensus_queue.cc:1048] T f41f4cff7dd4489f948700a45e9a7057 P d870164263bb44b6b780a557daea3656 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260130 02:58:17.015116 19625 tablet_bootstrap.cc:654] T f9e5f01e69254d658c369f3e21cef523 P d870164263bb44b6b780a557daea3656: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:17.030923 19523 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3aa76993beb94b12bc706a76db26a03b" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
I20260130 02:58:17.057672 19840 tablet_bootstrap.cc:654] T 2c79a3aecab844698e70834999e1caac P 78c2c46c7520492db35d51c244fb4f74: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:17.065649 18874 catalog_manager.cc:5941] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4: Sending DeleteTablet for 3 replicas of tablet ec28ff12927d450ead16d26cf016138d
I20260130 02:58:17.067512 18874 catalog_manager.cc:6724] ReplaceTablet: tablet ec28ff12927d450ead16d26cf016138d deleted and replaced by tablet 6d1f121f49cd47c2b8247daaf6874e01
I20260130 02:58:17.067942 18874 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:50284 (request call id 244) took 2314 ms. Trace:
I20260130 02:58:17.071792 20718 consensus_queue.cc:1048] T 03025d13f0e545cfa6288677a2715343 P d870164263bb44b6b780a557daea3656 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260130 02:58:17.074199 19313 leader_election.cc:343] T 3aa76993beb94b12bc706a76db26a03b P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:17.074620 19313 leader_election.cc:304] T 3aa76993beb94b12bc706a76db26a03b P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78c2c46c7520492db35d51c244fb4f74; no voters: 371a36d219164974abdf848841528152, d870164263bb44b6b780a557daea3656
I20260130 02:58:17.075197 20679 consensus_queue.cc:1048] T 78e9bc3e55fe4e4a98e8eb01da59d365 P d870164263bb44b6b780a557daea3656 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260130 02:58:17.075655 20697 raft_consensus.cc:2749] T 3aa76993beb94b12bc706a76db26a03b P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260130 02:58:17.068373 18874 rpcz_store.cc:276] 0130 02:58:14.753291 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:58:16.757418 (+2004127us) service_pool.cc:225] Handling call
0130 02:58:17.067891 (+310473us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0130 02:58:16.971843 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0130 02:58:16.972311 (+ 468us) write_op.cc:432] Acquiring schema lock in shared mode
0130 02:58:16.972336 (+ 25us) write_op.cc:435] Acquired schema lock
0130 02:58:16.972351 (+ 15us) tablet.cc:660] Decoding operations
0130 02:58:16.972747 (+ 396us) write_op.cc:620] Acquiring the partition lock for write op
0130 02:58:16.972790 (+ 43us) write_op.cc:641] Partition lock acquired for write op
0130 02:58:16.972815 (+ 25us) tablet.cc:683] Acquiring locks for 2 operations
0130 02:58:16.973049 (+ 234us) tablet.cc:699] Row locks acquired
0130 02:58:16.973062 (+ 13us) write_op.cc:260] PREPARE: finished
0130 02:58:16.973187 (+ 125us) write_op.cc:270] Start()
0130 02:58:16.973266 (+ 79us) write_op.cc:276] Timestamp: P: 1769741896973115 usec, L: 0
0130 02:58:16.973280 (+ 14us) op_driver.cc:348] REPLICATION: starting
0130 02:58:16.973779 (+ 499us) log.cc:844] Serialized 662 byte log entry
0130 02:58:17.010303 (+ 36524us) op_driver.cc:464] REPLICATION: finished
0130 02:58:17.010727 (+ 424us) write_op.cc:301] APPLY: starting
0130 02:58:17.010781 (+ 54us) tablet.cc:1365] starting BulkCheckPresence
0130 02:58:17.010875 (+ 94us) tablet.cc:1368] finished BulkCheckPresence
0130 02:58:17.010977 (+ 102us) tablet.cc:1370] starting ApplyRowOperation cycle
0130 02:58:17.011334 (+ 357us) tablet.cc:1381] finished ApplyRowOperation cycle
0130 02:58:17.011530 (+ 196us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0130 02:58:17.011571 (+ 41us) write_op.cc:312] APPLY: finished
0130 02:58:17.011746 (+ 175us) log.cc:844] Serialized 29 byte log entry
0130 02:58:17.011890 (+ 144us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:58:17.011955 (+ 65us) write_op.cc:454] Released schema lock
0130 02:58:17.011986 (+ 31us) write_op.cc:341] FINISH: Updating metrics
0130 02:58:17.012211 (+ 225us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:58:17.012234 (+ 23us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":279,"apply.run_cpu_time_us":1460,"apply.run_wall_time_us":1460,"num_ops":2,"prepare.queue_time_us":47303,"prepare.run_cpu_time_us":2194,"prepare.run_wall_time_us":2193,"raft.queue_time_us":1119,"raft.run_cpu_time_us":12,"raft.run_wall_time_us":12,"replication_time_us":36932}]]}
I20260130 02:58:17.081082 18874 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50284:
table { table_name: "default.Table_1cbb7ef2df504c1083290b333440c9f7" } new_table_name: "default.Table_805e07e3546e447d8c8b8029477013cc" modify_external_catalogs: true
I20260130 02:58:17.083262 19496 tablet_service.cc:1552] Processing DeleteTablet for tablet ec28ff12927d450ead16d26cf016138d with delete_type TABLET_DATA_DELETED (replaced by tablet 6d1f121f49cd47c2b8247daaf6874e01) from {username='slave'} at 127.0.0.1:59068
I20260130 02:58:17.083990 20571 tablet_replica.cc:333] T ec28ff12927d450ead16d26cf016138d P d870164263bb44b6b780a557daea3656: stopping tablet replica
I20260130 02:58:17.084666 20571 raft_consensus.cc:2243] T ec28ff12927d450ead16d26cf016138d P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260130 02:58:17.085131 20571 raft_consensus.cc:2272] T ec28ff12927d450ead16d26cf016138d P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260130 02:58:17.088025 20571 ts_tablet_manager.cc:1916] T ec28ff12927d450ead16d26cf016138d P d870164263bb44b6b780a557daea3656: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260130 02:58:17.101244 19360 tablet_service.cc:1552] Processing DeleteTablet for tablet ec28ff12927d450ead16d26cf016138d with delete_type TABLET_DATA_DELETED (replaced by tablet 6d1f121f49cd47c2b8247daaf6874e01) from {username='slave'} at 127.0.0.1:60110
I20260130 02:58:17.110360 19840 tablet_bootstrap.cc:492] T 2c79a3aecab844698e70834999e1caac P 78c2c46c7520492db35d51c244fb4f74: No bootstrap required, opened a new log
I20260130 02:58:17.110814 19840 ts_tablet_manager.cc:1403] T 2c79a3aecab844698e70834999e1caac P 78c2c46c7520492db35d51c244fb4f74: Time spent bootstrapping tablet: real 0.196s user 0.008s sys 0.002s
I20260130 02:58:17.113085 19840 raft_consensus.cc:359] T 2c79a3aecab844698e70834999e1caac P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:17.113724 19840 raft_consensus.cc:385] T 2c79a3aecab844698e70834999e1caac P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:17.113970 19840 raft_consensus.cc:740] T 2c79a3aecab844698e70834999e1caac P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78c2c46c7520492db35d51c244fb4f74, State: Initialized, Role: FOLLOWER
I20260130 02:58:17.117296 20570 tablet_replica.cc:333] T ec28ff12927d450ead16d26cf016138d P 78c2c46c7520492db35d51c244fb4f74: stopping tablet replica
I20260130 02:58:17.118021 20570 raft_consensus.cc:2243] T ec28ff12927d450ead16d26cf016138d P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260130 02:58:17.118486 20570 raft_consensus.cc:2272] T ec28ff12927d450ead16d26cf016138d P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260130 02:58:17.120349 20570 ts_tablet_manager.cc:1916] T ec28ff12927d450ead16d26cf016138d P 78c2c46c7520492db35d51c244fb4f74: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260130 02:58:17.128571 19625 tablet_bootstrap.cc:492] T f9e5f01e69254d658c369f3e21cef523 P d870164263bb44b6b780a557daea3656: No bootstrap required, opened a new log
I20260130 02:58:17.128989 19625 ts_tablet_manager.cc:1403] T f9e5f01e69254d658c369f3e21cef523 P d870164263bb44b6b780a557daea3656: Time spent bootstrapping tablet: real 0.259s user 0.012s sys 0.023s
W20260130 02:58:17.130429 19313 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.100.67:39571, user_credentials={real_user=slave}} blocked reactor thread for 54570.2us
I20260130 02:58:17.131227 19625 raft_consensus.cc:359] T f9e5f01e69254d658c369f3e21cef523 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:17.131832 19625 raft_consensus.cc:385] T f9e5f01e69254d658c369f3e21cef523 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:17.132059 19625 raft_consensus.cc:740] T f9e5f01e69254d658c369f3e21cef523 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d870164263bb44b6b780a557daea3656, State: Initialized, Role: FOLLOWER
I20260130 02:58:17.132692 19625 consensus_queue.cc:260] T f9e5f01e69254d658c369f3e21cef523 P d870164263bb44b6b780a557daea3656 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:17.134317 20570 ts_tablet_manager.cc:1929] T ec28ff12927d450ead16d26cf016138d P 78c2c46c7520492db35d51c244fb4f74: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260130 02:58:17.134783 20570 log.cc:1199] T ec28ff12927d450ead16d26cf016138d P 78c2c46c7520492db35d51c244fb4f74: Deleting WAL directory at /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-1/wal/wals/ec28ff12927d450ead16d26cf016138d
I20260130 02:58:17.139170 20570 ts_tablet_manager.cc:1950] T ec28ff12927d450ead16d26cf016138d P 78c2c46c7520492db35d51c244fb4f74: Deleting consensus metadata
I20260130 02:58:17.151989 18860 catalog_manager.cc:4985] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767): tablet ec28ff12927d450ead16d26cf016138d (table default.Table_0480357296ac4a27812d802ee4566351 [id=0da6197ab32e4645aa8d8787662ee5e6]) successfully deleted
I20260130 02:58:17.159122 19840 consensus_queue.cc:260] T 2c79a3aecab844698e70834999e1caac P 78c2c46c7520492db35d51c244fb4f74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:17.164954 19625 ts_tablet_manager.cc:1434] T f9e5f01e69254d658c369f3e21cef523 P d870164263bb44b6b780a557daea3656: Time spent starting tablet: real 0.036s user 0.005s sys 0.001s
I20260130 02:58:17.165903 19625 tablet_bootstrap.cc:492] T dcba6b534f9d406dad4fb502be40921c P d870164263bb44b6b780a557daea3656: Bootstrap starting.
I20260130 02:58:17.167999 19840 ts_tablet_manager.cc:1434] T 2c79a3aecab844698e70834999e1caac P 78c2c46c7520492db35d51c244fb4f74: Time spent starting tablet: real 0.057s user 0.003s sys 0.002s
I20260130 02:58:17.168946 18867 catalog_manager.cc:5746] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767) does not have the latest schema for tablet a8a5038e9def44eb8b3b1e4aeae25652 (table default.Table_dee034aec0b24982b8dc2a3fa1b326f1 [id=66033e75b3814a8584311cf7e4a07ad3]). Expected version 1 got 0
I20260130 02:58:17.170887 18867 catalog_manager.cc:5746] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767) does not have the latest schema for tablet b1e8b78eff0d4a0b92059adb2de4a3ce (table default.Table_d0b4516d24384e5386c1debaf2543adb [id=f1342f9497b74e7a9ecad1de8082578d]). Expected version 1 got 0
I20260130 02:58:17.172350 19840 tablet_bootstrap.cc:492] T f7607df5efd543438fb2dff3667cc8c8 P 78c2c46c7520492db35d51c244fb4f74: Bootstrap starting.
W20260130 02:58:17.180322 19462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.100.66:40767, user_credentials={real_user=slave}} blocked reactor thread for 45489.3us
I20260130 02:58:17.198251 18867 catalog_manager.cc:5746] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767) does not have the latest schema for tablet 274530359b174ff79f046c2d4a6afa76 (table default.Table_c4f4904b6fac4b9880979cf0314e71c1 [id=c456c6418c93499a9d08f652720854ca]). Expected version 1 got 0
I20260130 02:58:17.199337 19840 tablet_bootstrap.cc:654] T f7607df5efd543438fb2dff3667cc8c8 P 78c2c46c7520492db35d51c244fb4f74: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:17.200402 20796 tablet.cc:1722] T 4c139aa8cd184f488cec4e05ef7dcada P d870164263bb44b6b780a557daea3656: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260130 02:58:17.211046 18867 catalog_manager.cc:5746] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767) does not have the latest schema for tablet 345eaaa5625c48d099280b7fbb2e514b (table default.Table_c1886dd9a7934e7797e0e7e0b065d692 [id=7a2d5367224b492b844199088b01b149]). Expected version 1 got 0
I20260130 02:58:17.212832 18867 catalog_manager.cc:5746] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767) does not have the latest schema for tablet 16691c9315e94253bb9b3fb5058e5636 (table default.Table_8f311c5878744ef692b496ecfdfc3ede [id=4f499357303d464aafc17679ffcaaa07]). Expected version 1 got 0
I20260130 02:58:17.214658 18867 catalog_manager.cc:5746] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767) does not have the latest schema for tablet 9fe0448bcc4a4346b3c67fc3f626ed3c (table default.Table_5c1c447e85524ac5a61ffd8a48509d36 [id=70aaf21ad9bd4d959f46939c6b4a4880]). Expected version 1 got 0
I20260130 02:58:17.230167 19840 tablet_bootstrap.cc:492] T f7607df5efd543438fb2dff3667cc8c8 P 78c2c46c7520492db35d51c244fb4f74: No bootstrap required, opened a new log
I20260130 02:58:17.230638 19840 ts_tablet_manager.cc:1403] T f7607df5efd543438fb2dff3667cc8c8 P 78c2c46c7520492db35d51c244fb4f74: Time spent bootstrapping tablet: real 0.058s user 0.006s sys 0.004s
I20260130 02:58:17.233011 19840 raft_consensus.cc:359] T f7607df5efd543438fb2dff3667cc8c8 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:17.233652 19840 raft_consensus.cc:385] T f7607df5efd543438fb2dff3667cc8c8 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:17.233907 19840 raft_consensus.cc:740] T f7607df5efd543438fb2dff3667cc8c8 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78c2c46c7520492db35d51c244fb4f74, State: Initialized, Role: FOLLOWER
I20260130 02:58:17.250010 20571 ts_tablet_manager.cc:1929] T ec28ff12927d450ead16d26cf016138d P d870164263bb44b6b780a557daea3656: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260130 02:58:17.255929 20571 log.cc:1199] T ec28ff12927d450ead16d26cf016138d P d870164263bb44b6b780a557daea3656: Deleting WAL directory at /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-2/wal/wals/ec28ff12927d450ead16d26cf016138d
I20260130 02:58:17.263505 20571 ts_tablet_manager.cc:1950] T ec28ff12927d450ead16d26cf016138d P d870164263bb44b6b780a557daea3656: Deleting consensus metadata
I20260130 02:58:17.234517 19840 consensus_queue.cc:260] T f7607df5efd543438fb2dff3667cc8c8 P 78c2c46c7520492db35d51c244fb4f74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:17.211149 19202 tablet_service.cc:1552] Processing DeleteTablet for tablet ec28ff12927d450ead16d26cf016138d with delete_type TABLET_DATA_DELETED (replaced by tablet 6d1f121f49cd47c2b8247daaf6874e01) from {username='slave'} at 127.0.0.1:42870
I20260130 02:58:17.280107 19840 ts_tablet_manager.cc:1434] T f7607df5efd543438fb2dff3667cc8c8 P 78c2c46c7520492db35d51c244fb4f74: Time spent starting tablet: real 0.049s user 0.006s sys 0.000s
I20260130 02:58:17.282016 19840 tablet_bootstrap.cc:492] T 133e9bfe175f44aab1a53114abe67e4d P 78c2c46c7520492db35d51c244fb4f74: Bootstrap starting.
I20260130 02:58:17.289089 19625 tablet_bootstrap.cc:654] T dcba6b534f9d406dad4fb502be40921c P d870164263bb44b6b780a557daea3656: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:17.303112 20690 tablet_replica.cc:333] T ec28ff12927d450ead16d26cf016138d P 371a36d219164974abdf848841528152: stopping tablet replica
I20260130 02:58:17.326258 19807 tablet_bootstrap.cc:492] T 838e49e5e8874db393bdc15a1c92f87e P 371a36d219164974abdf848841528152: No bootstrap required, opened a new log
I20260130 02:58:17.327098 19807 ts_tablet_manager.cc:1403] T 838e49e5e8874db393bdc15a1c92f87e P 371a36d219164974abdf848841528152: Time spent bootstrapping tablet: real 0.371s user 0.005s sys 0.025s
I20260130 02:58:17.331914 19840 tablet_bootstrap.cc:654] T 133e9bfe175f44aab1a53114abe67e4d P 78c2c46c7520492db35d51c244fb4f74: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:17.331792 19807 raft_consensus.cc:359] T 838e49e5e8874db393bdc15a1c92f87e P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:17.273528 20796 tablet.cc:1722] T 7caf7cbad5054396b5617094e4d536a0 P d870164263bb44b6b780a557daea3656: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260130 02:58:17.349843 19840 tablet_bootstrap.cc:492] T 133e9bfe175f44aab1a53114abe67e4d P 78c2c46c7520492db35d51c244fb4f74: No bootstrap required, opened a new log
W20260130 02:58:17.346392 19462 outbound_call.cc:321] RPC callback for RPC call kudu.master.MasterService.Ping -> {remote=127.18.100.124:34629, user_credentials={real_user=slave}} blocked reactor thread for 52907.1us
I20260130 02:58:17.350234 19840 ts_tablet_manager.cc:1403] T 133e9bfe175f44aab1a53114abe67e4d P 78c2c46c7520492db35d51c244fb4f74: Time spent bootstrapping tablet: real 0.068s user 0.008s sys 0.000s
I20260130 02:58:17.352255 20760 tablet.cc:1722] T 7caf7cbad5054396b5617094e4d536a0 P 78c2c46c7520492db35d51c244fb4f74: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260130 02:58:17.352391 19840 raft_consensus.cc:359] T 133e9bfe175f44aab1a53114abe67e4d P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:17.352988 19840 raft_consensus.cc:385] T 133e9bfe175f44aab1a53114abe67e4d P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:17.353219 19840 raft_consensus.cc:740] T 133e9bfe175f44aab1a53114abe67e4d P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78c2c46c7520492db35d51c244fb4f74, State: Initialized, Role: FOLLOWER
I20260130 02:58:17.362119 20763 tablet.cc:1722] T 7caf7cbad5054396b5617094e4d536a0 P 371a36d219164974abdf848841528152: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20260130 02:58:17.341600 19159 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.100.66:40767, user_credentials={real_user=slave}} blocked reactor thread for 55975.7us
I20260130 02:58:17.369329 20765 consensus_queue.cc:1048] T 5fbe92a2227d44d88b4531a7f8aa0e08 P 371a36d219164974abdf848841528152 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260130 02:58:17.371726 20765 consensus_queue.cc:1048] T a90712651a604b83ba187f893ab17711 P 371a36d219164974abdf848841528152 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260130 02:58:17.373276 20765 consensus_queue.cc:1048] T 274530359b174ff79f046c2d4a6afa76 P 371a36d219164974abdf848841528152 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260130 02:58:17.354689 19807 raft_consensus.cc:385] T 838e49e5e8874db393bdc15a1c92f87e P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:17.374271 19807 raft_consensus.cc:740] T 838e49e5e8874db393bdc15a1c92f87e P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 371a36d219164974abdf848841528152, State: Initialized, Role: FOLLOWER
I20260130 02:58:17.374888 19807 consensus_queue.cc:260] T 838e49e5e8874db393bdc15a1c92f87e P 371a36d219164974abdf848841528152 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:17.376492 19807 ts_tablet_manager.cc:1434] T 838e49e5e8874db393bdc15a1c92f87e P 371a36d219164974abdf848841528152: Time spent starting tablet: real 0.049s user 0.005s sys 0.000s
I20260130 02:58:17.377465 19807 tablet_bootstrap.cc:492] T 25fd48e693424537b91ba7c3fcc96878 P 371a36d219164974abdf848841528152: Bootstrap starting.
I20260130 02:58:17.366281 19840 consensus_queue.cc:260] T 133e9bfe175f44aab1a53114abe67e4d P 78c2c46c7520492db35d51c244fb4f74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:17.392647 19840 ts_tablet_manager.cc:1434] T 133e9bfe175f44aab1a53114abe67e4d P 78c2c46c7520492db35d51c244fb4f74: Time spent starting tablet: real 0.042s user 0.003s sys 0.003s
I20260130 02:58:17.393563 19840 tablet_bootstrap.cc:492] T a3258852aed24ed582a79f77860e6f0d P 78c2c46c7520492db35d51c244fb4f74: Bootstrap starting.
I20260130 02:58:17.294140 18860 catalog_manager.cc:4985] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): tablet ec28ff12927d450ead16d26cf016138d (table default.Table_0480357296ac4a27812d802ee4566351 [id=0da6197ab32e4645aa8d8787662ee5e6]) successfully deleted
W20260130 02:58:17.399060 18860 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.18.100.67:39571, user_credentials={real_user=slave}} blocked reactor thread for 104954us
I20260130 02:58:17.399300 18867 catalog_manager.cc:5746] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767) does not have the latest schema for tablet 4c139aa8cd184f488cec4e05ef7dcada (table default.Table_e4d07634d84a4678b1c94374e3adc476 [id=fcbd7e3a7fc64d89aa740494d7e1a0f7]). Expected version 1 got 0
I20260130 02:58:17.400059 18867 catalog_manager.cc:5654] T 4907294442c145c6b03eb1de057c66a8 P 78c2c46c7520492db35d51c244fb4f74 reported cstate change: term changed from 0 to 1, leader changed from <none> to 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66). New cstate: current_term: 1 leader_uuid: "78c2c46c7520492db35d51c244fb4f74" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } health_report { overall_health: UNKNOWN } } }
I20260130 02:58:17.401679 18867 catalog_manager.cc:5746] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767) does not have the latest schema for tablet 7caf7cbad5054396b5617094e4d536a0 (table default.Table_007e969f70d345a7bcb3dfbaabbc025d [id=77d4fa8705a3419cb0a84d3ca1bb4ce3]). Expected version 1 got 0
I20260130 02:58:17.402313 18867 catalog_manager.cc:5746] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767) does not have the latest schema for tablet 0e8018c094024c4788c484e8f5466e00 (table default.Table_f17d8e4d9f25449b84aa751af93936b2 [id=43254e466e7b4a2fb59fbcec68cfde36]). Expected version 1 got 0
I20260130 02:58:17.407166 19840 tablet_bootstrap.cc:654] T a3258852aed24ed582a79f77860e6f0d P 78c2c46c7520492db35d51c244fb4f74: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:17.421888 19521 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6218f2f7d73a42428b57a45858f53afd" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656"
I20260130 02:58:17.422545 19521 raft_consensus.cc:3060] T 6218f2f7d73a42428b57a45858f53afd P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:17.422787 19379 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6218f2f7d73a42428b57a45858f53afd" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74"
I20260130 02:58:17.423364 19379 raft_consensus.cc:3060] T 6218f2f7d73a42428b57a45858f53afd P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:17.428673 19840 tablet_bootstrap.cc:492] T a3258852aed24ed582a79f77860e6f0d P 78c2c46c7520492db35d51c244fb4f74: No bootstrap required, opened a new log
I20260130 02:58:17.429080 19840 ts_tablet_manager.cc:1403] T a3258852aed24ed582a79f77860e6f0d P 78c2c46c7520492db35d51c244fb4f74: Time spent bootstrapping tablet: real 0.036s user 0.004s sys 0.004s
I20260130 02:58:17.430557 19379 raft_consensus.cc:2468] T 6218f2f7d73a42428b57a45858f53afd P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 371a36d219164974abdf848841528152 in term 1.
I20260130 02:58:17.434248 19840 raft_consensus.cc:359] T a3258852aed24ed582a79f77860e6f0d P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:17.434931 19840 raft_consensus.cc:385] T a3258852aed24ed582a79f77860e6f0d P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:17.435173 19840 raft_consensus.cc:740] T a3258852aed24ed582a79f77860e6f0d P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78c2c46c7520492db35d51c244fb4f74, State: Initialized, Role: FOLLOWER
I20260130 02:58:17.441557 19840 consensus_queue.cc:260] T a3258852aed24ed582a79f77860e6f0d P 78c2c46c7520492db35d51c244fb4f74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:17.443850 19625 tablet_bootstrap.cc:492] T dcba6b534f9d406dad4fb502be40921c P d870164263bb44b6b780a557daea3656: No bootstrap required, opened a new log
I20260130 02:58:17.444271 19625 ts_tablet_manager.cc:1403] T dcba6b534f9d406dad4fb502be40921c P d870164263bb44b6b780a557daea3656: Time spent bootstrapping tablet: real 0.279s user 0.006s sys 0.004s
I20260130 02:58:17.446722 19625 raft_consensus.cc:359] T dcba6b534f9d406dad4fb502be40921c P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:17.449718 19625 raft_consensus.cc:385] T dcba6b534f9d406dad4fb502be40921c P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:17.450793 19625 raft_consensus.cc:740] T dcba6b534f9d406dad4fb502be40921c P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d870164263bb44b6b780a557daea3656, State: Initialized, Role: FOLLOWER
I20260130 02:58:17.452260 19840 ts_tablet_manager.cc:1434] T a3258852aed24ed582a79f77860e6f0d P 78c2c46c7520492db35d51c244fb4f74: Time spent starting tablet: real 0.023s user 0.006s sys 0.000s
I20260130 02:58:17.453141 19840 tablet_bootstrap.cc:492] T dfbbe9045e7c4aa5b520218687e7e4c1 P 78c2c46c7520492db35d51c244fb4f74: Bootstrap starting.
I20260130 02:58:17.464802 19625 consensus_queue.cc:260] T dcba6b534f9d406dad4fb502be40921c P d870164263bb44b6b780a557daea3656 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:17.465574 20679 raft_consensus.cc:493] T dcba6b534f9d406dad4fb502be40921c P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:17.465969 20679 raft_consensus.cc:515] T dcba6b534f9d406dad4fb502be40921c P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:17.469815 20679 leader_election.cc:290] T dcba6b534f9d406dad4fb502be40921c P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767)
I20260130 02:58:17.472023 19840 tablet_bootstrap.cc:654] T dfbbe9045e7c4aa5b520218687e7e4c1 P 78c2c46c7520492db35d51c244fb4f74: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:17.471793 19221 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dcba6b534f9d406dad4fb502be40921c" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152" is_pre_election: true
I20260130 02:58:17.473193 19375 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dcba6b534f9d406dad4fb502be40921c" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74" is_pre_election: true
I20260130 02:58:17.473913 19375 raft_consensus.cc:2393] T dcba6b534f9d406dad4fb502be40921c P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d870164263bb44b6b780a557daea3656 in current term 1: Already voted for candidate 371a36d219164974abdf848841528152 in this term.
I20260130 02:58:17.475950 19462 leader_election.cc:304] T dcba6b534f9d406dad4fb502be40921c P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d870164263bb44b6b780a557daea3656; no voters: 371a36d219164974abdf848841528152, 78c2c46c7520492db35d51c244fb4f74
I20260130 02:58:17.476990 20679 raft_consensus.cc:3060] T dcba6b534f9d406dad4fb502be40921c P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:17.484093 19840 tablet_bootstrap.cc:492] T dfbbe9045e7c4aa5b520218687e7e4c1 P 78c2c46c7520492db35d51c244fb4f74: No bootstrap required, opened a new log
I20260130 02:58:17.484509 19840 ts_tablet_manager.cc:1403] T dfbbe9045e7c4aa5b520218687e7e4c1 P 78c2c46c7520492db35d51c244fb4f74: Time spent bootstrapping tablet: real 0.032s user 0.007s sys 0.001s
I20260130 02:58:17.493039 20679 raft_consensus.cc:2749] T dcba6b534f9d406dad4fb502be40921c P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260130 02:58:17.493216 19840 raft_consensus.cc:359] T dfbbe9045e7c4aa5b520218687e7e4c1 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:17.493844 19840 raft_consensus.cc:385] T dfbbe9045e7c4aa5b520218687e7e4c1 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:17.494083 19840 raft_consensus.cc:740] T dfbbe9045e7c4aa5b520218687e7e4c1 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78c2c46c7520492db35d51c244fb4f74, State: Initialized, Role: FOLLOWER
I20260130 02:58:17.494665 19840 consensus_queue.cc:260] T dfbbe9045e7c4aa5b520218687e7e4c1 P 78c2c46c7520492db35d51c244fb4f74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:17.496670 19840 ts_tablet_manager.cc:1434] T dfbbe9045e7c4aa5b520218687e7e4c1 P 78c2c46c7520492db35d51c244fb4f74: Time spent starting tablet: real 0.012s user 0.005s sys 0.000s
I20260130 02:58:17.497474 19840 tablet_bootstrap.cc:492] T aae68390dbbe4dc58ce3d3bd01725ae6 P 78c2c46c7520492db35d51c244fb4f74: Bootstrap starting.
I20260130 02:58:17.519898 19840 tablet_bootstrap.cc:654] T aae68390dbbe4dc58ce3d3bd01725ae6 P 78c2c46c7520492db35d51c244fb4f74: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:17.540380 19840 tablet_bootstrap.cc:492] T aae68390dbbe4dc58ce3d3bd01725ae6 P 78c2c46c7520492db35d51c244fb4f74: No bootstrap required, opened a new log
I20260130 02:58:17.540774 19840 ts_tablet_manager.cc:1403] T aae68390dbbe4dc58ce3d3bd01725ae6 P 78c2c46c7520492db35d51c244fb4f74: Time spent bootstrapping tablet: real 0.043s user 0.008s sys 0.000s
I20260130 02:58:17.546010 19807 tablet_bootstrap.cc:654] T 25fd48e693424537b91ba7c3fcc96878 P 371a36d219164974abdf848841528152: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:17.549623 19840 raft_consensus.cc:359] T aae68390dbbe4dc58ce3d3bd01725ae6 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } }
I20260130 02:58:17.550256 19840 raft_consensus.cc:385] T aae68390dbbe4dc58ce3d3bd01725ae6 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:17.550503 19840 raft_consensus.cc:740] T aae68390dbbe4dc58ce3d3bd01725ae6 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78c2c46c7520492db35d51c244fb4f74, State: Initialized, Role: FOLLOWER
I20260130 02:58:17.552588 20840 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.100.124:34629 every 8 connection(s)
I20260130 02:58:17.449090 19521 raft_consensus.cc:2468] T 6218f2f7d73a42428b57a45858f53afd P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 371a36d219164974abdf848841528152 in term 1.
I20260130 02:58:17.559612 19872 rpc_server.cc:307] RPC server started. Bound to: 127.18.100.124:34629
I20260130 02:58:17.564460 19625 ts_tablet_manager.cc:1434] T dcba6b534f9d406dad4fb502be40921c P d870164263bb44b6b780a557daea3656: Time spent starting tablet: real 0.120s user 0.007s sys 0.001s
I20260130 02:58:17.567342 19840 consensus_queue.cc:260] T aae68390dbbe4dc58ce3d3bd01725ae6 P 78c2c46c7520492db35d51c244fb4f74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } }
I20260130 02:58:17.568624 19872 server_base.cc:1192] Dumped server information to /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/master-2/data/info.pb
I20260130 02:58:17.575440 18833 external_mini_cluster.cc:1428] Started /tmp/dist-test-tasktGe2wI/build/tsan/bin/kudu as pid 19872
I20260130 02:58:17.578051 19840 ts_tablet_manager.cc:1434] T aae68390dbbe4dc58ce3d3bd01725ae6 P 78c2c46c7520492db35d51c244fb4f74: Time spent starting tablet: real 0.037s user 0.006s sys 0.000s
I20260130 02:58:17.591960 19840 tablet_bootstrap.cc:492] T 657e9b0d8c05413f904129dce745710b P 78c2c46c7520492db35d51c244fb4f74: Bootstrap starting.
I20260130 02:58:17.595979 18871 catalog_manager.cc:5746] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021) does not have the latest schema for tablet 345eaaa5625c48d099280b7fbb2e514b (table default.Table_c1886dd9a7934e7797e0e7e0b065d692 [id=7a2d5367224b492b844199088b01b149]). Expected version 1 got 0
I20260130 02:58:17.597033 18871 catalog_manager.cc:5746] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021) does not have the latest schema for tablet a8a5038e9def44eb8b3b1e4aeae25652 (table default.Table_dee034aec0b24982b8dc2a3fa1b326f1 [id=66033e75b3814a8584311cf7e4a07ad3]). Expected version 1 got 0
I20260130 02:58:17.598654 18871 catalog_manager.cc:5654] T b1e8b78eff0d4a0b92059adb2de4a3ce P 371a36d219164974abdf848841528152 reported cstate change: term changed from 0 to 1, leader changed from <none> to 371a36d219164974abdf848841528152 (127.18.100.65). New cstate: current_term: 1 leader_uuid: "371a36d219164974abdf848841528152" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } health_report { overall_health: UNKNOWN } } }
I20260130 02:58:17.610005 18871 catalog_manager.cc:5746] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021) does not have the latest schema for tablet b1e8b78eff0d4a0b92059adb2de4a3ce (table default.Table_d0b4516d24384e5386c1debaf2543adb [id=f1342f9497b74e7a9ecad1de8082578d]). Expected version 1 got 0
I20260130 02:58:17.618543 20690 raft_consensus.cc:2243] T ec28ff12927d450ead16d26cf016138d P 371a36d219164974abdf848841528152 [term 1 LEADER]: Raft consensus shutting down.
I20260130 02:58:17.619655 20690 raft_consensus.cc:2272] T ec28ff12927d450ead16d26cf016138d P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260130 02:58:17.620350 18871 catalog_manager.cc:5654] T 7783db3568f4434e91e9e456799cc312 P 371a36d219164974abdf848841528152 reported cstate change: term changed from 0 to 1, leader changed from <none> to 371a36d219164974abdf848841528152 (127.18.100.65). New cstate: current_term: 1 leader_uuid: "371a36d219164974abdf848841528152" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } health_report { overall_health: UNKNOWN } } }
I20260130 02:58:17.621922 18871 catalog_manager.cc:5746] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021) does not have the latest schema for tablet 7caf7cbad5054396b5617094e4d536a0 (table default.Table_007e969f70d345a7bcb3dfbaabbc025d [id=77d4fa8705a3419cb0a84d3ca1bb4ce3]). Expected version 1 got 0
I20260130 02:58:17.622062 20690 ts_tablet_manager.cc:1916] T ec28ff12927d450ead16d26cf016138d P 371a36d219164974abdf848841528152: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260130 02:58:17.622870 20844 sys_catalog.cc:263] Verifying existing consensus state
I20260130 02:58:17.623874 20703 raft_consensus.cc:493] T 133e9bfe175f44aab1a53114abe67e4d P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:17.624327 20703 raft_consensus.cc:515] T 133e9bfe175f44aab1a53114abe67e4d P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:17.625094 18871 catalog_manager.cc:5746] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021) does not have the latest schema for tablet 274530359b174ff79f046c2d4a6afa76 (table default.Table_c4f4904b6fac4b9880979cf0314e71c1 [id=c456c6418c93499a9d08f652720854ca]). Expected version 1 got 0
I20260130 02:58:17.626101 20703 leader_election.cc:290] T 133e9bfe175f44aab1a53114abe67e4d P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d870164263bb44b6b780a557daea3656 (127.18.100.67:39571), 371a36d219164974abdf848841528152 (127.18.100.65:42021)
I20260130 02:58:17.626835 19521 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "133e9bfe175f44aab1a53114abe67e4d" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
I20260130 02:58:17.628448 19221 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "133e9bfe175f44aab1a53114abe67e4d" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152" is_pre_election: true
W20260130 02:58:17.629573 19314 leader_election.cc:343] T 133e9bfe175f44aab1a53114abe67e4d P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Illegal state: must be running to vote when last-logged opid is not known
W20260130 02:58:17.630697 19313 leader_election.cc:343] T 133e9bfe175f44aab1a53114abe67e4d P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:17.631034 19313 leader_election.cc:304] T 133e9bfe175f44aab1a53114abe67e4d P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78c2c46c7520492db35d51c244fb4f74; no voters: 371a36d219164974abdf848841528152, d870164263bb44b6b780a557daea3656
W20260130 02:58:17.631822 19159 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.100.67:39571, user_credentials={real_user=slave}} blocked reactor thread for 68600.5us
I20260130 02:58:17.633260 19840 tablet_bootstrap.cc:654] T 657e9b0d8c05413f904129dce745710b P 78c2c46c7520492db35d51c244fb4f74: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:17.633666 20703 raft_consensus.cc:2749] T 133e9bfe175f44aab1a53114abe67e4d P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260130 02:58:17.642846 20765 consensus_queue.cc:1048] T e2aaaf00f53d418b87d5c7af286e4d84 P 371a36d219164974abdf848841528152 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260130 02:58:17.643297 19625 tablet_bootstrap.cc:492] T cf1b66a20fce4c4d96bc0bc32a7bafa8 P d870164263bb44b6b780a557daea3656: Bootstrap starting.
I20260130 02:58:17.652899 19625 tablet_bootstrap.cc:654] T cf1b66a20fce4c4d96bc0bc32a7bafa8 P d870164263bb44b6b780a557daea3656: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:17.660972 18871 catalog_manager.cc:5746] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021) does not have the latest schema for tablet 16691c9315e94253bb9b3fb5058e5636 (table default.Table_8f311c5878744ef692b496ecfdfc3ede [id=4f499357303d464aafc17679ffcaaa07]). Expected version 1 got 0
I20260130 02:58:17.670336 19159 leader_election.cc:304] T 6218f2f7d73a42428b57a45858f53afd P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 371a36d219164974abdf848841528152, d870164263bb44b6b780a557daea3656; no voters:
I20260130 02:58:17.674800 20765 raft_consensus.cc:2804] T 6218f2f7d73a42428b57a45858f53afd P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Leader election won for term 1
I20260130 02:58:17.675499 20765 raft_consensus.cc:697] T 6218f2f7d73a42428b57a45858f53afd P 371a36d219164974abdf848841528152 [term 1 LEADER]: Becoming Leader. State: Replica: 371a36d219164974abdf848841528152, State: Running, Role: LEADER
I20260130 02:58:17.676327 20765 consensus_queue.cc:237] T 6218f2f7d73a42428b57a45858f53afd P 371a36d219164974abdf848841528152 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:17.674804 18871 catalog_manager.cc:5746] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021) does not have the latest schema for tablet 4c139aa8cd184f488cec4e05ef7dcada (table default.Table_e4d07634d84a4678b1c94374e3adc476 [id=fcbd7e3a7fc64d89aa740494d7e1a0f7]). Expected version 1 got 0
I20260130 02:58:17.677728 18871 catalog_manager.cc:5746] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021) does not have the latest schema for tablet 9fe0448bcc4a4346b3c67fc3f626ed3c (table default.Table_5c1c447e85524ac5a61ffd8a48509d36 [id=70aaf21ad9bd4d959f46939c6b4a4880]). Expected version 1 got 0
I20260130 02:58:17.681594 19625 tablet_bootstrap.cc:492] T cf1b66a20fce4c4d96bc0bc32a7bafa8 P d870164263bb44b6b780a557daea3656: No bootstrap required, opened a new log
I20260130 02:58:17.682065 19625 ts_tablet_manager.cc:1403] T cf1b66a20fce4c4d96bc0bc32a7bafa8 P d870164263bb44b6b780a557daea3656: Time spent bootstrapping tablet: real 0.039s user 0.009s sys 0.001s
I20260130 02:58:17.682163 20844 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3749baa287ed4f249329a74071781edd: Bootstrap starting.
I20260130 02:58:17.679529 18871 catalog_manager.cc:5746] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021) does not have the latest schema for tablet 0e8018c094024c4788c484e8f5466e00 (table default.Table_f17d8e4d9f25449b84aa751af93936b2 [id=43254e466e7b4a2fb59fbcec68cfde36]). Expected version 1 got 0
I20260130 02:58:17.698024 19625 raft_consensus.cc:359] T cf1b66a20fce4c4d96bc0bc32a7bafa8 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:17.698496 19840 tablet_bootstrap.cc:492] T 657e9b0d8c05413f904129dce745710b P 78c2c46c7520492db35d51c244fb4f74: No bootstrap required, opened a new log
I20260130 02:58:17.698915 19840 ts_tablet_manager.cc:1403] T 657e9b0d8c05413f904129dce745710b P 78c2c46c7520492db35d51c244fb4f74: Time spent bootstrapping tablet: real 0.107s user 0.015s sys 0.000s
I20260130 02:58:17.701146 19840 raft_consensus.cc:359] T 657e9b0d8c05413f904129dce745710b P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:17.701761 19840 raft_consensus.cc:385] T 657e9b0d8c05413f904129dce745710b P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:17.702013 19840 raft_consensus.cc:740] T 657e9b0d8c05413f904129dce745710b P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78c2c46c7520492db35d51c244fb4f74, State: Initialized, Role: FOLLOWER
I20260130 02:58:17.702420 19625 raft_consensus.cc:385] T cf1b66a20fce4c4d96bc0bc32a7bafa8 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:17.703258 19840 consensus_queue.cc:260] T 657e9b0d8c05413f904129dce745710b P 78c2c46c7520492db35d51c244fb4f74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:17.703998 19625 raft_consensus.cc:740] T cf1b66a20fce4c4d96bc0bc32a7bafa8 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d870164263bb44b6b780a557daea3656, State: Initialized, Role: FOLLOWER
I20260130 02:58:17.704890 19625 consensus_queue.cc:260] T cf1b66a20fce4c4d96bc0bc32a7bafa8 P d870164263bb44b6b780a557daea3656 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:17.705313 19840 ts_tablet_manager.cc:1434] T 657e9b0d8c05413f904129dce745710b P 78c2c46c7520492db35d51c244fb4f74: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20260130 02:58:17.703002 18871 catalog_manager.cc:5654] T ba91a508b1cb4430a4d8ca26946aecc8 P 371a36d219164974abdf848841528152 reported cstate change: term changed from 0 to 1, leader changed from <none> to 371a36d219164974abdf848841528152 (127.18.100.65). New cstate: current_term: 1 leader_uuid: "371a36d219164974abdf848841528152" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } health_report { overall_health: UNKNOWN } } }
I20260130 02:58:17.707589 19840 tablet_bootstrap.cc:492] T d480bf18140245e3870a62582d5a8318 P 78c2c46c7520492db35d51c244fb4f74: Bootstrap starting.
I20260130 02:58:17.711633 20703 raft_consensus.cc:493] T 320dbe98a32a43b9899bbeed80b20f55 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:17.712070 20703 raft_consensus.cc:515] T 320dbe98a32a43b9899bbeed80b20f55 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:17.712836 19840 tablet_bootstrap.cc:654] T d480bf18140245e3870a62582d5a8318 P 78c2c46c7520492db35d51c244fb4f74: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:17.713301 19625 ts_tablet_manager.cc:1434] T cf1b66a20fce4c4d96bc0bc32a7bafa8 P d870164263bb44b6b780a557daea3656: Time spent starting tablet: real 0.031s user 0.005s sys 0.000s
I20260130 02:58:17.713845 20703 leader_election.cc:290] T 320dbe98a32a43b9899bbeed80b20f55 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d870164263bb44b6b780a557daea3656 (127.18.100.67:39571), 371a36d219164974abdf848841528152 (127.18.100.65:42021)
I20260130 02:58:17.714488 19521 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "320dbe98a32a43b9899bbeed80b20f55" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
W20260130 02:58:17.715477 19313 leader_election.cc:343] T 320dbe98a32a43b9899bbeed80b20f55 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:17.715447 19221 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "320dbe98a32a43b9899bbeed80b20f55" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152" is_pre_election: true
W20260130 02:58:17.716583 19314 leader_election.cc:343] T 320dbe98a32a43b9899bbeed80b20f55 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:17.716917 19314 leader_election.cc:304] T 320dbe98a32a43b9899bbeed80b20f55 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78c2c46c7520492db35d51c244fb4f74; no voters: 371a36d219164974abdf848841528152, d870164263bb44b6b780a557daea3656
I20260130 02:58:17.718255 20703 raft_consensus.cc:2749] T 320dbe98a32a43b9899bbeed80b20f55 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260130 02:58:17.730060 19807 tablet_bootstrap.cc:492] T 25fd48e693424537b91ba7c3fcc96878 P 371a36d219164974abdf848841528152: No bootstrap required, opened a new log
I20260130 02:58:17.730530 19807 ts_tablet_manager.cc:1403] T 25fd48e693424537b91ba7c3fcc96878 P 371a36d219164974abdf848841528152: Time spent bootstrapping tablet: real 0.353s user 0.005s sys 0.004s
I20260130 02:58:17.733544 19807 raft_consensus.cc:359] T 25fd48e693424537b91ba7c3fcc96878 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:17.734175 19807 raft_consensus.cc:385] T 25fd48e693424537b91ba7c3fcc96878 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:17.734424 19807 raft_consensus.cc:740] T 25fd48e693424537b91ba7c3fcc96878 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 371a36d219164974abdf848841528152, State: Initialized, Role: FOLLOWER
I20260130 02:58:17.742236 20844 log.cc:826] T 00000000000000000000000000000000 P 3749baa287ed4f249329a74071781edd: Log is configured to *not* fsync() on all Append() calls
I20260130 02:58:17.763077 20703 raft_consensus.cc:493] T f7607df5efd543438fb2dff3667cc8c8 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:17.763540 20703 raft_consensus.cc:515] T f7607df5efd543438fb2dff3667cc8c8 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:17.764317 19840 tablet_bootstrap.cc:492] T d480bf18140245e3870a62582d5a8318 P 78c2c46c7520492db35d51c244fb4f74: No bootstrap required, opened a new log
I20260130 02:58:17.764704 19840 ts_tablet_manager.cc:1403] T d480bf18140245e3870a62582d5a8318 P 78c2c46c7520492db35d51c244fb4f74: Time spent bootstrapping tablet: real 0.057s user 0.005s sys 0.003s
I20260130 02:58:17.765287 20697 raft_consensus.cc:493] T a3258852aed24ed582a79f77860e6f0d P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:17.765801 20697 raft_consensus.cc:515] T a3258852aed24ed582a79f77860e6f0d P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:17.766413 19528 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f7607df5efd543438fb2dff3667cc8c8" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
I20260130 02:58:17.767545 20697 leader_election.cc:290] T a3258852aed24ed582a79f77860e6f0d P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d870164263bb44b6b780a557daea3656 (127.18.100.67:39571), 371a36d219164974abdf848841528152 (127.18.100.65:42021)
W20260130 02:58:17.767679 19313 leader_election.cc:343] T f7607df5efd543438fb2dff3667cc8c8 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:17.767604 19840 raft_consensus.cc:359] T d480bf18140245e3870a62582d5a8318 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } }
I20260130 02:58:17.768023 19528 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a3258852aed24ed582a79f77860e6f0d" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
I20260130 02:58:17.768236 19840 raft_consensus.cc:385] T d480bf18140245e3870a62582d5a8318 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:17.768471 19840 raft_consensus.cc:740] T d480bf18140245e3870a62582d5a8318 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78c2c46c7520492db35d51c244fb4f74, State: Initialized, Role: FOLLOWER
I20260130 02:58:17.768981 19840 consensus_queue.cc:260] T d480bf18140245e3870a62582d5a8318 P 78c2c46c7520492db35d51c244fb4f74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } }
I20260130 02:58:17.769747 19221 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f7607df5efd543438fb2dff3667cc8c8" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152" is_pre_election: true
I20260130 02:58:17.769872 19222 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a3258852aed24ed582a79f77860e6f0d" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152" is_pre_election: true
W20260130 02:58:17.770357 19313 leader_election.cc:343] T a3258852aed24ed582a79f77860e6f0d P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:17.771044 19625 tablet_bootstrap.cc:492] T 25fd48e693424537b91ba7c3fcc96878 P d870164263bb44b6b780a557daea3656: Bootstrap starting.
W20260130 02:58:17.771158 19314 leader_election.cc:343] T f7607df5efd543438fb2dff3667cc8c8 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:17.771531 19314 leader_election.cc:304] T f7607df5efd543438fb2dff3667cc8c8 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78c2c46c7520492db35d51c244fb4f74; no voters: 371a36d219164974abdf848841528152, d870164263bb44b6b780a557daea3656
I20260130 02:58:17.765321 20703 leader_election.cc:290] T f7607df5efd543438fb2dff3667cc8c8 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d870164263bb44b6b780a557daea3656 (127.18.100.67:39571), 371a36d219164974abdf848841528152 (127.18.100.65:42021)
I20260130 02:58:17.777707 19625 tablet_bootstrap.cc:654] T 25fd48e693424537b91ba7c3fcc96878 P d870164263bb44b6b780a557daea3656: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:17.782104 20697 raft_consensus.cc:2749] T f7607df5efd543438fb2dff3667cc8c8 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260130 02:58:17.782348 19314 leader_election.cc:343] T a3258852aed24ed582a79f77860e6f0d P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:17.782718 19314 leader_election.cc:304] T a3258852aed24ed582a79f77860e6f0d P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78c2c46c7520492db35d51c244fb4f74; no voters: 371a36d219164974abdf848841528152, d870164263bb44b6b780a557daea3656
I20260130 02:58:17.783658 20704 raft_consensus.cc:2749] T a3258852aed24ed582a79f77860e6f0d P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260130 02:58:17.784008 19840 ts_tablet_manager.cc:1434] T d480bf18140245e3870a62582d5a8318 P 78c2c46c7520492db35d51c244fb4f74: Time spent starting tablet: real 0.019s user 0.006s sys 0.001s
I20260130 02:58:17.784835 19840 tablet_bootstrap.cc:492] T 49efcfced0b74c48962393149ac2a500 P 78c2c46c7520492db35d51c244fb4f74: Bootstrap starting.
I20260130 02:58:17.795604 20690 ts_tablet_manager.cc:1929] T ec28ff12927d450ead16d26cf016138d P 371a36d219164974abdf848841528152: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260130 02:58:17.796131 20690 log.cc:1199] T ec28ff12927d450ead16d26cf016138d P 371a36d219164974abdf848841528152: Deleting WAL directory at /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-0/wal/wals/ec28ff12927d450ead16d26cf016138d
I20260130 02:58:17.797070 20690 ts_tablet_manager.cc:1950] T ec28ff12927d450ead16d26cf016138d P 371a36d219164974abdf848841528152: Deleting consensus metadata
I20260130 02:58:17.800264 18861 catalog_manager.cc:4985] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021): tablet ec28ff12927d450ead16d26cf016138d (table default.Table_0480357296ac4a27812d802ee4566351 [id=0da6197ab32e4645aa8d8787662ee5e6]) successfully deleted
I20260130 02:58:17.809877 19840 tablet_bootstrap.cc:654] T 49efcfced0b74c48962393149ac2a500 P 78c2c46c7520492db35d51c244fb4f74: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:17.815920 20697 raft_consensus.cc:493] T aae68390dbbe4dc58ce3d3bd01725ae6 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:17.816388 20697 raft_consensus.cc:515] T aae68390dbbe4dc58ce3d3bd01725ae6 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } }
I20260130 02:58:17.816692 20799 raft_consensus.cc:493] T f9e5f01e69254d658c369f3e21cef523 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:17.817116 20799 raft_consensus.cc:515] T f9e5f01e69254d658c369f3e21cef523 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:17.818125 20697 leader_election.cc:290] T aae68390dbbe4dc58ce3d3bd01725ae6 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d870164263bb44b6b780a557daea3656 (127.18.100.67:39571), 371a36d219164974abdf848841528152 (127.18.100.65:42021)
I20260130 02:58:17.819005 20799 leader_election.cc:290] T f9e5f01e69254d658c369f3e21cef523 P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d870164263bb44b6b780a557daea3656 (127.18.100.67:39571), 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767)
I20260130 02:58:17.819953 19222 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "aae68390dbbe4dc58ce3d3bd01725ae6" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152" is_pre_election: true
W20260130 02:58:17.821113 19314 leader_election.cc:343] T aae68390dbbe4dc58ce3d3bd01725ae6 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:17.821674 19528 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "aae68390dbbe4dc58ce3d3bd01725ae6" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
W20260130 02:58:17.830828 19313 leader_election.cc:343] T aae68390dbbe4dc58ce3d3bd01725ae6 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:17.831364 19313 leader_election.cc:304] T aae68390dbbe4dc58ce3d3bd01725ae6 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78c2c46c7520492db35d51c244fb4f74; no voters: 371a36d219164974abdf848841528152, d870164263bb44b6b780a557daea3656
I20260130 02:58:17.832289 20697 raft_consensus.cc:2749] T aae68390dbbe4dc58ce3d3bd01725ae6 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260130 02:58:17.836024 19807 consensus_queue.cc:260] T 25fd48e693424537b91ba7c3fcc96878 P 371a36d219164974abdf848841528152 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:17.838671 19807 ts_tablet_manager.cc:1434] T 25fd48e693424537b91ba7c3fcc96878 P 371a36d219164974abdf848841528152: Time spent starting tablet: real 0.107s user 0.006s sys 0.000s
I20260130 02:58:17.840610 19840 tablet_bootstrap.cc:492] T 49efcfced0b74c48962393149ac2a500 P 78c2c46c7520492db35d51c244fb4f74: No bootstrap required, opened a new log
I20260130 02:58:17.841001 19840 ts_tablet_manager.cc:1403] T 49efcfced0b74c48962393149ac2a500 P 78c2c46c7520492db35d51c244fb4f74: Time spent bootstrapping tablet: real 0.056s user 0.004s sys 0.004s
I20260130 02:58:17.856393 19840 raft_consensus.cc:359] T 49efcfced0b74c48962393149ac2a500 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } }
I20260130 02:58:17.857244 19840 raft_consensus.cc:385] T 49efcfced0b74c48962393149ac2a500 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:17.857512 19840 raft_consensus.cc:740] T 49efcfced0b74c48962393149ac2a500 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78c2c46c7520492db35d51c244fb4f74, State: Initialized, Role: FOLLOWER
I20260130 02:58:17.860617 19425 heartbeater.cc:344] Connected to a master server at 127.18.100.124:34629
I20260130 02:58:17.860886 19840 consensus_queue.cc:260] T 49efcfced0b74c48962393149ac2a500 P 78c2c46c7520492db35d51c244fb4f74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } }
I20260130 02:58:17.868067 19573 heartbeater.cc:344] Connected to a master server at 127.18.100.124:34629
I20260130 02:58:17.870759 20799 raft_consensus.cc:493] T 25fd48e693424537b91ba7c3fcc96878 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:17.871258 20799 raft_consensus.cc:515] T 25fd48e693424537b91ba7c3fcc96878 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:17.873111 20799 leader_election.cc:290] T 25fd48e693424537b91ba7c3fcc96878 P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767), d870164263bb44b6b780a557daea3656 (127.18.100.67:39571)
I20260130 02:58:17.875504 19840 ts_tablet_manager.cc:1434] T 49efcfced0b74c48962393149ac2a500 P 78c2c46c7520492db35d51c244fb4f74: Time spent starting tablet: real 0.034s user 0.006s sys 0.001s
I20260130 02:58:17.876363 19840 tablet_bootstrap.cc:492] T ca06f07bb98f45a7bf63aebc28348e08 P 78c2c46c7520492db35d51c244fb4f74: Bootstrap starting.
I20260130 02:58:17.892596 19625 tablet_bootstrap.cc:492] T 25fd48e693424537b91ba7c3fcc96878 P d870164263bb44b6b780a557daea3656: No bootstrap required, opened a new log
I20260130 02:58:17.895714 20704 raft_consensus.cc:493] T 49efcfced0b74c48962393149ac2a500 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:17.896335 19625 ts_tablet_manager.cc:1403] T 25fd48e693424537b91ba7c3fcc96878 P d870164263bb44b6b780a557daea3656: Time spent bootstrapping tablet: real 0.125s user 0.006s sys 0.004s
I20260130 02:58:17.896143 20704 raft_consensus.cc:515] T 49efcfced0b74c48962393149ac2a500 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } }
I20260130 02:58:17.897861 20704 leader_election.cc:290] T 49efcfced0b74c48962393149ac2a500 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d870164263bb44b6b780a557daea3656 (127.18.100.67:39571), 371a36d219164974abdf848841528152 (127.18.100.65:42021)
I20260130 02:58:17.900609 19522 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "49efcfced0b74c48962393149ac2a500" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
W20260130 02:58:17.907776 19313 leader_election.cc:343] T 49efcfced0b74c48962393149ac2a500 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:17.909165 19625 raft_consensus.cc:359] T 25fd48e693424537b91ba7c3fcc96878 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:17.909787 19625 raft_consensus.cc:385] T 25fd48e693424537b91ba7c3fcc96878 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:17.910027 19625 raft_consensus.cc:740] T 25fd48e693424537b91ba7c3fcc96878 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d870164263bb44b6b780a557daea3656, State: Initialized, Role: FOLLOWER
I20260130 02:58:17.922876 19840 tablet_bootstrap.cc:654] T ca06f07bb98f45a7bf63aebc28348e08 P 78c2c46c7520492db35d51c244fb4f74: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:17.924196 19625 consensus_queue.cc:260] T 25fd48e693424537b91ba7c3fcc96878 P d870164263bb44b6b780a557daea3656 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:17.938895 19840 tablet_bootstrap.cc:492] T ca06f07bb98f45a7bf63aebc28348e08 P 78c2c46c7520492db35d51c244fb4f74: No bootstrap required, opened a new log
I20260130 02:58:17.939466 19840 ts_tablet_manager.cc:1403] T ca06f07bb98f45a7bf63aebc28348e08 P 78c2c46c7520492db35d51c244fb4f74: Time spent bootstrapping tablet: real 0.063s user 0.010s sys 0.000s
I20260130 02:58:17.941881 19840 raft_consensus.cc:359] T ca06f07bb98f45a7bf63aebc28348e08 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:17.942664 19840 raft_consensus.cc:385] T ca06f07bb98f45a7bf63aebc28348e08 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:17.944295 19840 raft_consensus.cc:740] T ca06f07bb98f45a7bf63aebc28348e08 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78c2c46c7520492db35d51c244fb4f74, State: Initialized, Role: FOLLOWER
I20260130 02:58:17.945012 19840 consensus_queue.cc:260] T ca06f07bb98f45a7bf63aebc28348e08 P 78c2c46c7520492db35d51c244fb4f74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:17.947981 19840 ts_tablet_manager.cc:1434] T ca06f07bb98f45a7bf63aebc28348e08 P 78c2c46c7520492db35d51c244fb4f74: Time spent starting tablet: real 0.008s user 0.004s sys 0.000s
I20260130 02:58:17.953990 19807 tablet_bootstrap.cc:492] T e12d130bcdf34965888753f12c1ff848 P 371a36d219164974abdf848841528152: Bootstrap starting.
I20260130 02:58:17.956220 19840 tablet_bootstrap.cc:492] T e34d5da0ad864b2eaea0d575c982e447 P 78c2c46c7520492db35d51c244fb4f74: Bootstrap starting.
I20260130 02:58:17.961540 20844 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3749baa287ed4f249329a74071781edd: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260130 02:58:17.965471 19840 tablet_bootstrap.cc:654] T e34d5da0ad864b2eaea0d575c982e447 P 78c2c46c7520492db35d51c244fb4f74: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:17.980276 19625 ts_tablet_manager.cc:1434] T 25fd48e693424537b91ba7c3fcc96878 P d870164263bb44b6b780a557daea3656: Time spent starting tablet: real 0.073s user 0.003s sys 0.001s
I20260130 02:58:17.987462 20844 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3749baa287ed4f249329a74071781edd: Bootstrap complete.
I20260130 02:58:17.988490 19222 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "49efcfced0b74c48962393149ac2a500" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152" is_pre_election: true
I20260130 02:58:17.972033 20704 raft_consensus.cc:493] T 2c79a3aecab844698e70834999e1caac P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:17.992853 20704 raft_consensus.cc:515] T 2c79a3aecab844698e70834999e1caac P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:17.994771 20704 leader_election.cc:290] T 2c79a3aecab844698e70834999e1caac P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d870164263bb44b6b780a557daea3656 (127.18.100.67:39571), 371a36d219164974abdf848841528152 (127.18.100.65:42021)
I20260130 02:58:17.986512 19360 tablet_service.cc:1505] Processing CreateTablet for tablet 00ddbfe6218c44d788fbf65245a31dc6 (DEFAULT_TABLE table=default.Table_9cb9a9f9e5d145a6a42d7ccbc42aced9 [id=5f1f179fadc247da9382dd7796dcf2c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:17.995463 19521 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2c79a3aecab844698e70834999e1caac" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
I20260130 02:58:17.996165 19360 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00ddbfe6218c44d788fbf65245a31dc6. 1 dirs total, 0 dirs full, 0 dirs failed
W20260130 02:58:17.996507 19313 leader_election.cc:343] T 2c79a3aecab844698e70834999e1caac P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:18.015928 20718 raft_consensus.cc:493] T f9e5f01e69254d658c369f3e21cef523 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:18.016389 19496 tablet_service.cc:1505] Processing CreateTablet for tablet 00ddbfe6218c44d788fbf65245a31dc6 (DEFAULT_TABLE table=default.Table_9cb9a9f9e5d145a6a42d7ccbc42aced9 [id=5f1f179fadc247da9382dd7796dcf2c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:18.016367 20718 raft_consensus.cc:515] T f9e5f01e69254d658c369f3e21cef523 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:18.019878 19374 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f9e5f01e69254d658c369f3e21cef523" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74" is_pre_election: true
I20260130 02:58:18.020540 19374 raft_consensus.cc:2468] T f9e5f01e69254d658c369f3e21cef523 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d870164263bb44b6b780a557daea3656 in term 0.
I20260130 02:58:18.019297 20718 leader_election.cc:290] T f9e5f01e69254d658c369f3e21cef523 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767)
I20260130 02:58:18.022665 19462 leader_election.cc:304] T f9e5f01e69254d658c369f3e21cef523 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78c2c46c7520492db35d51c244fb4f74, d870164263bb44b6b780a557daea3656; no voters:
I20260130 02:58:18.023955 19840 tablet_bootstrap.cc:492] T e34d5da0ad864b2eaea0d575c982e447 P 78c2c46c7520492db35d51c244fb4f74: No bootstrap required, opened a new log
I20260130 02:58:18.024410 19840 ts_tablet_manager.cc:1403] T e34d5da0ad864b2eaea0d575c982e447 P 78c2c46c7520492db35d51c244fb4f74: Time spent bootstrapping tablet: real 0.068s user 0.009s sys 0.002s
I20260130 02:58:18.025353 20718 raft_consensus.cc:2804] T f9e5f01e69254d658c369f3e21cef523 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260130 02:58:18.025686 20718 raft_consensus.cc:493] T f9e5f01e69254d658c369f3e21cef523 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260130 02:58:18.026494 20718 raft_consensus.cc:3060] T f9e5f01e69254d658c369f3e21cef523 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:18.026717 19840 raft_consensus.cc:359] T e34d5da0ad864b2eaea0d575c982e447 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:18.030972 19840 raft_consensus.cc:385] T e34d5da0ad864b2eaea0d575c982e447 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:18.031271 19840 raft_consensus.cc:740] T e34d5da0ad864b2eaea0d575c982e447 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78c2c46c7520492db35d51c244fb4f74, State: Initialized, Role: FOLLOWER
I20260130 02:58:18.033644 18866 catalog_manager.cc:5654] T 6e12a8eeb3204325a9a8b7779f72ab29 P d870164263bb44b6b780a557daea3656 reported cstate change: term changed from 0 to 1, leader changed from <none> to d870164263bb44b6b780a557daea3656 (127.18.100.67). New cstate: current_term: 1 leader_uuid: "d870164263bb44b6b780a557daea3656" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } health_report { overall_health: HEALTHY } } }
I20260130 02:58:18.031912 19840 consensus_queue.cc:260] T e34d5da0ad864b2eaea0d575c982e447 P 78c2c46c7520492db35d51c244fb4f74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:18.040704 18866 catalog_manager.cc:5746] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571) does not have the latest schema for tablet a8a5038e9def44eb8b3b1e4aeae25652 (table default.Table_dee034aec0b24982b8dc2a3fa1b326f1 [id=66033e75b3814a8584311cf7e4a07ad3]). Expected version 1 got 0
I20260130 02:58:18.042970 18866 catalog_manager.cc:5746] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571) does not have the latest schema for tablet 345eaaa5625c48d099280b7fbb2e514b (table default.Table_c1886dd9a7934e7797e0e7e0b065d692 [id=7a2d5367224b492b844199088b01b149]). Expected version 1 got 0
I20260130 02:58:18.036435 19840 ts_tablet_manager.cc:1434] T e34d5da0ad864b2eaea0d575c982e447 P 78c2c46c7520492db35d51c244fb4f74: Time spent starting tablet: real 0.012s user 0.000s sys 0.004s
I20260130 02:58:18.045157 19840 tablet_bootstrap.cc:492] T 2b0edaad6877460cb7982efc62b00b66 P 78c2c46c7520492db35d51c244fb4f74: Bootstrap starting.
I20260130 02:58:18.045722 19807 tablet_bootstrap.cc:654] T e12d130bcdf34965888753f12c1ff848 P 371a36d219164974abdf848841528152: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:18.048867 18866 catalog_manager.cc:5746] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571) does not have the latest schema for tablet b1e8b78eff0d4a0b92059adb2de4a3ce (table default.Table_d0b4516d24384e5386c1debaf2543adb [id=f1342f9497b74e7a9ecad1de8082578d]). Expected version 1 got 0
I20260130 02:58:18.054044 18866 catalog_manager.cc:5746] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571) does not have the latest schema for tablet 7caf7cbad5054396b5617094e4d536a0 (table default.Table_007e969f70d345a7bcb3dfbaabbc025d [id=77d4fa8705a3419cb0a84d3ca1bb4ce3]). Expected version 1 got 0
I20260130 02:58:18.054747 18866 catalog_manager.cc:5746] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571) does not have the latest schema for tablet 274530359b174ff79f046c2d4a6afa76 (table default.Table_c4f4904b6fac4b9880979cf0314e71c1 [id=c456c6418c93499a9d08f652720854ca]). Expected version 1 got 0
I20260130 02:58:18.055337 19496 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00ddbfe6218c44d788fbf65245a31dc6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:18.062408 18866 catalog_manager.cc:5746] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571) does not have the latest schema for tablet 16691c9315e94253bb9b3fb5058e5636 (table default.Table_8f311c5878744ef692b496ecfdfc3ede [id=4f499357303d464aafc17679ffcaaa07]). Expected version 1 got 0
I20260130 02:58:18.063575 19840 tablet_bootstrap.cc:654] T 2b0edaad6877460cb7982efc62b00b66 P 78c2c46c7520492db35d51c244fb4f74: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:18.064924 18866 catalog_manager.cc:5746] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571) does not have the latest schema for tablet 4c139aa8cd184f488cec4e05ef7dcada (table default.Table_e4d07634d84a4678b1c94374e3adc476 [id=fcbd7e3a7fc64d89aa740494d7e1a0f7]). Expected version 1 got 0
I20260130 02:58:18.065551 18866 catalog_manager.cc:5654] T 03025d13f0e545cfa6288677a2715343 P d870164263bb44b6b780a557daea3656 reported cstate change: term changed from 0 to 1, leader changed from <none> to d870164263bb44b6b780a557daea3656 (127.18.100.67). New cstate: current_term: 1 leader_uuid: "d870164263bb44b6b780a557daea3656" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } health_report { overall_health: UNKNOWN } } }
I20260130 02:58:18.068310 18866 catalog_manager.cc:5746] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571) does not have the latest schema for tablet 9fe0448bcc4a4346b3c67fc3f626ed3c (table default.Table_5c1c447e85524ac5a61ffd8a48509d36 [id=70aaf21ad9bd4d959f46939c6b4a4880]). Expected version 1 got 0
I20260130 02:58:18.069247 18866 catalog_manager.cc:5746] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571) does not have the latest schema for tablet 0e8018c094024c4788c484e8f5466e00 (table default.Table_f17d8e4d9f25449b84aa751af93936b2 [id=43254e466e7b4a2fb59fbcec68cfde36]). Expected version 1 got 0
I20260130 02:58:18.070526 18866 catalog_manager.cc:5654] T f41f4cff7dd4489f948700a45e9a7057 P d870164263bb44b6b780a557daea3656 reported cstate change: term changed from 0 to 1, leader changed from <none> to d870164263bb44b6b780a557daea3656 (127.18.100.67). New cstate: current_term: 1 leader_uuid: "d870164263bb44b6b780a557daea3656" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } health_report { overall_health: HEALTHY } } }
I20260130 02:58:18.085677 19807 tablet_bootstrap.cc:492] T e12d130bcdf34965888753f12c1ff848 P 371a36d219164974abdf848841528152: No bootstrap required, opened a new log
I20260130 02:58:18.078322 20844 raft_consensus.cc:359] T 00000000000000000000000000000000 P 3749baa287ed4f249329a74071781edd [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "658299197a424f53a37f14fadf089ae4" member_type: VOTER last_known_addr { host: "127.18.100.126" port: 38191 } } peers { permanent_uuid: "107f274cdda543038832f06a56ab9130" member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39711 } } peers { permanent_uuid: "3749baa287ed4f249329a74071781edd" member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34629 } }
I20260130 02:58:18.086087 19807 ts_tablet_manager.cc:1403] T e12d130bcdf34965888753f12c1ff848 P 371a36d219164974abdf848841528152: Time spent bootstrapping tablet: real 0.164s user 0.007s sys 0.006s
I20260130 02:58:18.086089 20844 raft_consensus.cc:740] T 00000000000000000000000000000000 P 3749baa287ed4f249329a74071781edd [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3749baa287ed4f249329a74071781edd, State: Initialized, Role: FOLLOWER
I20260130 02:58:18.088338 19807 raft_consensus.cc:359] T e12d130bcdf34965888753f12c1ff848 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:18.088922 19807 raft_consensus.cc:385] T e12d130bcdf34965888753f12c1ff848 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:18.089182 19807 raft_consensus.cc:740] T e12d130bcdf34965888753f12c1ff848 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 371a36d219164974abdf848841528152, State: Initialized, Role: FOLLOWER
I20260130 02:58:18.091153 20718 raft_consensus.cc:515] T f9e5f01e69254d658c369f3e21cef523 P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
W20260130 02:58:18.103123 19314 leader_election.cc:343] T 49efcfced0b74c48962393149ac2a500 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:18.103662 19314 leader_election.cc:304] T 49efcfced0b74c48962393149ac2a500 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78c2c46c7520492db35d51c244fb4f74; no voters: 371a36d219164974abdf848841528152, d870164263bb44b6b780a557daea3656
I20260130 02:58:18.104303 20704 raft_consensus.cc:2749] T 49efcfced0b74c48962393149ac2a500 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260130 02:58:18.106271 19840 tablet_bootstrap.cc:492] T 2b0edaad6877460cb7982efc62b00b66 P 78c2c46c7520492db35d51c244fb4f74: No bootstrap required, opened a new log
I20260130 02:58:18.106676 19840 ts_tablet_manager.cc:1403] T 2b0edaad6877460cb7982efc62b00b66 P 78c2c46c7520492db35d51c244fb4f74: Time spent bootstrapping tablet: real 0.062s user 0.008s sys 0.001s
I20260130 02:58:18.106753 19225 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f9e5f01e69254d658c369f3e21cef523" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152"
I20260130 02:58:18.107259 19225 raft_consensus.cc:3060] T f9e5f01e69254d658c369f3e21cef523 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:18.108722 19840 raft_consensus.cc:359] T 2b0edaad6877460cb7982efc62b00b66 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:18.109318 19840 raft_consensus.cc:385] T 2b0edaad6877460cb7982efc62b00b66 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:18.109546 19840 raft_consensus.cc:740] T 2b0edaad6877460cb7982efc62b00b66 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78c2c46c7520492db35d51c244fb4f74, State: Initialized, Role: FOLLOWER
I20260130 02:58:18.111006 19224 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f9e5f01e69254d658c369f3e21cef523" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152" is_pre_election: true
I20260130 02:58:18.110065 19840 consensus_queue.cc:260] T 2b0edaad6877460cb7982efc62b00b66 P 78c2c46c7520492db35d51c244fb4f74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:18.113426 19840 ts_tablet_manager.cc:1434] T 2b0edaad6877460cb7982efc62b00b66 P 78c2c46c7520492db35d51c244fb4f74: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20260130 02:58:18.113819 19225 raft_consensus.cc:2468] T f9e5f01e69254d658c369f3e21cef523 P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d870164263bb44b6b780a557daea3656 in term 1.
I20260130 02:58:18.114231 19840 tablet_bootstrap.cc:492] T ab6afa40408a4f458495abf20290e55b P 78c2c46c7520492db35d51c244fb4f74: Bootstrap starting.
I20260130 02:58:18.119266 19840 tablet_bootstrap.cc:654] T ab6afa40408a4f458495abf20290e55b P 78c2c46c7520492db35d51c244fb4f74: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:18.121853 20844 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3749baa287ed4f249329a74071781edd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "658299197a424f53a37f14fadf089ae4" member_type: VOTER last_known_addr { host: "127.18.100.126" port: 38191 } } peers { permanent_uuid: "107f274cdda543038832f06a56ab9130" member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39711 } } peers { permanent_uuid: "3749baa287ed4f249329a74071781edd" member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34629 } }
I20260130 02:58:18.122779 19625 tablet_bootstrap.cc:492] T 838e49e5e8874db393bdc15a1c92f87e P d870164263bb44b6b780a557daea3656: Bootstrap starting.
I20260130 02:58:18.122833 19807 consensus_queue.cc:260] T e12d130bcdf34965888753f12c1ff848 P 371a36d219164974abdf848841528152 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:18.124918 19807 ts_tablet_manager.cc:1434] T e12d130bcdf34965888753f12c1ff848 P 371a36d219164974abdf848841528152: Time spent starting tablet: real 0.039s user 0.001s sys 0.004s
I20260130 02:58:18.125768 19807 tablet_bootstrap.cc:492] T 3aa76993beb94b12bc706a76db26a03b P 371a36d219164974abdf848841528152: Bootstrap starting.
I20260130 02:58:18.128585 19625 tablet_bootstrap.cc:654] T 838e49e5e8874db393bdc15a1c92f87e P d870164263bb44b6b780a557daea3656: Neither blocks nor log segments found. Creating new log.
W20260130 02:58:18.135064 19159 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.100.66:40767, user_credentials={real_user=slave}} blocked reactor thread for 174153us
I20260130 02:58:18.135957 20866 consensus_queue.cc:1048] T e2aaaf00f53d418b87d5c7af286e4d84 P 371a36d219164974abdf848841528152 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260130 02:58:18.137930 19202 tablet_service.cc:1505] Processing CreateTablet for tablet 00ddbfe6218c44d788fbf65245a31dc6 (DEFAULT_TABLE table=default.Table_9cb9a9f9e5d145a6a42d7ccbc42aced9 [id=5f1f179fadc247da9382dd7796dcf2c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:18.142155 19202 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00ddbfe6218c44d788fbf65245a31dc6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:18.145378 19221 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2c79a3aecab844698e70834999e1caac" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152" is_pre_election: true
I20260130 02:58:18.148443 20867 consensus_queue.cc:1048] T cf1b66a20fce4c4d96bc0bc32a7bafa8 P 371a36d219164974abdf848841528152 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260130 02:58:18.150601 20704 raft_consensus.cc:493] T dfbbe9045e7c4aa5b520218687e7e4c1 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:18.151010 20704 raft_consensus.cc:515] T dfbbe9045e7c4aa5b520218687e7e4c1 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:18.152801 20704 leader_election.cc:290] T dfbbe9045e7c4aa5b520218687e7e4c1 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d870164263bb44b6b780a557daea3656 (127.18.100.67:39571), 371a36d219164974abdf848841528152 (127.18.100.65:42021)
I20260130 02:58:18.155396 20872 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3749baa287ed4f249329a74071781edd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "658299197a424f53a37f14fadf089ae4" member_type: VOTER last_known_addr { host: "127.18.100.126" port: 38191 } } peers { permanent_uuid: "107f274cdda543038832f06a56ab9130" member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39711 } } peers { permanent_uuid: "3749baa287ed4f249329a74071781edd" member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34629 } } }
I20260130 02:58:18.160962 20844 sys_catalog.cc:565] T 00000000000000000000000000000000 P 3749baa287ed4f249329a74071781edd [sys.catalog]: configured and running, proceeding with master startup.
I20260130 02:58:18.156206 20872 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3749baa287ed4f249329a74071781edd [sys.catalog]: This master's current role is: FOLLOWER
I20260130 02:58:18.164561 19840 tablet_bootstrap.cc:492] T ab6afa40408a4f458495abf20290e55b P 78c2c46c7520492db35d51c244fb4f74: No bootstrap required, opened a new log
I20260130 02:58:18.164971 19840 ts_tablet_manager.cc:1403] T ab6afa40408a4f458495abf20290e55b P 78c2c46c7520492db35d51c244fb4f74: Time spent bootstrapping tablet: real 0.051s user 0.011s sys 0.002s
I20260130 02:58:18.167112 19840 raft_consensus.cc:359] T ab6afa40408a4f458495abf20290e55b P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:18.167711 19840 raft_consensus.cc:385] T ab6afa40408a4f458495abf20290e55b P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:18.167935 19840 raft_consensus.cc:740] T ab6afa40408a4f458495abf20290e55b P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78c2c46c7520492db35d51c244fb4f74, State: Initialized, Role: FOLLOWER
W20260130 02:58:18.168617 19314 leader_election.cc:343] T 2c79a3aecab844698e70834999e1caac P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:18.168447 19840 consensus_queue.cc:260] T ab6afa40408a4f458495abf20290e55b P 78c2c46c7520492db35d51c244fb4f74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:18.168978 19314 leader_election.cc:304] T 2c79a3aecab844698e70834999e1caac P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78c2c46c7520492db35d51c244fb4f74; no voters: 371a36d219164974abdf848841528152, d870164263bb44b6b780a557daea3656
I20260130 02:58:18.170056 20697 raft_consensus.cc:2749] T 2c79a3aecab844698e70834999e1caac P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260130 02:58:18.172319 19224 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dfbbe9045e7c4aa5b520218687e7e4c1" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152" is_pre_election: true
I20260130 02:58:18.175382 20704 raft_consensus.cc:493] T 657e9b0d8c05413f904129dce745710b P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:18.175824 20704 raft_consensus.cc:515] T 657e9b0d8c05413f904129dce745710b P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:18.177541 20704 leader_election.cc:290] T 657e9b0d8c05413f904129dce745710b P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), d870164263bb44b6b780a557daea3656 (127.18.100.67:39571)
I20260130 02:58:18.179911 19807 tablet_bootstrap.cc:654] T 3aa76993beb94b12bc706a76db26a03b P 371a36d219164974abdf848841528152: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:18.170471 19840 ts_tablet_manager.cc:1434] T ab6afa40408a4f458495abf20290e55b P 78c2c46c7520492db35d51c244fb4f74: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260130 02:58:18.181402 19840 tablet_bootstrap.cc:492] T eb7a755ab62b4552bb8588264c5f7b75 P 78c2c46c7520492db35d51c244fb4f74: Bootstrap starting.
W20260130 02:58:18.182376 19314 leader_election.cc:343] T dfbbe9045e7c4aa5b520218687e7e4c1 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:18.183856 19224 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "657e9b0d8c05413f904129dce745710b" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152" is_pre_election: true
I20260130 02:58:18.188103 19378 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f9e5f01e69254d658c369f3e21cef523" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74" is_pre_election: true
I20260130 02:58:18.188604 19378 raft_consensus.cc:2468] T f9e5f01e69254d658c369f3e21cef523 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 371a36d219164974abdf848841528152 in term 0.
I20260130 02:58:18.188906 18869 catalog_manager.cc:5746] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767) does not have the latest schema for tablet a8a5038e9def44eb8b3b1e4aeae25652 (table default.Table_dee034aec0b24982b8dc2a3fa1b326f1 [id=66033e75b3814a8584311cf7e4a07ad3]). Expected version 1 got 0
I20260130 02:58:18.190081 19378 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "25fd48e693424537b91ba7c3fcc96878" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74" is_pre_election: true
I20260130 02:58:18.190532 19378 raft_consensus.cc:2468] T 25fd48e693424537b91ba7c3fcc96878 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 371a36d219164974abdf848841528152 in term 0.
I20260130 02:58:18.195207 19840 tablet_bootstrap.cc:654] T eb7a755ab62b4552bb8588264c5f7b75 P 78c2c46c7520492db35d51c244fb4f74: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:18.199391 18869 catalog_manager.cc:5746] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767) does not have the latest schema for tablet 345eaaa5625c48d099280b7fbb2e514b (table default.Table_c1886dd9a7934e7797e0e7e0b065d692 [id=7a2d5367224b492b844199088b01b149]). Expected version 1 got 0
I20260130 02:58:18.141891 19463 leader_election.cc:304] T f9e5f01e69254d658c369f3e21cef523 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 371a36d219164974abdf848841528152, d870164263bb44b6b780a557daea3656; no voters:
W20260130 02:58:18.204239 19463 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.100.65:42021, user_credentials={real_user=slave}} blocked reactor thread for 62415.5us
I20260130 02:58:18.205009 18869 catalog_manager.cc:5746] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767) does not have the latest schema for tablet b1e8b78eff0d4a0b92059adb2de4a3ce (table default.Table_d0b4516d24384e5386c1debaf2543adb [id=f1342f9497b74e7a9ecad1de8082578d]). Expected version 1 got 0
I20260130 02:58:18.155419 19521 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dfbbe9045e7c4aa5b520218687e7e4c1" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
I20260130 02:58:18.207741 20704 raft_consensus.cc:493] T ca06f07bb98f45a7bf63aebc28348e08 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:18.208173 20704 raft_consensus.cc:515] T ca06f07bb98f45a7bf63aebc28348e08 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:18.209053 19528 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f9e5f01e69254d658c369f3e21cef523" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
I20260130 02:58:18.209672 19528 raft_consensus.cc:2393] T f9e5f01e69254d658c369f3e21cef523 P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 371a36d219164974abdf848841528152 in current term 1: Already voted for candidate d870164263bb44b6b780a557daea3656 in this term.
I20260130 02:58:18.209918 20704 leader_election.cc:290] T ca06f07bb98f45a7bf63aebc28348e08 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d870164263bb44b6b780a557daea3656 (127.18.100.67:39571), 371a36d219164974abdf848841528152 (127.18.100.65:42021)
I20260130 02:58:18.210285 19526 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "25fd48e693424537b91ba7c3fcc96878" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
I20260130 02:58:18.210800 19526 raft_consensus.cc:2468] T 25fd48e693424537b91ba7c3fcc96878 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 371a36d219164974abdf848841528152 in term 0.
I20260130 02:58:18.211640 18869 catalog_manager.cc:5746] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767) does not have the latest schema for tablet 16691c9315e94253bb9b3fb5058e5636 (table default.Table_8f311c5878744ef692b496ecfdfc3ede [id=4f499357303d464aafc17679ffcaaa07]). Expected version 1 got 0
I20260130 02:58:18.180032 19522 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "657e9b0d8c05413f904129dce745710b" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
I20260130 02:58:18.213517 18869 catalog_manager.cc:5746] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767) does not have the latest schema for tablet 9fe0448bcc4a4346b3c67fc3f626ed3c (table default.Table_5c1c447e85524ac5a61ffd8a48509d36 [id=70aaf21ad9bd4d959f46939c6b4a4880]). Expected version 1 got 0
I20260130 02:58:18.214035 18869 catalog_manager.cc:5746] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767) does not have the latest schema for tablet 25fd48e693424537b91ba7c3fcc96878 (table default.Table_ebc8b8379c274c18b0a06a919dbe0ee0 [id=28156e0843674fce88f5914a3e4195ae]). Expected version 1 got 0
I20260130 02:58:18.214565 19522 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ca06f07bb98f45a7bf63aebc28348e08" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
I20260130 02:58:18.215745 20864 raft_consensus.cc:2804] T f9e5f01e69254d658c369f3e21cef523 P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Leader election won for term 1
I20260130 02:58:18.216163 20864 raft_consensus.cc:697] T f9e5f01e69254d658c369f3e21cef523 P d870164263bb44b6b780a557daea3656 [term 1 LEADER]: Becoming Leader. State: Replica: d870164263bb44b6b780a557daea3656, State: Running, Role: LEADER
I20260130 02:58:18.216856 20864 consensus_queue.cc:237] T f9e5f01e69254d658c369f3e21cef523 P d870164263bb44b6b780a557daea3656 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:18.219551 18869 catalog_manager.cc:5746] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767) does not have the latest schema for tablet 4c139aa8cd184f488cec4e05ef7dcada (table default.Table_e4d07634d84a4678b1c94374e3adc476 [id=fcbd7e3a7fc64d89aa740494d7e1a0f7]). Expected version 1 got 0
I20260130 02:58:18.220664 18869 catalog_manager.cc:5746] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767) does not have the latest schema for tablet 274530359b174ff79f046c2d4a6afa76 (table default.Table_c4f4904b6fac4b9880979cf0314e71c1 [id=c456c6418c93499a9d08f652720854ca]). Expected version 1 got 0
I20260130 02:58:18.221331 18869 catalog_manager.cc:5746] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767) does not have the latest schema for tablet 7caf7cbad5054396b5617094e4d536a0 (table default.Table_007e969f70d345a7bcb3dfbaabbc025d [id=77d4fa8705a3419cb0a84d3ca1bb4ce3]). Expected version 1 got 0
I20260130 02:58:18.225857 20704 raft_consensus.cc:493] T ab6afa40408a4f458495abf20290e55b P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:18.226426 18869 catalog_manager.cc:5746] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767) does not have the latest schema for tablet 838e49e5e8874db393bdc15a1c92f87e (table default.Table_da3d581b15a94840898777f74611994e [id=b7e241ced53e4160ac6fa1f65ef3674e]). Expected version 1 got 0
I20260130 02:58:18.226303 20704 raft_consensus.cc:515] T ab6afa40408a4f458495abf20290e55b P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:18.228063 20704 leader_election.cc:290] T ab6afa40408a4f458495abf20290e55b P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d870164263bb44b6b780a557daea3656 (127.18.100.67:39571), 371a36d219164974abdf848841528152 (127.18.100.65:42021)
W20260130 02:58:18.230538 19313 leader_election.cc:343] T dfbbe9045e7c4aa5b520218687e7e4c1 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:18.230952 19313 leader_election.cc:304] T dfbbe9045e7c4aa5b520218687e7e4c1 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78c2c46c7520492db35d51c244fb4f74; no voters: 371a36d219164974abdf848841528152, d870164263bb44b6b780a557daea3656
I20260130 02:58:18.233134 20718 leader_election.cc:290] T f9e5f01e69254d658c369f3e21cef523 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 election: Requested vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767)
I20260130 02:58:18.234565 20704 raft_consensus.cc:2749] T dfbbe9045e7c4aa5b520218687e7e4c1 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260130 02:58:18.235805 19313 leader_election.cc:343] T 657e9b0d8c05413f904129dce745710b P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): Illegal state: must be running to vote when last-logged opid is not known
W20260130 02:58:18.236570 19313 leader_election.cc:343] T ca06f07bb98f45a7bf63aebc28348e08 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:18.237990 19522 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ab6afa40408a4f458495abf20290e55b" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
I20260130 02:58:18.243983 19374 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f9e5f01e69254d658c369f3e21cef523" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74"
I20260130 02:58:18.244505 19374 raft_consensus.cc:3060] T f9e5f01e69254d658c369f3e21cef523 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Advancing to term 1
W20260130 02:58:18.244958 19313 leader_election.cc:343] T ab6afa40408a4f458495abf20290e55b P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): Illegal state: must be running to vote when last-logged opid is not known
W20260130 02:58:18.249397 19314 leader_election.cc:343] T 657e9b0d8c05413f904129dce745710b P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:18.249790 19314 leader_election.cc:304] T 657e9b0d8c05413f904129dce745710b P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78c2c46c7520492db35d51c244fb4f74; no voters: 371a36d219164974abdf848841528152, d870164263bb44b6b780a557daea3656
I20260130 02:58:18.251223 19374 raft_consensus.cc:2468] T f9e5f01e69254d658c369f3e21cef523 P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d870164263bb44b6b780a557daea3656 in term 1.
I20260130 02:58:18.259362 19224 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ca06f07bb98f45a7bf63aebc28348e08" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152" is_pre_election: true
I20260130 02:58:18.261183 19222 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ab6afa40408a4f458495abf20290e55b" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152" is_pre_election: true
I20260130 02:58:18.267457 20866 raft_consensus.cc:493] T 838e49e5e8874db393bdc15a1c92f87e P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:18.267920 20866 raft_consensus.cc:515] T 838e49e5e8874db393bdc15a1c92f87e P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:18.269893 20866 leader_election.cc:290] T 838e49e5e8874db393bdc15a1c92f87e P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767), d870164263bb44b6b780a557daea3656 (127.18.100.67:39571)
W20260130 02:58:18.270970 19314 leader_election.cc:343] T ca06f07bb98f45a7bf63aebc28348e08 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:18.271373 19314 leader_election.cc:304] T ca06f07bb98f45a7bf63aebc28348e08 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78c2c46c7520492db35d51c244fb4f74; no voters: 371a36d219164974abdf848841528152, d870164263bb44b6b780a557daea3656
I20260130 02:58:18.272567 20697 raft_consensus.cc:2749] T 657e9b0d8c05413f904129dce745710b P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260130 02:58:18.273087 20697 raft_consensus.cc:2749] T ca06f07bb98f45a7bf63aebc28348e08 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260130 02:58:18.275552 18873 catalog_manager.cc:5746] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021) does not have the latest schema for tablet a8a5038e9def44eb8b3b1e4aeae25652 (table default.Table_dee034aec0b24982b8dc2a3fa1b326f1 [id=66033e75b3814a8584311cf7e4a07ad3]). Expected version 1 got 0
I20260130 02:58:18.276227 18873 catalog_manager.cc:5746] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021) does not have the latest schema for tablet 345eaaa5625c48d099280b7fbb2e514b (table default.Table_c1886dd9a7934e7797e0e7e0b065d692 [id=7a2d5367224b492b844199088b01b149]). Expected version 1 got 0
W20260130 02:58:18.283167 19314 leader_election.cc:343] T ab6afa40408a4f458495abf20290e55b P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:18.283666 19840 tablet_bootstrap.cc:492] T eb7a755ab62b4552bb8588264c5f7b75 P 78c2c46c7520492db35d51c244fb4f74: No bootstrap required, opened a new log
I20260130 02:58:18.284082 19840 ts_tablet_manager.cc:1403] T eb7a755ab62b4552bb8588264c5f7b75 P 78c2c46c7520492db35d51c244fb4f74: Time spent bootstrapping tablet: real 0.103s user 0.018s sys 0.010s
I20260130 02:58:18.283550 19314 leader_election.cc:304] T ab6afa40408a4f458495abf20290e55b P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78c2c46c7520492db35d51c244fb4f74; no voters: 371a36d219164974abdf848841528152, d870164263bb44b6b780a557daea3656
I20260130 02:58:18.284746 20697 raft_consensus.cc:2749] T ab6afa40408a4f458495abf20290e55b P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260130 02:58:18.285284 18873 catalog_manager.cc:5654] T cf1b66a20fce4c4d96bc0bc32a7bafa8 P 371a36d219164974abdf848841528152 reported cstate change: term changed from 0 to 1, leader changed from <none> to 371a36d219164974abdf848841528152 (127.18.100.65). New cstate: current_term: 1 leader_uuid: "371a36d219164974abdf848841528152" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } health_report { overall_health: UNKNOWN } } }
I20260130 02:58:18.286929 18873 catalog_manager.cc:5746] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021) does not have the latest schema for tablet b1e8b78eff0d4a0b92059adb2de4a3ce (table default.Table_d0b4516d24384e5386c1debaf2543adb [id=f1342f9497b74e7a9ecad1de8082578d]). Expected version 1 got 0
I20260130 02:58:18.288925 18873 catalog_manager.cc:5746] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021) does not have the latest schema for tablet 7caf7cbad5054396b5617094e4d536a0 (table default.Table_007e969f70d345a7bcb3dfbaabbc025d [id=77d4fa8705a3419cb0a84d3ca1bb4ce3]). Expected version 1 got 0
I20260130 02:58:18.289570 18873 catalog_manager.cc:5654] T 274530359b174ff79f046c2d4a6afa76 P 371a36d219164974abdf848841528152 reported cstate change: term changed from 0 to 1, leader changed from <none> to 371a36d219164974abdf848841528152 (127.18.100.65). New cstate: current_term: 1 leader_uuid: "371a36d219164974abdf848841528152" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } health_report { overall_health: UNKNOWN } } }
W20260130 02:58:18.299248 19313 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.100.67:39571, user_credentials={real_user=slave}} blocked reactor thread for 53642.9us
I20260130 02:58:18.286298 19840 raft_consensus.cc:359] T eb7a755ab62b4552bb8588264c5f7b75 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:18.300005 18873 catalog_manager.cc:5746] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021) does not have the latest schema for tablet 274530359b174ff79f046c2d4a6afa76 (table default.Table_c4f4904b6fac4b9880979cf0314e71c1 [id=c456c6418c93499a9d08f652720854ca]). Expected version 1 got 0
I20260130 02:58:18.300307 19840 raft_consensus.cc:385] T eb7a755ab62b4552bb8588264c5f7b75 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:18.300351 19807 tablet_bootstrap.cc:492] T 3aa76993beb94b12bc706a76db26a03b P 371a36d219164974abdf848841528152: No bootstrap required, opened a new log
I20260130 02:58:18.300590 19840 raft_consensus.cc:740] T eb7a755ab62b4552bb8588264c5f7b75 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78c2c46c7520492db35d51c244fb4f74, State: Initialized, Role: FOLLOWER
I20260130 02:58:18.300763 19807 ts_tablet_manager.cc:1403] T 3aa76993beb94b12bc706a76db26a03b P 371a36d219164974abdf848841528152: Time spent bootstrapping tablet: real 0.175s user 0.008s sys 0.000s
I20260130 02:58:18.301141 19840 consensus_queue.cc:260] T eb7a755ab62b4552bb8588264c5f7b75 P 78c2c46c7520492db35d51c244fb4f74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:18.303077 19840 ts_tablet_manager.cc:1434] T eb7a755ab62b4552bb8588264c5f7b75 P 78c2c46c7520492db35d51c244fb4f74: Time spent starting tablet: real 0.019s user 0.004s sys 0.003s
I20260130 02:58:18.303880 19840 tablet_bootstrap.cc:492] T 0855d88806914a36b4bf6b49036f0ad0 P 78c2c46c7520492db35d51c244fb4f74: Bootstrap starting.
I20260130 02:58:18.303387 18873 catalog_manager.cc:5654] T 5fbe92a2227d44d88b4531a7f8aa0e08 P 371a36d219164974abdf848841528152 reported cstate change: term changed from 0 to 1, leader changed from <none> to 371a36d219164974abdf848841528152 (127.18.100.65). New cstate: current_term: 1 leader_uuid: "371a36d219164974abdf848841528152" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } health_report { overall_health: UNKNOWN } } }
I20260130 02:58:18.305981 18873 catalog_manager.cc:5746] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021) does not have the latest schema for tablet 4c139aa8cd184f488cec4e05ef7dcada (table default.Table_e4d07634d84a4678b1c94374e3adc476 [id=fcbd7e3a7fc64d89aa740494d7e1a0f7]). Expected version 1 got 0
I20260130 02:58:18.303092 19807 raft_consensus.cc:359] T 3aa76993beb94b12bc706a76db26a03b P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:18.306864 18873 catalog_manager.cc:5654] T a90712651a604b83ba187f893ab17711 P 371a36d219164974abdf848841528152 reported cstate change: term changed from 0 to 1, leader changed from <none> to 371a36d219164974abdf848841528152 (127.18.100.65). New cstate: current_term: 1 leader_uuid: "371a36d219164974abdf848841528152" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } health_report { overall_health: UNKNOWN } } }
I20260130 02:58:18.316071 18873 catalog_manager.cc:5746] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021) does not have the latest schema for tablet 9fe0448bcc4a4346b3c67fc3f626ed3c (table default.Table_5c1c447e85524ac5a61ffd8a48509d36 [id=70aaf21ad9bd4d959f46939c6b4a4880]). Expected version 1 got 0
I20260130 02:58:18.321867 19840 tablet_bootstrap.cc:654] T 0855d88806914a36b4bf6b49036f0ad0 P 78c2c46c7520492db35d51c244fb4f74: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:18.326722 19807 raft_consensus.cc:385] T 3aa76993beb94b12bc706a76db26a03b P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:18.327128 19807 raft_consensus.cc:740] T 3aa76993beb94b12bc706a76db26a03b P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 371a36d219164974abdf848841528152, State: Initialized, Role: FOLLOWER
I20260130 02:58:18.328673 19807 consensus_queue.cc:260] T 3aa76993beb94b12bc706a76db26a03b P 371a36d219164974abdf848841528152 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:18.330611 19807 ts_tablet_manager.cc:1434] T 3aa76993beb94b12bc706a76db26a03b P 371a36d219164974abdf848841528152: Time spent starting tablet: real 0.030s user 0.004s sys 0.001s
I20260130 02:58:18.331475 19807 tablet_bootstrap.cc:492] T 13f0ace4d9de43abab08eca32cdb30c1 P 371a36d219164974abdf848841528152: Bootstrap starting.
I20260130 02:58:18.331944 19625 tablet_bootstrap.cc:492] T 838e49e5e8874db393bdc15a1c92f87e P d870164263bb44b6b780a557daea3656: No bootstrap required, opened a new log
I20260130 02:58:18.332333 19625 ts_tablet_manager.cc:1403] T 838e49e5e8874db393bdc15a1c92f87e P d870164263bb44b6b780a557daea3656: Time spent bootstrapping tablet: real 0.313s user 0.008s sys 0.001s
I20260130 02:58:18.335749 19840 tablet_bootstrap.cc:492] T 0855d88806914a36b4bf6b49036f0ad0 P 78c2c46c7520492db35d51c244fb4f74: No bootstrap required, opened a new log
I20260130 02:58:18.336126 19840 ts_tablet_manager.cc:1403] T 0855d88806914a36b4bf6b49036f0ad0 P 78c2c46c7520492db35d51c244fb4f74: Time spent bootstrapping tablet: real 0.032s user 0.004s sys 0.004s
I20260130 02:58:18.340304 18875 catalog_manager.cc:6199] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4: default.Table_1cbb7ef2df504c1083290b333440c9f7 [id=1515db389f284e2f9ed1a3cd0061d0f1] alter complete (version 6)
W20260130 02:58:18.320199 19159 consensus_peers.cc:597] T cf1b66a20fce4c4d96bc0bc32a7bafa8 P 371a36d219164974abdf848841528152 -> Peer d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): Couldn't send request to peer d870164263bb44b6b780a557daea3656. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20260130 02:58:18.363772 18874 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:50284 (request call id 247) took 2437 ms. Trace:
I20260130 02:58:18.366511 19625 raft_consensus.cc:359] T 838e49e5e8874db393bdc15a1c92f87e P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:18.366816 18874 rpcz_store.cc:276] 0130 02:58:15.926441 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:58:17.080746 (+1154305us) service_pool.cc:225] Handling call
0130 02:58:17.107432 (+ 26686us) catalog_manager.cc:2393] Looking up, locking, and authorizing table
0130 02:58:18.344610 (+1237178us) catalog_manager.cc:3708] Apply alter schema
0130 02:58:18.345147 (+ 537us) catalog_manager.cc:3740] Acquired catalog manager lock
0130 02:58:18.345348 (+ 201us) catalog_manager.cc:3898] Updating metadata on disk
0130 02:58:18.362172 (+ 16824us) catalog_manager.cc:3929] Committing alterations to in-memory state
0130 02:58:18.363715 (+ 1543us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0130 02:58:18.346642 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0130 02:58:18.347054 (+ 412us) write_op.cc:432] Acquiring schema lock in shared mode
0130 02:58:18.347079 (+ 25us) write_op.cc:435] Acquired schema lock
0130 02:58:18.347095 (+ 16us) tablet.cc:660] Decoding operations
0130 02:58:18.347376 (+ 281us) write_op.cc:620] Acquiring the partition lock for write op
0130 02:58:18.347428 (+ 52us) write_op.cc:641] Partition lock acquired for write op
0130 02:58:18.347455 (+ 27us) tablet.cc:683] Acquiring locks for 1 operations
0130 02:58:18.347607 (+ 152us) tablet.cc:699] Row locks acquired
0130 02:58:18.347624 (+ 17us) write_op.cc:260] PREPARE: finished
0130 02:58:18.347735 (+ 111us) write_op.cc:270] Start()
0130 02:58:18.347820 (+ 85us) write_op.cc:276] Timestamp: P: 1769741898347688 usec, L: 0
0130 02:58:18.347838 (+ 18us) op_driver.cc:348] REPLICATION: starting
0130 02:58:18.348349 (+ 511us) log.cc:844] Serialized 405 byte log entry
0130 02:58:18.356064 (+ 7715us) op_driver.cc:464] REPLICATION: finished
0130 02:58:18.356373 (+ 309us) write_op.cc:301] APPLY: starting
0130 02:58:18.356438 (+ 65us) tablet.cc:1365] starting BulkCheckPresence
0130 02:58:18.356519 (+ 81us) tablet.cc:1368] finished BulkCheckPresence
0130 02:58:18.356604 (+ 85us) tablet.cc:1370] starting ApplyRowOperation cycle
0130 02:58:18.356752 (+ 148us) tablet.cc:1381] finished ApplyRowOperation cycle
0130 02:58:18.356877 (+ 125us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0130 02:58:18.356919 (+ 42us) write_op.cc:312] APPLY: finished
0130 02:58:18.357079 (+ 160us) log.cc:844] Serialized 23 byte log entry
0130 02:58:18.357232 (+ 153us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:58:18.357292 (+ 60us) write_op.cc:454] Released schema lock
0130 02:58:18.357331 (+ 39us) write_op.cc:341] FINISH: Updating metrics
0130 02:58:18.357555 (+ 224us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:58:18.357580 (+ 25us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":175,"apply.run_cpu_time_us":1138,"apply.run_wall_time_us":1138,"num_ops":1,"prepare.queue_time_us":115,"prepare.run_cpu_time_us":2012,"prepare.run_wall_time_us":2012,"raft.queue_time_us":1059,"raft.run_cpu_time_us":592,"raft.run_wall_time_us":593,"replication_time_us":8137}]]}
W20260130 02:58:18.371053 18874 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.18.100.66:59691 (request call id 35) took 1239 ms (client timeout 1000 ms). Trace:
W20260130 02:58:18.371438 18874 rpcz_store.cc:269] 0130 02:58:17.131591 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:58:18.370752 (+1239161us) service_pool.cc:212] Skipping call since client already timed out
0130 02:58:18.370991 (+ 239us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20260130 02:58:18.372231 19625 raft_consensus.cc:385] T 838e49e5e8874db393bdc15a1c92f87e P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:18.372511 19625 raft_consensus.cc:740] T 838e49e5e8874db393bdc15a1c92f87e P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d870164263bb44b6b780a557daea3656, State: Initialized, Role: FOLLOWER
I20260130 02:58:18.379025 19840 raft_consensus.cc:359] T 0855d88806914a36b4bf6b49036f0ad0 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:18.379647 18868 catalog_manager.cc:5941] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4: Sending DeleteTablet for 3 replicas of tablet c7c621b99deb457998c44ab1644d3939
I20260130 02:58:18.379760 19840 raft_consensus.cc:385] T 0855d88806914a36b4bf6b49036f0ad0 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:18.379992 19840 raft_consensus.cc:740] T 0855d88806914a36b4bf6b49036f0ad0 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78c2c46c7520492db35d51c244fb4f74, State: Initialized, Role: FOLLOWER
I20260130 02:58:18.380573 19840 consensus_queue.cc:260] T 0855d88806914a36b4bf6b49036f0ad0 P 78c2c46c7520492db35d51c244fb4f74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:18.393751 19202 tablet_service.cc:1552] Processing DeleteTablet for tablet c7c621b99deb457998c44ab1644d3939 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:58:16 UTC) from {username='slave'} at 127.0.0.1:42870
I20260130 02:58:18.394179 19625 consensus_queue.cc:260] T 838e49e5e8874db393bdc15a1c92f87e P d870164263bb44b6b780a557daea3656 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:18.391062 20704 raft_consensus.cc:493] T d480bf18140245e3870a62582d5a8318 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:18.396662 19573 heartbeater.cc:344] Connected to a master server at 127.18.100.124:34629
I20260130 02:58:18.403755 19360 tablet_service.cc:1552] Processing DeleteTablet for tablet c7c621b99deb457998c44ab1644d3939 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:58:16 UTC) from {username='slave'} at 127.0.0.1:60110
W20260130 02:58:18.405342 20890 log.cc:927] Time spent T cf1b66a20fce4c4d96bc0bc32a7bafa8 P 78c2c46c7520492db35d51c244fb4f74: Append to log took a long time: real 0.076s user 0.001s sys 0.000s
I20260130 02:58:18.396528 20704 raft_consensus.cc:515] T d480bf18140245e3870a62582d5a8318 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } }
I20260130 02:58:18.407196 20704 leader_election.cc:290] T d480bf18140245e3870a62582d5a8318 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d870164263bb44b6b780a557daea3656 (127.18.100.67:39571), 371a36d219164974abdf848841528152 (127.18.100.65:42021)
I20260130 02:58:18.411670 19221 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d480bf18140245e3870a62582d5a8318" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152" is_pre_election: true
W20260130 02:58:18.414460 19314 leader_election.cc:343] T d480bf18140245e3870a62582d5a8318 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:18.411137 19625 ts_tablet_manager.cc:1434] T 838e49e5e8874db393bdc15a1c92f87e P d870164263bb44b6b780a557daea3656: Time spent starting tablet: real 0.079s user 0.007s sys 0.000s
I20260130 02:58:18.416030 19625 tablet_bootstrap.cc:492] T 3aa76993beb94b12bc706a76db26a03b P d870164263bb44b6b780a557daea3656: Bootstrap starting.
W20260130 02:58:18.415984 19313 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.100.67:39571, user_credentials={real_user=slave}} blocked reactor thread for 73918.3us
I20260130 02:58:18.416025 19840 ts_tablet_manager.cc:1434] T 0855d88806914a36b4bf6b49036f0ad0 P 78c2c46c7520492db35d51c244fb4f74: Time spent starting tablet: real 0.078s user 0.015s sys 0.001s
I20260130 02:58:18.398491 19159 leader_election.cc:304] T 25fd48e693424537b91ba7c3fcc96878 P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 371a36d219164974abdf848841528152, d870164263bb44b6b780a557daea3656; no voters:
I20260130 02:58:18.417058 19840 tablet_bootstrap.cc:492] T 1fc8f09d6329407c8078542b4123ef4d P 78c2c46c7520492db35d51c244fb4f74: Bootstrap starting.
I20260130 02:58:18.417594 19522 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d480bf18140245e3870a62582d5a8318" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
I20260130 02:58:18.420501 20697 raft_consensus.cc:493] T e34d5da0ad864b2eaea0d575c982e447 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:18.421121 20864 raft_consensus.cc:493] T cf1b66a20fce4c4d96bc0bc32a7bafa8 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:18.420948 20697 raft_consensus.cc:515] T e34d5da0ad864b2eaea0d575c982e447 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:18.421526 20864 raft_consensus.cc:515] T cf1b66a20fce4c4d96bc0bc32a7bafa8 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:18.422971 19840 tablet_bootstrap.cc:654] T 1fc8f09d6329407c8078542b4123ef4d P 78c2c46c7520492db35d51c244fb4f74: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:18.423410 20864 leader_election.cc:290] T cf1b66a20fce4c4d96bc0bc32a7bafa8 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767)
I20260130 02:58:18.423558 20608 raft_consensus.cc:2804] T 25fd48e693424537b91ba7c3fcc96878 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260130 02:58:18.423923 20608 raft_consensus.cc:493] T 25fd48e693424537b91ba7c3fcc96878 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260130 02:58:18.424219 20608 raft_consensus.cc:3060] T 25fd48e693424537b91ba7c3fcc96878 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:18.424984 20697 leader_election.cc:290] T e34d5da0ad864b2eaea0d575c982e447 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d870164263bb44b6b780a557daea3656 (127.18.100.67:39571), 371a36d219164974abdf848841528152 (127.18.100.65:42021)
I20260130 02:58:18.423084 20867 consensus_queue.cc:1048] T dcba6b534f9d406dad4fb502be40921c P 371a36d219164974abdf848841528152 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.008s
I20260130 02:58:18.431092 19221 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cf1b66a20fce4c4d96bc0bc32a7bafa8" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152" is_pre_election: true
I20260130 02:58:18.432410 19522 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e34d5da0ad864b2eaea0d575c982e447" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
I20260130 02:58:18.432740 20608 raft_consensus.cc:515] T 25fd48e693424537b91ba7c3fcc96878 P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
W20260130 02:58:18.433563 19313 leader_election.cc:343] T d480bf18140245e3870a62582d5a8318 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:18.433943 19313 leader_election.cc:304] T d480bf18140245e3870a62582d5a8318 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78c2c46c7520492db35d51c244fb4f74; no voters: 371a36d219164974abdf848841528152, d870164263bb44b6b780a557daea3656
I20260130 02:58:18.434006 19379 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cf1b66a20fce4c4d96bc0bc32a7bafa8" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74" is_pre_election: true
I20260130 02:58:18.441910 20697 raft_consensus.cc:2749] T d480bf18140245e3870a62582d5a8318 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260130 02:58:18.437644 19221 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e34d5da0ad864b2eaea0d575c982e447" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152" is_pre_election: true
W20260130 02:58:18.459941 19314 leader_election.cc:343] T e34d5da0ad864b2eaea0d575c982e447 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:18.460749 20608 leader_election.cc:290] T 25fd48e693424537b91ba7c3fcc96878 P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 election: Requested vote from peers 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767), d870164263bb44b6b780a557daea3656 (127.18.100.67:39571)
I20260130 02:58:18.467877 18872 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:50284 (request call id 245) took 3570 ms. Trace:
I20260130 02:58:18.468200 18872 rpcz_store.cc:276] 0130 02:58:14.897538 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:58:16.778458 (+1880920us) service_pool.cc:225] Handling call
0130 02:58:16.780610 (+ 2152us) catalog_manager.cc:2393] Looking up, locking, and authorizing table
0130 02:58:16.782334 (+ 1724us) catalog_manager.cc:3708] Apply alter schema
0130 02:58:16.782934 (+ 600us) catalog_manager.cc:3740] Acquired catalog manager lock
0130 02:58:16.783147 (+ 213us) catalog_manager.cc:3898] Updating metadata on disk
0130 02:58:16.970077 (+186930us) catalog_manager.cc:3929] Committing alterations to in-memory state
0130 02:58:18.467817 (+1497740us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0130 02:58:16.787698 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0130 02:58:16.788159 (+ 461us) write_op.cc:432] Acquiring schema lock in shared mode
0130 02:58:16.788185 (+ 26us) write_op.cc:435] Acquired schema lock
0130 02:58:16.788203 (+ 18us) tablet.cc:660] Decoding operations
0130 02:58:16.788474 (+ 271us) write_op.cc:620] Acquiring the partition lock for write op
0130 02:58:16.788508 (+ 34us) write_op.cc:641] Partition lock acquired for write op
0130 02:58:16.788587 (+ 79us) tablet.cc:683] Acquiring locks for 1 operations
0130 02:58:16.788759 (+ 172us) tablet.cc:699] Row locks acquired
0130 02:58:16.789982 (+ 1223us) write_op.cc:260] PREPARE: finished
0130 02:58:16.790112 (+ 130us) write_op.cc:270] Start()
0130 02:58:16.790354 (+ 242us) write_op.cc:276] Timestamp: P: 1769741896790047 usec, L: 0
0130 02:58:16.790374 (+ 20us) op_driver.cc:348] REPLICATION: starting
0130 02:58:16.790918 (+ 544us) log.cc:844] Serialized 405 byte log entry
0130 02:58:16.967187 (+176269us) op_driver.cc:464] REPLICATION: finished
0130 02:58:16.967622 (+ 435us) write_op.cc:301] APPLY: starting
0130 02:58:16.967673 (+ 51us) tablet.cc:1365] starting BulkCheckPresence
0130 02:58:16.967826 (+ 153us) tablet.cc:1368] finished BulkCheckPresence
0130 02:58:16.967922 (+ 96us) tablet.cc:1370] starting ApplyRowOperation cycle
0130 02:58:16.968076 (+ 154us) tablet.cc:1381] finished ApplyRowOperation cycle
0130 02:58:16.968225 (+ 149us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0130 02:58:16.968265 (+ 40us) write_op.cc:312] APPLY: finished
0130 02:58:16.968427 (+ 162us) log.cc:844] Serialized 23 byte log entry
0130 02:58:16.969098 (+ 671us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:58:16.969171 (+ 73us) write_op.cc:454] Released schema lock
0130 02:58:16.969207 (+ 36us) write_op.cc:341] FINISH: Updating metrics
0130 02:58:16.976173 (+ 6966us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:58:16.976355 (+ 182us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":284,"apply.run_cpu_time_us":1304,"apply.run_wall_time_us":8346,"num_ops":1,"prepare.queue_time_us":3455,"prepare.run_cpu_time_us":2276,"prepare.run_wall_time_us":7428,"replication_time_us":176700}]]}
I20260130 02:58:18.437860 20897 tablet_replica.cc:333] T c7c621b99deb457998c44ab1644d3939 P 371a36d219164974abdf848841528152: stopping tablet replica
I20260130 02:58:18.479326 20897 raft_consensus.cc:2243] T c7c621b99deb457998c44ab1644d3939 P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260130 02:58:18.480031 20897 raft_consensus.cc:2272] T c7c621b99deb457998c44ab1644d3939 P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260130 02:58:18.482406 20897 ts_tablet_manager.cc:1916] T c7c621b99deb457998c44ab1644d3939 P 371a36d219164974abdf848841528152: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260130 02:58:18.483922 19625 tablet_bootstrap.cc:654] T 3aa76993beb94b12bc706a76db26a03b P d870164263bb44b6b780a557daea3656: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:18.489905 19379 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "838e49e5e8874db393bdc15a1c92f87e" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74" is_pre_election: true
I20260130 02:58:18.490561 19379 raft_consensus.cc:2468] T 838e49e5e8874db393bdc15a1c92f87e P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 371a36d219164974abdf848841528152 in term 0.
I20260130 02:58:18.492519 19522 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "838e49e5e8874db393bdc15a1c92f87e" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
I20260130 02:58:18.493124 19522 raft_consensus.cc:2468] T 838e49e5e8874db393bdc15a1c92f87e P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 371a36d219164974abdf848841528152 in term 0.
I20260130 02:58:18.496062 19204 tablet_service.cc:1552] Processing DeleteTablet for tablet d55291d7cea64a0d9a345ce712fbfa80 with delete_type TABLET_DATA_DELETED (replaced by tablet cd30d7adcc4545848e3501a205a10293) from {username='slave'} at 127.0.0.1:42870
I20260130 02:58:18.497004 19202 tablet_service.cc:1552] Processing DeleteTablet for tablet 38a7a10e36874183bf9fe55a38d829f6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:58:14 UTC) from {username='slave'} at 127.0.0.1:42870
I20260130 02:58:18.458659 19807 tablet_bootstrap.cc:654] T 13f0ace4d9de43abab08eca32cdb30c1 P 371a36d219164974abdf848841528152: Neither blocks nor log segments found. Creating new log.
W20260130 02:58:18.511019 19313 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.100.67:39571, user_credentials={real_user=slave}} blocked reactor thread for 62675.8us
W20260130 02:58:18.511705 19313 leader_election.cc:343] T e34d5da0ad864b2eaea0d575c982e447 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:18.512043 19313 leader_election.cc:304] T e34d5da0ad864b2eaea0d575c982e447 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78c2c46c7520492db35d51c244fb4f74; no voters: 371a36d219164974abdf848841528152, d870164263bb44b6b780a557daea3656
I20260130 02:58:18.513191 20703 raft_consensus.cc:2749] T e34d5da0ad864b2eaea0d575c982e447 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260130 02:58:18.547267 18869 catalog_manager.cc:6199] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4: default.Table_f17d8e4d9f25449b84aa751af93936b2 [id=43254e466e7b4a2fb59fbcec68cfde36] alter complete (version 1)
I20260130 02:58:18.550196 20616 raft_consensus.cc:493] T 3389db58493448aea3cf00aca411047c P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 78c2c46c7520492db35d51c244fb4f74)
I20260130 02:58:18.551851 20616 raft_consensus.cc:515] T 3389db58493448aea3cf00aca411047c P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:18.553658 20616 leader_election.cc:290] T 3389db58493448aea3cf00aca411047c P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767), d870164263bb44b6b780a557daea3656 (127.18.100.67:39571)
I20260130 02:58:18.562350 19840 tablet_bootstrap.cc:492] T 1fc8f09d6329407c8078542b4123ef4d P 78c2c46c7520492db35d51c244fb4f74: No bootstrap required, opened a new log
I20260130 02:58:18.562875 19840 ts_tablet_manager.cc:1403] T 1fc8f09d6329407c8078542b4123ef4d P 78c2c46c7520492db35d51c244fb4f74: Time spent bootstrapping tablet: real 0.146s user 0.019s sys 0.022s
I20260130 02:58:18.564047 20899 tablet_replica.cc:333] T c7c621b99deb457998c44ab1644d3939 P 78c2c46c7520492db35d51c244fb4f74: stopping tablet replica
I20260130 02:58:18.565995 19462 leader_election.cc:304] T cf1b66a20fce4c4d96bc0bc32a7bafa8 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d870164263bb44b6b780a557daea3656; no voters: 371a36d219164974abdf848841528152, 78c2c46c7520492db35d51c244fb4f74
I20260130 02:58:18.567131 20864 raft_consensus.cc:2749] T cf1b66a20fce4c4d96bc0bc32a7bafa8 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260130 02:58:18.571077 19159 leader_election.cc:304] T f9e5f01e69254d658c369f3e21cef523 P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 371a36d219164974abdf848841528152, 78c2c46c7520492db35d51c244fb4f74; no voters: d870164263bb44b6b780a557daea3656
I20260130 02:58:18.571794 20866 raft_consensus.cc:2764] T f9e5f01e69254d658c369f3e21cef523 P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20260130 02:58:18.565038 19840 raft_consensus.cc:359] T 1fc8f09d6329407c8078542b4123ef4d P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:18.576937 19840 raft_consensus.cc:385] T 1fc8f09d6329407c8078542b4123ef4d P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:18.577224 19840 raft_consensus.cc:740] T 1fc8f09d6329407c8078542b4123ef4d P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78c2c46c7520492db35d51c244fb4f74, State: Initialized, Role: FOLLOWER
I20260130 02:58:18.577780 19840 consensus_queue.cc:260] T 1fc8f09d6329407c8078542b4123ef4d P 78c2c46c7520492db35d51c244fb4f74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:18.580518 19807 tablet_bootstrap.cc:492] T 13f0ace4d9de43abab08eca32cdb30c1 P 371a36d219164974abdf848841528152: No bootstrap required, opened a new log
I20260130 02:58:18.580932 19807 ts_tablet_manager.cc:1403] T 13f0ace4d9de43abab08eca32cdb30c1 P 371a36d219164974abdf848841528152: Time spent bootstrapping tablet: real 0.250s user 0.005s sys 0.005s
W20260130 02:58:18.594470 18875 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.18.100.65:51021 (request call id 36) took 4102 ms (client timeout 1000 ms). Trace:
I20260130 02:58:18.611778 19840 ts_tablet_manager.cc:1434] T 1fc8f09d6329407c8078542b4123ef4d P 78c2c46c7520492db35d51c244fb4f74: Time spent starting tablet: real 0.049s user 0.004s sys 0.001s
I20260130 02:58:18.612622 19840 tablet_bootstrap.cc:492] T 306452e208ca4e719609f67345114a13 P 78c2c46c7520492db35d51c244fb4f74: Bootstrap starting.
I20260130 02:58:18.615201 19202 tablet_service.cc:1552] Processing DeleteTablet for tablet d27ca7dd87f44102b7680e1f4a88954a with delete_type TABLET_DATA_DELETED (replaced by tablet d480bf18140245e3870a62582d5a8318) from {username='slave'} at 127.0.0.1:42870
I20260130 02:58:18.616657 20704 raft_consensus.cc:493] T eb7a755ab62b4552bb8588264c5f7b75 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:18.617100 20704 raft_consensus.cc:515] T eb7a755ab62b4552bb8588264c5f7b75 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:18.618081 19204 tablet_service.cc:1552] Processing DeleteTablet for tablet 3389db58493448aea3cf00aca411047c with delete_type TABLET_DATA_DELETED (replaced by tablet af0f90071a334016aa17fae2e3a97fc8) from {username='slave'} at 127.0.0.1:42870
I20260130 02:58:18.618027 20866 consensus_queue.cc:1048] T dcba6b534f9d406dad4fb502be40921c P 371a36d219164974abdf848841528152 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260130 02:58:18.623395 19528 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "eb7a755ab62b4552bb8588264c5f7b75" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
W20260130 02:58:18.624409 19313 leader_election.cc:343] T eb7a755ab62b4552bb8588264c5f7b75 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): Illegal state: must be running to vote when last-logged opid is not known
W20260130 02:58:18.602309 18875 rpcz_store.cc:269] 0130 02:58:14.491563 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:58:14.672570 (+181007us) service_pool.cc:225] Handling call
0130 02:58:18.594388 (+3921818us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0130 02:58:16.359566 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0130 02:58:16.360443 (+ 877us) write_op.cc:432] Acquiring schema lock in shared mode
0130 02:58:16.360473 (+ 30us) write_op.cc:435] Acquired schema lock
0130 02:58:16.360487 (+ 14us) tablet.cc:660] Decoding operations
0130 02:58:16.360828 (+ 341us) write_op.cc:620] Acquiring the partition lock for write op
0130 02:58:16.360874 (+ 46us) write_op.cc:641] Partition lock acquired for write op
0130 02:58:16.360903 (+ 29us) tablet.cc:683] Acquiring locks for 1 operations
0130 02:58:16.361075 (+ 172us) tablet.cc:699] Row locks acquired
0130 02:58:16.361088 (+ 13us) write_op.cc:260] PREPARE: finished
0130 02:58:16.361228 (+ 140us) write_op.cc:270] Start()
0130 02:58:16.361330 (+ 102us) write_op.cc:276] Timestamp: P: 1769741896361161 usec, L: 0
0130 02:58:16.361349 (+ 19us) op_driver.cc:348] REPLICATION: starting
0130 02:58:16.374125 (+ 12776us) log.cc:844] Serialized 356 byte log entry
0130 02:58:16.375837 (+ 1712us) spinlock_profiling.cc:243] Waited 12.2 ms on lock 0x7b5400030554. stack: 00007fb0bd3f7373 00007fb0bd3f7270 00007fb0bcf6c574 0000000000508744 00000000005086d9 00007fb0c44485a1 00007fb0c44daaa8 00007fb0c494c4d1 00007fb0c494bca8 00007fb0c4950941 00007fb0c49508f9 00007fb0c4950889 00007fb0c4950851 00007fb0c494fa7d 00000000005aa2b4 00000000005aa0e9
0130 02:58:16.397239 (+ 21402us) op_driver.cc:464] REPLICATION: finished
0130 02:58:16.397583 (+ 344us) write_op.cc:301] APPLY: starting
0130 02:58:16.397642 (+ 59us) tablet.cc:1365] starting BulkCheckPresence
0130 02:58:16.397746 (+ 104us) tablet.cc:1368] finished BulkCheckPresence
0130 02:58:16.397766 (+ 20us) tablet.cc:1370] starting ApplyRowOperation cycle
0130 02:58:16.397957 (+ 191us) tablet.cc:1381] finished ApplyRowOperation cycle
0130 02:58:16.398111 (+ 154us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0130 02:58:16.398163 (+ 52us) write_op.cc:312] APPLY: finished
0130 02:58:16.398347 (+ 184us) log.cc:844] Serialized 23 byte log entry
0130 02:58:16.398497 (+ 150us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:58:16.398561 (+ 64us) write_op.cc:454] Released schema lock
0130 02:58:16.399150 (+ 589us) write_op.cc:341] FINISH: Updating metrics
0130 02:58:16.399367 (+ 217us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:58:16.399393 (+ 26us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":104566,"reported_tablets":43,"child_traces":[["op",{"apply.queue_time_us":189,"apply.run_cpu_time_us":1239,"apply.run_wall_time_us":1749,"num_ops":1,"prepare.queue_time_us":50806,"prepare.run_cpu_time_us":4199,"prepare.run_wall_time_us":33012,"raft.queue_time_us":4426,"raft.run_cpu_time_us":630,"raft.run_wall_time_us":630,"replication_time_us":35795,"spinlock_wait_cycles":27963264}]]}
W20260130 02:58:18.626047 19314 consensus_peers.cc:597] T c7c621b99deb457998c44ab1644d3939 P 78c2c46c7520492db35d51c244fb4f74 -> Peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Couldn't send request to peer 371a36d219164974abdf848841528152. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20260130 02:58:18.628830 20704 leader_election.cc:290] T eb7a755ab62b4552bb8588264c5f7b75 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d870164263bb44b6b780a557daea3656 (127.18.100.67:39571), 371a36d219164974abdf848841528152 (127.18.100.65:42021)
I20260130 02:58:18.637403 19840 tablet_bootstrap.cc:654] T 306452e208ca4e719609f67345114a13 P 78c2c46c7520492db35d51c244fb4f74: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:18.640743 19222 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "eb7a755ab62b4552bb8588264c5f7b75" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152" is_pre_election: true
W20260130 02:58:18.642052 19314 leader_election.cc:343] T eb7a755ab62b4552bb8588264c5f7b75 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:18.642450 19314 leader_election.cc:304] T eb7a755ab62b4552bb8588264c5f7b75 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78c2c46c7520492db35d51c244fb4f74; no voters: 371a36d219164974abdf848841528152, d870164263bb44b6b780a557daea3656
I20260130 02:58:18.644596 20697 raft_consensus.cc:2749] T eb7a755ab62b4552bb8588264c5f7b75 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260130 02:58:18.651660 19356 tablet_service.cc:1552] Processing DeleteTablet for tablet ec28ff12927d450ead16d26cf016138d with delete_type TABLET_DATA_DELETED (replaced by tablet 6d1f121f49cd47c2b8247daaf6874e01) from {username='slave'} at 127.0.0.1:60110
I20260130 02:58:18.652537 19356 tablet_service.cc:1552] Processing DeleteTablet for tablet 78e9bc3e55fe4e4a98e8eb01da59d365 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:58:08 UTC) from {username='slave'} at 127.0.0.1:60110
I20260130 02:58:18.653635 19356 tablet_service.cc:1552] Processing DeleteTablet for tablet e2aaaf00f53d418b87d5c7af286e4d84 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:58:08 UTC) from {username='slave'} at 127.0.0.1:60110
I20260130 02:58:18.654668 19356 tablet_service.cc:1552] Processing DeleteTablet for tablet 3389db58493448aea3cf00aca411047c with delete_type TABLET_DATA_DELETED (replaced by tablet af0f90071a334016aa17fae2e3a97fc8) from {username='slave'} at 127.0.0.1:60110
I20260130 02:58:18.657562 19807 raft_consensus.cc:359] T 13f0ace4d9de43abab08eca32cdb30c1 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:18.658303 19807 raft_consensus.cc:385] T 13f0ace4d9de43abab08eca32cdb30c1 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:18.658646 19807 raft_consensus.cc:740] T 13f0ace4d9de43abab08eca32cdb30c1 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 371a36d219164974abdf848841528152, State: Initialized, Role: FOLLOWER
W20260130 02:58:18.664479 19159 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.100.66:40767, user_credentials={real_user=slave}} blocked reactor thread for 56760.7us
I20260130 02:58:18.664844 19625 tablet_bootstrap.cc:492] T 3aa76993beb94b12bc706a76db26a03b P d870164263bb44b6b780a557daea3656: No bootstrap required, opened a new log
I20260130 02:58:18.665279 19625 ts_tablet_manager.cc:1403] T 3aa76993beb94b12bc706a76db26a03b P d870164263bb44b6b780a557daea3656: Time spent bootstrapping tablet: real 0.249s user 0.011s sys 0.003s
I20260130 02:58:18.666422 20608 raft_consensus.cc:493] T e12d130bcdf34965888753f12c1ff848 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:18.679313 20608 raft_consensus.cc:515] T e12d130bcdf34965888753f12c1ff848 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:18.679576 19625 raft_consensus.cc:359] T 3aa76993beb94b12bc706a76db26a03b P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:18.680219 19625 raft_consensus.cc:385] T 3aa76993beb94b12bc706a76db26a03b P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:18.680464 19625 raft_consensus.cc:740] T 3aa76993beb94b12bc706a76db26a03b P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d870164263bb44b6b780a557daea3656, State: Initialized, Role: FOLLOWER
I20260130 02:58:18.681834 20912 tablet.cc:1722] T 8e57ae31ab3941f789ac69627384e3b0 P 78c2c46c7520492db35d51c244fb4f74: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260130 02:58:18.686672 20608 leader_election.cc:290] T e12d130bcdf34965888753f12c1ff848 P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d870164263bb44b6b780a557daea3656 (127.18.100.67:39571), 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767)
I20260130 02:58:18.698174 18868 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:50284 (request call id 246) took 2913 ms. Trace:
I20260130 02:58:18.698889 18868 rpcz_store.cc:276] 0130 02:58:15.784373 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:58:16.807289 (+1022916us) service_pool.cc:225] Handling call
0130 02:58:16.809736 (+ 2447us) catalog_manager.cc:2393] Looking up, locking, and authorizing table
0130 02:58:16.810538 (+ 802us) catalog_manager.cc:2815] Modifying in-memory table state
0130 02:58:16.810694 (+ 156us) catalog_manager.cc:2824] Locking tablets
0130 02:58:16.889733 (+ 79039us) catalog_manager.cc:2838] Removing table and tablets from system table
0130 02:58:17.008638 (+118905us) catalog_manager.cc:2856] Removing table from by-name map
0130 02:58:17.065265 (+ 56627us) catalog_manager.cc:2872] Committing in-memory state
0130 02:58:18.379550 (+1314285us) catalog_manager.cc:2876] Aborting table tasks
0130 02:58:18.698111 (+318561us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0130 02:58:16.897233 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0130 02:58:16.897738 (+ 505us) write_op.cc:432] Acquiring schema lock in shared mode
0130 02:58:16.897765 (+ 27us) write_op.cc:435] Acquired schema lock
0130 02:58:16.897781 (+ 16us) tablet.cc:660] Decoding operations
0130 02:58:16.898173 (+ 392us) write_op.cc:620] Acquiring the partition lock for write op
0130 02:58:16.898213 (+ 40us) write_op.cc:641] Partition lock acquired for write op
0130 02:58:16.898238 (+ 25us) tablet.cc:683] Acquiring locks for 2 operations
0130 02:58:16.898508 (+ 270us) tablet.cc:699] Row locks acquired
0130 02:58:16.898526 (+ 18us) write_op.cc:260] PREPARE: finished
0130 02:58:16.898672 (+ 146us) write_op.cc:270] Start()
0130 02:58:16.898763 (+ 91us) write_op.cc:276] Timestamp: P: 1769741896898606 usec, L: 0
0130 02:58:16.898783 (+ 20us) op_driver.cc:348] REPLICATION: starting
0130 02:58:16.959177 (+ 60394us) log.cc:844] Serialized 771 byte log entry
0130 02:58:16.970657 (+ 11480us) spinlock_profiling.cc:243] Waited 59.8 ms on lock 0x7b5400030554. stack: 00007fb0bd3f7373 00007fb0bd3f7270 00007fb0bcf6c574 0000000000508744 00000000005086d9 00007fb0c44485a1 00007fb0c44daaa8 00007fb0c494c4d1 00007fb0c494bca8 00007fb0c4950941 00007fb0c49508f9 00007fb0c4950889 00007fb0c4950851 00007fb0c494fa7d 00000000005aa2b4 00000000005aa0e9
0130 02:58:16.991703 (+ 21046us) op_driver.cc:464] REPLICATION: finished
0130 02:58:16.992318 (+ 615us) write_op.cc:301] APPLY: starting
0130 02:58:16.992495 (+ 177us) tablet.cc:1365] starting BulkCheckPresence
0130 02:58:16.992663 (+ 168us) tablet.cc:1368] finished BulkCheckPresence
0130 02:58:16.992680 (+ 17us) tablet.cc:1370] starting ApplyRowOperation cycle
0130 02:58:16.995282 (+ 2602us) tablet.cc:1381] finished ApplyRowOperation cycle
0130 02:58:16.996192 (+ 910us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0130 02:58:16.996464 (+ 272us) write_op.cc:312] APPLY: finished
0130 02:58:16.997780 (+ 1316us) log.cc:844] Serialized 29 byte log entry
0130 02:58:17.007143 (+ 9363us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:58:17.007693 (+ 550us) write_op.cc:454] Released schema lock
0130 02:58:17.007965 (+ 272us) write_op.cc:341] FINISH: Updating metrics
0130 02:58:17.037460 (+ 29495us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:58:17.037539 (+ 79us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":429,"apply.run_cpu_time_us":2260,"apply.run_wall_time_us":17065,"num_ops":2,"prepare.queue_time_us":5589,"prepare.run_cpu_time_us":3840,"prepare.run_wall_time_us":73681,"raft.queue_time_us":293,"raft.run_cpu_time_us":655,"raft.run_wall_time_us":654,"replication_time_us":92821,"spinlock_wait_cycles":139413888}]]}
I20260130 02:58:18.703151 19496 tablet_service.cc:1552] Processing DeleteTablet for tablet c7c621b99deb457998c44ab1644d3939 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:58:16 UTC) from {username='slave'} at 127.0.0.1:59068
I20260130 02:58:18.711035 19356 tablet_service.cc:1552] Processing DeleteTablet for tablet 1ca34772ef484c1f809a99d126e28eff with delete_type TABLET_DATA_DELETED (replaced by tablet 00ddbfe6218c44d788fbf65245a31dc6) from {username='slave'} at 127.0.0.1:60110
I20260130 02:58:18.712458 19356 tablet_service.cc:1552] Processing DeleteTablet for tablet d27ca7dd87f44102b7680e1f4a88954a with delete_type TABLET_DATA_DELETED (replaced by tablet d480bf18140245e3870a62582d5a8318) from {username='slave'} at 127.0.0.1:60110
I20260130 02:58:18.716653 19625 consensus_queue.cc:260] T 3aa76993beb94b12bc706a76db26a03b P d870164263bb44b6b780a557daea3656 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:18.734860 18868 master_service.cc:954] ReplaceTablet: received request to replace tablet 04f4ac27800c4ac3a79ae12d35479757 from {username='slave'} at 127.0.0.1:50284
W20260130 02:58:18.743309 18869 catalog_manager.cc:5846] Failed to send AlterTable RPC for tablet 25fd48e693424537b91ba7c3fcc96878 (table default.Table_ebc8b8379c274c18b0a06a919dbe0ee0 [id=28156e0843674fce88f5914a3e4195ae], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet 25fd48e693424537b91ba7c3fcc96878 consensus state has no leader
I20260130 02:58:18.744791 20864 raft_consensus.cc:493] T 3aa76993beb94b12bc706a76db26a03b P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:18.745332 20864 raft_consensus.cc:515] T 3aa76993beb94b12bc706a76db26a03b P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:18.747833 19222 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3aa76993beb94b12bc706a76db26a03b" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152" is_pre_election: true
I20260130 02:58:18.748322 20864 leader_election.cc:290] T 3aa76993beb94b12bc706a76db26a03b P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767)
I20260130 02:58:18.748485 19222 raft_consensus.cc:2468] T 3aa76993beb94b12bc706a76db26a03b P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d870164263bb44b6b780a557daea3656 in term 0.
I20260130 02:58:18.749471 19463 leader_election.cc:304] T 3aa76993beb94b12bc706a76db26a03b P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 371a36d219164974abdf848841528152, d870164263bb44b6b780a557daea3656; no voters:
I20260130 02:58:18.750444 19379 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3aa76993beb94b12bc706a76db26a03b" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74" is_pre_election: true
I20260130 02:58:18.750679 20864 raft_consensus.cc:2804] T 3aa76993beb94b12bc706a76db26a03b P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260130 02:58:18.751014 20864 raft_consensus.cc:493] T 3aa76993beb94b12bc706a76db26a03b P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260130 02:58:18.751319 20864 raft_consensus.cc:3060] T 3aa76993beb94b12bc706a76db26a03b P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:18.751189 19379 raft_consensus.cc:2468] T 3aa76993beb94b12bc706a76db26a03b P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d870164263bb44b6b780a557daea3656 in term 0.
I20260130 02:58:18.756279 19807 consensus_queue.cc:260] T 13f0ace4d9de43abab08eca32cdb30c1 P 371a36d219164974abdf848841528152 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:18.761780 19356 tablet_service.cc:1552] Processing DeleteTablet for tablet 38a7a10e36874183bf9fe55a38d829f6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:58:14 UTC) from {username='slave'} at 127.0.0.1:60110
I20260130 02:58:18.766706 19204 tablet_service.cc:1552] Processing DeleteTablet for tablet 1ca34772ef484c1f809a99d126e28eff with delete_type TABLET_DATA_DELETED (replaced by tablet 00ddbfe6218c44d788fbf65245a31dc6) from {username='slave'} at 127.0.0.1:42870
I20260130 02:58:18.768718 19625 ts_tablet_manager.cc:1434] T 3aa76993beb94b12bc706a76db26a03b P d870164263bb44b6b780a557daea3656: Time spent starting tablet: real 0.103s user 0.006s sys 0.001s
I20260130 02:58:18.770546 19501 tablet_service.cc:1552] Processing DeleteTablet for tablet ec28ff12927d450ead16d26cf016138d with delete_type TABLET_DATA_DELETED (replaced by tablet 6d1f121f49cd47c2b8247daaf6874e01) from {username='slave'} at 127.0.0.1:59068
I20260130 02:58:18.772173 19625 tablet_bootstrap.cc:492] T e12d130bcdf34965888753f12c1ff848 P d870164263bb44b6b780a557daea3656: Bootstrap starting.
I20260130 02:58:18.758323 20913 tablet.cc:1722] T 8e57ae31ab3941f789ac69627384e3b0 P 371a36d219164974abdf848841528152: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260130 02:58:18.779098 19159 leader_election.cc:304] T 838e49e5e8874db393bdc15a1c92f87e P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 371a36d219164974abdf848841528152, 78c2c46c7520492db35d51c244fb4f74; no voters:
I20260130 02:58:18.788462 19204 tablet_service.cc:1552] Processing DeleteTablet for tablet ec28ff12927d450ead16d26cf016138d with delete_type TABLET_DATA_DELETED (replaced by tablet 6d1f121f49cd47c2b8247daaf6874e01) from {username='slave'} at 127.0.0.1:42870
I20260130 02:58:18.798934 20920 raft_consensus.cc:493] T 3aa76993beb94b12bc706a76db26a03b P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:18.799451 20920 raft_consensus.cc:515] T 3aa76993beb94b12bc706a76db26a03b P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:18.801219 20920 leader_election.cc:290] T 3aa76993beb94b12bc706a76db26a03b P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d870164263bb44b6b780a557daea3656 (127.18.100.67:39571), 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767)
W20260130 02:58:18.747792 18869 catalog_manager.cc:5846] Failed to send AlterTable RPC for tablet 838e49e5e8874db393bdc15a1c92f87e (table default.Table_da3d581b15a94840898777f74611994e [id=b7e241ced53e4160ac6fa1f65ef3674e], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet 838e49e5e8874db393bdc15a1c92f87e consensus state has no leader
W20260130 02:58:18.807689 18869 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.18.100.66:59691 (request call id 34) took 2621 ms (client timeout 1000 ms). Trace:
W20260130 02:58:18.808470 18860 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet 25fd48e693424537b91ba7c3fcc96878 (table default.Table_ebc8b8379c274c18b0a06a919dbe0ee0 [id=28156e0843674fce88f5914a3e4195ae], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 25fd48e693424537b91ba7c3fcc96878 consensus state has no leader
I20260130 02:58:18.772446 19501 tablet_service.cc:1552] Processing DeleteTablet for tablet 38a7a10e36874183bf9fe55a38d829f6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:58:14 UTC) from {username='slave'} at 127.0.0.1:59068
I20260130 02:58:18.773814 19505 tablet_service.cc:1552] Processing DeleteTablet for tablet 3389db58493448aea3cf00aca411047c with delete_type TABLET_DATA_DELETED (replaced by tablet af0f90071a334016aa17fae2e3a97fc8) from {username='slave'} at 127.0.0.1:59068
I20260130 02:58:18.775171 19508 tablet_service.cc:1552] Processing DeleteTablet for tablet 1ca34772ef484c1f809a99d126e28eff with delete_type TABLET_DATA_DELETED (replaced by tablet 00ddbfe6218c44d788fbf65245a31dc6) from {username='slave'} at 127.0.0.1:59068
I20260130 02:58:18.776724 19502 tablet_service.cc:1552] Processing DeleteTablet for tablet d27ca7dd87f44102b7680e1f4a88954a with delete_type TABLET_DATA_DELETED (replaced by tablet d480bf18140245e3870a62582d5a8318) from {username='slave'} at 127.0.0.1:59068
I20260130 02:58:18.778118 20916 tablet_replica.cc:333] T c7c621b99deb457998c44ab1644d3939 P d870164263bb44b6b780a557daea3656: stopping tablet replica
I20260130 02:58:18.815805 19840 tablet_bootstrap.cc:492] T 306452e208ca4e719609f67345114a13 P 78c2c46c7520492db35d51c244fb4f74: No bootstrap required, opened a new log
I20260130 02:58:18.816247 19840 ts_tablet_manager.cc:1403] T 306452e208ca4e719609f67345114a13 P 78c2c46c7520492db35d51c244fb4f74: Time spent bootstrapping tablet: real 0.204s user 0.027s sys 0.032s
I20260130 02:58:18.816578 20916 raft_consensus.cc:2243] T c7c621b99deb457998c44ab1644d3939 P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260130 02:58:18.817147 20916 raft_consensus.cc:2272] T c7c621b99deb457998c44ab1644d3939 P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260130 02:58:18.818845 19840 raft_consensus.cc:359] T 306452e208ca4e719609f67345114a13 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:18.819455 19840 raft_consensus.cc:385] T 306452e208ca4e719609f67345114a13 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:18.779161 20917 tablet.cc:1722] T 8e57ae31ab3941f789ac69627384e3b0 P d870164263bb44b6b780a557daea3656: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260130 02:58:18.819702 19840 raft_consensus.cc:740] T 306452e208ca4e719609f67345114a13 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78c2c46c7520492db35d51c244fb4f74, State: Initialized, Role: FOLLOWER
I20260130 02:58:18.820238 19502 tablet_service.cc:1552] Processing DeleteTablet for tablet d55291d7cea64a0d9a345ce712fbfa80 with delete_type TABLET_DATA_DELETED (replaced by tablet cd30d7adcc4545848e3501a205a10293) from {username='slave'} at 127.0.0.1:59068
W20260130 02:58:18.793906 19463 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.100.65:42021, user_credentials={real_user=slave}} blocked reactor thread for 44494us
I20260130 02:58:18.820252 19840 consensus_queue.cc:260] T 306452e208ca4e719609f67345114a13 P 78c2c46c7520492db35d51c244fb4f74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:18.821823 19840 ts_tablet_manager.cc:1434] T 306452e208ca4e719609f67345114a13 P 78c2c46c7520492db35d51c244fb4f74: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260130 02:58:18.824836 20864 raft_consensus.cc:515] T 3aa76993beb94b12bc706a76db26a03b P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:18.827117 19840 tablet_bootstrap.cc:492] T cd30d7adcc4545848e3501a205a10293 P 78c2c46c7520492db35d51c244fb4f74: Bootstrap starting.
W20260130 02:58:18.828147 18871 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.18.100.65:51021 (request call id 37) took 3075 ms (client timeout 1000 ms). Trace:
I20260130 02:58:18.833997 19840 tablet_bootstrap.cc:654] T cd30d7adcc4545848e3501a205a10293 P 78c2c46c7520492db35d51c244fb4f74: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:18.834396 19356 tablet_service.cc:1552] Processing DeleteTablet for tablet d55291d7cea64a0d9a345ce712fbfa80 with delete_type TABLET_DATA_DELETED (replaced by tablet cd30d7adcc4545848e3501a205a10293) from {username='slave'} at 127.0.0.1:60110
W20260130 02:58:18.844259 18866 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.18.100.67:47213 (request call id 31) took 2484 ms (client timeout 1000 ms). Trace:
I20260130 02:58:18.802539 20920 raft_consensus.cc:2804] T 838e49e5e8874db393bdc15a1c92f87e P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260130 02:58:18.844699 20920 raft_consensus.cc:493] T 838e49e5e8874db393bdc15a1c92f87e P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260130 02:58:18.845048 20920 raft_consensus.cc:3060] T 838e49e5e8874db393bdc15a1c92f87e P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Advancing to term 1
W20260130 02:58:18.844743 18866 rpcz_store.cc:269] 0130 02:58:16.360078 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:58:16.528266 (+168188us) service_pool.cc:225] Handling call
0130 02:58:16.708593 (+180327us) spinlock_profiling.cc:243] Waited 47.1 ms on lock 0x7b5400030554. stack: 00007fb0bd3f7373 00007fb0bd3f7270 00007fb0bcf6c574 0000000000508744 00000000005086d9 00007fb0c44490c0 00007fb0c44e7bba 00007fb0c61e6eac 00007fb0c62e110e 00007fb0c3efe4a4 00007fb0c3efe432 00007fb0c3efe361 00007fb0c3efe2dc 00007fb0c3efd592 00007fb0c21dd34c 00007fb0c21dc786
0130 02:58:18.844179 (+2135586us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0130 02:58:18.080649 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0130 02:58:18.081097 (+ 448us) write_op.cc:432] Acquiring schema lock in shared mode
0130 02:58:18.081124 (+ 27us) write_op.cc:435] Acquired schema lock
0130 02:58:18.081139 (+ 15us) tablet.cc:660] Decoding operations
0130 02:58:18.081620 (+ 481us) write_op.cc:620] Acquiring the partition lock for write op
0130 02:58:18.081664 (+ 44us) write_op.cc:641] Partition lock acquired for write op
0130 02:58:18.081687 (+ 23us) tablet.cc:683] Acquiring locks for 3 operations
0130 02:58:18.081999 (+ 312us) tablet.cc:699] Row locks acquired
0130 02:58:18.082017 (+ 18us) write_op.cc:260] PREPARE: finished
0130 02:58:18.082158 (+ 141us) write_op.cc:270] Start()
0130 02:58:18.082257 (+ 99us) write_op.cc:276] Timestamp: P: 1769741898082071 usec, L: 0
0130 02:58:18.082273 (+ 16us) op_driver.cc:348] REPLICATION: starting
0130 02:58:18.083022 (+ 749us) log.cc:844] Serialized 1293 byte log entry
0130 02:58:18.092004 (+ 8982us) op_driver.cc:464] REPLICATION: finished
0130 02:58:18.093209 (+ 1205us) write_op.cc:301] APPLY: starting
0130 02:58:18.093293 (+ 84us) tablet.cc:1365] starting BulkCheckPresence
0130 02:58:18.093462 (+ 169us) tablet.cc:1368] finished BulkCheckPresence
0130 02:58:18.093600 (+ 138us) tablet.cc:1370] starting ApplyRowOperation cycle
0130 02:58:18.094091 (+ 491us) tablet.cc:1381] finished ApplyRowOperation cycle
0130 02:58:18.094236 (+ 145us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=3
0130 02:58:18.094274 (+ 38us) write_op.cc:312] APPLY: finished
0130 02:58:18.094450 (+ 176us) log.cc:844] Serialized 35 byte log entry
0130 02:58:18.094600 (+ 150us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:58:18.094673 (+ 73us) write_op.cc:454] Released schema lock
0130 02:58:18.094702 (+ 29us) write_op.cc:341] FINISH: Updating metrics
0130 02:58:18.094905 (+ 203us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:58:18.094928 (+ 23us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":322548,"reported_tablets":52,"spinlock_wait_cycles":108232576,"child_traces":[["op",{"apply.queue_time_us":909,"apply.run_cpu_time_us":1555,"apply.run_wall_time_us":1711,"num_ops":3,"prepare.queue_time_us":2873,"prepare.run_cpu_time_us":2518,"prepare.run_wall_time_us":2668,"raft.queue_time_us":173,"raft.run_cpu_time_us":636,"raft.run_wall_time_us":635,"replication_time_us":9637}]]}
I20260130 02:58:18.855821 19840 tablet_bootstrap.cc:492] T cd30d7adcc4545848e3501a205a10293 P 78c2c46c7520492db35d51c244fb4f74: No bootstrap required, opened a new log
I20260130 02:58:18.856235 19840 ts_tablet_manager.cc:1403] T cd30d7adcc4545848e3501a205a10293 P 78c2c46c7520492db35d51c244fb4f74: Time spent bootstrapping tablet: real 0.034s user 0.011s sys 0.000s
I20260130 02:58:18.858412 19840 raft_consensus.cc:359] T cd30d7adcc4545848e3501a205a10293 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:18.859056 19840 raft_consensus.cc:385] T cd30d7adcc4545848e3501a205a10293 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:18.859313 19840 raft_consensus.cc:740] T cd30d7adcc4545848e3501a205a10293 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78c2c46c7520492db35d51c244fb4f74, State: Initialized, Role: FOLLOWER
I20260130 02:58:18.859900 19840 consensus_queue.cc:260] T cd30d7adcc4545848e3501a205a10293 P 78c2c46c7520492db35d51c244fb4f74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:18.864980 20697 raft_consensus.cc:493] T 2b0edaad6877460cb7982efc62b00b66 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:18.865437 20697 raft_consensus.cc:515] T 2b0edaad6877460cb7982efc62b00b66 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:18.867440 20697 leader_election.cc:290] T 2b0edaad6877460cb7982efc62b00b66 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d870164263bb44b6b780a557daea3656 (127.18.100.67:39571), 371a36d219164974abdf848841528152 (127.18.100.65:42021)
I20260130 02:58:18.869258 19840 ts_tablet_manager.cc:1434] T cd30d7adcc4545848e3501a205a10293 P 78c2c46c7520492db35d51c244fb4f74: Time spent starting tablet: real 0.013s user 0.006s sys 0.000s
I20260130 02:58:18.870229 19840 tablet_bootstrap.cc:492] T cfd12a72726846e587cff9d6a4d61bd6 P 78c2c46c7520492db35d51c244fb4f74: Bootstrap starting.
I20260130 02:58:18.880232 19840 tablet_bootstrap.cc:654] T cfd12a72726846e587cff9d6a4d61bd6 P 78c2c46c7520492db35d51c244fb4f74: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:18.889092 19356 tablet_service.cc:1552] Processing DeleteTablet for tablet e364f4e70e5749e083e14e7fa7416ff0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:58:10 UTC) from {username='slave'} at 127.0.0.1:60110
I20260130 02:58:18.902233 20899 raft_consensus.cc:2243] T c7c621b99deb457998c44ab1644d3939 P 78c2c46c7520492db35d51c244fb4f74 [term 1 LEADER]: Raft consensus shutting down.
I20260130 02:58:18.903396 20899 raft_consensus.cc:2272] T c7c621b99deb457998c44ab1644d3939 P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260130 02:58:18.906219 20899 ts_tablet_manager.cc:1916] T c7c621b99deb457998c44ab1644d3939 P 78c2c46c7520492db35d51c244fb4f74: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260130 02:58:18.919840 19807 ts_tablet_manager.cc:1434] T 13f0ace4d9de43abab08eca32cdb30c1 P 371a36d219164974abdf848841528152: Time spent starting tablet: real 0.339s user 0.007s sys 0.001s
W20260130 02:58:18.873354 19463 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.100.65:42021, user_credentials={real_user=slave}} blocked reactor thread for 51917.4us
I20260130 02:58:18.926980 19222 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2b0edaad6877460cb7982efc62b00b66" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152" is_pre_election: true
I20260130 02:58:18.928694 20864 leader_election.cc:290] T 3aa76993beb94b12bc706a76db26a03b P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 election: Requested vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767)
I20260130 02:58:18.847632 20916 ts_tablet_manager.cc:1916] T c7c621b99deb457998c44ab1644d3939 P d870164263bb44b6b780a557daea3656: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260130 02:58:18.878329 19508 tablet_service.cc:1552] Processing DeleteTablet for tablet e364f4e70e5749e083e14e7fa7416ff0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:58:10 UTC) from {username='slave'} at 127.0.0.1:59068
I20260130 02:58:18.885730 19502 tablet_service.cc:1552] Processing DeleteTablet for tablet f54addb1a4304ce69c711ae0cc3b452c with delete_type TABLET_DATA_DELETED (replaced by tablet 7783db3568f4434e91e9e456799cc312) from {username='slave'} at 127.0.0.1:59068
I20260130 02:58:18.885990 19505 tablet_service.cc:1552] Processing DeleteTablet for tablet 8bd3e1c06b89426da57e2b259e7983af with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:58:08 UTC) from {username='slave'} at 127.0.0.1:59068
I20260130 02:58:18.886772 19501 tablet_service.cc:1552] Processing DeleteTablet for tablet b2ac91da01954c2390b6b7dcc06c394f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:58:04 UTC) from {username='slave'} at 127.0.0.1:59068
I20260130 02:58:18.887492 19496 tablet_service.cc:1552] Processing DeleteTablet for tablet 78e9bc3e55fe4e4a98e8eb01da59d365 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:58:08 UTC) from {username='slave'} at 127.0.0.1:59068
I20260130 02:58:18.880172 18868 catalog_manager.cc:5941] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4: Sending DeleteTablet for 3 replicas of tablet 04f4ac27800c4ac3a79ae12d35479757
W20260130 02:58:18.808007 18869 rpcz_store.cc:269] 0130 02:58:16.185782 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:58:16.358720 (+172938us) service_pool.cc:225] Handling call
0130 02:58:16.859699 (+500979us) spinlock_profiling.cc:243] Waited 152 ms on lock 0x7b5400030554. stack: 00007fb0bd3f7373 00007fb0bd3f7270 00007fb0bcf6c574 0000000000508744 00000000005086d9 00007fb0c44490c0 00007fb0c44e7bba 00007fb0c61e6eac 00007fb0c62e110e 00007fb0c3efe4a4 00007fb0c3efe432 00007fb0c3efe361 00007fb0c3efe2dc 00007fb0c3efd592 00007fb0c21dd34c 00007fb0c21dc786
0130 02:58:18.546877 (+1687178us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:58:18.546907 (+ 30us) write_op.cc:454] Released schema lock
0130 02:58:18.807624 (+260717us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0130 02:58:18.518841 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0130 02:58:18.519302 (+ 461us) write_op.cc:432] Acquiring schema lock in shared mode
0130 02:58:18.519331 (+ 29us) write_op.cc:435] Acquired schema lock
0130 02:58:18.519347 (+ 16us) tablet.cc:660] Decoding operations
0130 02:58:18.519654 (+ 307us) write_op.cc:620] Acquiring the partition lock for write op
0130 02:58:18.519696 (+ 42us) write_op.cc:641] Partition lock acquired for write op
0130 02:58:18.519724 (+ 28us) tablet.cc:683] Acquiring locks for 1 operations
0130 02:58:18.519878 (+ 154us) tablet.cc:699] Row locks acquired
0130 02:58:18.519896 (+ 18us) write_op.cc:260] PREPARE: finished
0130 02:58:18.520017 (+ 121us) write_op.cc:270] Start()
0130 02:58:18.520113 (+ 96us) write_op.cc:276] Timestamp: P: 1769741898519950 usec, L: 0
0130 02:58:18.520131 (+ 18us) op_driver.cc:348] REPLICATION: starting
0130 02:58:18.520677 (+ 546us) log.cc:844] Serialized 356 byte log entry
0130 02:58:18.530718 (+ 10041us) op_driver.cc:464] REPLICATION: finished
0130 02:58:18.531012 (+ 294us) write_op.cc:301] APPLY: starting
0130 02:58:18.531067 (+ 55us) tablet.cc:1365] starting BulkCheckPresence
0130 02:58:18.531150 (+ 83us) tablet.cc:1368] finished BulkCheckPresence
0130 02:58:18.531246 (+ 96us) tablet.cc:1370] starting ApplyRowOperation cycle
0130 02:58:18.531414 (+ 168us) tablet.cc:1381] finished ApplyRowOperation cycle
0130 02:58:18.531542 (+ 128us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0130 02:58:18.531582 (+ 40us) write_op.cc:312] APPLY: finished
0130 02:58:18.531749 (+ 167us) log.cc:844] Serialized 23 byte log entry
0130 02:58:18.531883 (+ 134us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:58:18.531945 (+ 62us) write_op.cc:454] Released schema lock
0130 02:58:18.531974 (+ 29us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"mutex_wait_us":25386,"reported_tablets":61,"spinlock_wait_cycles":349606656,"child_traces":[["op",{"apply.queue_time_us":162,"apply.run_cpu_time_us":1129,"apply.run_wall_time_us":1129,"num_ops":1,"prepare.queue_time_us":28787,"prepare.run_cpu_time_us":2168,"prepare.run_wall_time_us":9257,"raft.queue_time_us":998,"raft.run_cpu_time_us":647,"raft.run_wall_time_us":673,"replication_time_us":10425}]]}
W20260130 02:58:18.913347 18870 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.18.100.67:47213 (request call id 30) took 4219 ms (client timeout 1000 ms). Trace:
W20260130 02:58:18.828573 18871 rpcz_store.cc:269] 0130 02:58:15.753159 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:58:15.805178 (+ 52019us) service_pool.cc:225] Handling call
0130 02:58:18.828074 (+3022896us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0130 02:58:17.736966 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0130 02:58:17.737403 (+ 437us) write_op.cc:432] Acquiring schema lock in shared mode
0130 02:58:17.737440 (+ 37us) write_op.cc:435] Acquired schema lock
0130 02:58:17.737456 (+ 16us) tablet.cc:660] Decoding operations
0130 02:58:17.737949 (+ 493us) write_op.cc:620] Acquiring the partition lock for write op
0130 02:58:17.737993 (+ 44us) write_op.cc:641] Partition lock acquired for write op
0130 02:58:17.738022 (+ 29us) tablet.cc:683] Acquiring locks for 3 operations
0130 02:58:17.738334 (+ 312us) tablet.cc:699] Row locks acquired
0130 02:58:17.738352 (+ 18us) write_op.cc:260] PREPARE: finished
0130 02:58:17.738500 (+ 148us) write_op.cc:270] Start()
0130 02:58:17.738617 (+ 117us) write_op.cc:276] Timestamp: P: 1769741897738457 usec, L: 0
0130 02:58:17.738633 (+ 16us) op_driver.cc:348] REPLICATION: starting
0130 02:58:17.739195 (+ 562us) log.cc:844] Serialized 1293 byte log entry
0130 02:58:17.748897 (+ 9702us) op_driver.cc:464] REPLICATION: finished
0130 02:58:17.749228 (+ 331us) write_op.cc:301] APPLY: starting
0130 02:58:17.749283 (+ 55us) tablet.cc:1365] starting BulkCheckPresence
0130 02:58:17.749405 (+ 122us) tablet.cc:1368] finished BulkCheckPresence
0130 02:58:17.749491 (+ 86us) tablet.cc:1370] starting ApplyRowOperation cycle
0130 02:58:17.749828 (+ 337us) tablet.cc:1381] finished ApplyRowOperation cycle
0130 02:58:17.749977 (+ 149us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=3
0130 02:58:17.750020 (+ 43us) write_op.cc:312] APPLY: finished
0130 02:58:17.750334 (+ 314us) log.cc:844] Serialized 35 byte log entry
0130 02:58:17.750487 (+ 153us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:58:17.750599 (+ 112us) write_op.cc:454] Released schema lock
0130 02:58:17.750652 (+ 53us) write_op.cc:341] FINISH: Updating metrics
0130 02:58:17.750939 (+ 287us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:58:17.750970 (+ 31us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":274291,"reported_tablets":50,"child_traces":[["op",{"apply.queue_time_us":183,"apply.run_cpu_time_us":1660,"apply.run_wall_time_us":1656,"num_ops":3,"prepare.queue_time_us":99,"prepare.run_cpu_time_us":2512,"prepare.run_wall_time_us":2513,"raft.queue_time_us":992,"raft.run_cpu_time_us":652,"raft.run_wall_time_us":653,"replication_time_us":10166}]]}
W20260130 02:58:18.922220 18867 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.18.100.66:59691 (request call id 33) took 4228 ms (client timeout 1000 ms). Trace:
I20260130 02:58:18.942823 18866 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50284:
table { table_name: "default.Table_a675d7371dcd46b293f893ebe4939e7a" } modify_external_catalogs: true
I20260130 02:58:18.943818 18866 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50284:
table { table_name: "default.Table_a675d7371dcd46b293f893ebe4939e7a" } modify_external_catalogs: true
I20260130 02:58:18.947053 19353 tablet_service.cc:1552] Processing DeleteTablet for tablet 04f4ac27800c4ac3a79ae12d35479757 with delete_type TABLET_DATA_DELETED (replaced by tablet 3738385130c044e0a5a5803da74c567d) from {username='slave'} at 127.0.0.1:60110
I20260130 02:58:18.948659 19496 tablet_service.cc:1552] Processing DeleteTablet for tablet 04f4ac27800c4ac3a79ae12d35479757 with delete_type TABLET_DATA_DELETED (replaced by tablet 3738385130c044e0a5a5803da74c567d) from {username='slave'} at 127.0.0.1:59068
I20260130 02:58:18.946061 18868 catalog_manager.cc:6724] ReplaceTablet: tablet 04f4ac27800c4ac3a79ae12d35479757 deleted and replaced by tablet 3738385130c044e0a5a5803da74c567d
I20260130 02:58:18.949894 18868 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:50284 (request call id 248) took 2140 ms. Trace:
I20260130 02:58:18.895660 19625 tablet_bootstrap.cc:654] T e12d130bcdf34965888753f12c1ff848 P d870164263bb44b6b780a557daea3656: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:18.916282 19503 tablet_service.cc:1552] Processing DeleteTablet for tablet 7d96e93d95f14af79cfb268626559b1e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:58:04 UTC) from {username='slave'} at 127.0.0.1:59068
I20260130 02:58:18.921569 19525 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2b0edaad6877460cb7982efc62b00b66" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
I20260130 02:58:18.950438 18868 rpcz_store.cc:276] 0130 02:58:16.809041 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:58:18.734752 (+1925711us) service_pool.cc:225] Handling call
0130 02:58:18.949835 (+215083us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0130 02:58:18.810815 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0130 02:58:18.811287 (+ 472us) write_op.cc:432] Acquiring schema lock in shared mode
0130 02:58:18.811317 (+ 30us) write_op.cc:435] Acquired schema lock
0130 02:58:18.811333 (+ 16us) tablet.cc:660] Decoding operations
0130 02:58:18.811678 (+ 345us) write_op.cc:620] Acquiring the partition lock for write op
0130 02:58:18.811724 (+ 46us) write_op.cc:641] Partition lock acquired for write op
0130 02:58:18.811747 (+ 23us) tablet.cc:683] Acquiring locks for 2 operations
0130 02:58:18.811954 (+ 207us) tablet.cc:699] Row locks acquired
0130 02:58:18.811970 (+ 16us) write_op.cc:260] PREPARE: finished
0130 02:58:18.812098 (+ 128us) write_op.cc:270] Start()
0130 02:58:18.812185 (+ 87us) write_op.cc:276] Timestamp: P: 1769741898812049 usec, L: 0
0130 02:58:18.812199 (+ 14us) op_driver.cc:348] REPLICATION: starting
0130 02:58:18.812755 (+ 556us) log.cc:844] Serialized 662 byte log entry
0130 02:58:18.870544 (+ 57789us) op_driver.cc:464] REPLICATION: finished
0130 02:58:18.870900 (+ 356us) write_op.cc:301] APPLY: starting
0130 02:58:18.870957 (+ 57us) tablet.cc:1365] starting BulkCheckPresence
0130 02:58:18.871073 (+ 116us) tablet.cc:1368] finished BulkCheckPresence
0130 02:58:18.871198 (+ 125us) tablet.cc:1370] starting ApplyRowOperation cycle
0130 02:58:18.871646 (+ 448us) tablet.cc:1381] finished ApplyRowOperation cycle
0130 02:58:18.871863 (+ 217us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0130 02:58:18.871909 (+ 46us) write_op.cc:312] APPLY: finished
0130 02:58:18.872093 (+ 184us) log.cc:844] Serialized 29 byte log entry
0130 02:58:18.872245 (+ 152us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:58:18.872352 (+ 107us) write_op.cc:454] Released schema lock
0130 02:58:18.872388 (+ 36us) write_op.cc:341] FINISH: Updating metrics
0130 02:58:18.872625 (+ 237us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:58:18.872649 (+ 24us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":168,"apply.run_cpu_time_us":1687,"apply.run_wall_time_us":1687,"num_ops":2,"prepare.queue_time_us":72750,"prepare.run_cpu_time_us":2239,"prepare.run_wall_time_us":2230,"raft.queue_time_us":1196,"raft.run_cpu_time_us":734,"raft.run_wall_time_us":734,"replication_time_us":58223}]]}
I20260130 02:58:18.954213 18868 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50284:
name: "default.Table_0ca1d7370d2f416782b386c5b47111ab"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260130 02:58:18.955706 19313 leader_election.cc:343] T 2b0edaad6877460cb7982efc62b00b66 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): Illegal state: must be running to vote when last-logged opid is not known
W20260130 02:58:18.956326 18868 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0ca1d7370d2f416782b386c5b47111ab in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260130 02:58:18.945307 18870 rpcz_store.cc:269] 0130 02:58:14.693433 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:58:15.629643 (+936210us) service_pool.cc:225] Handling call
0130 02:58:18.913275 (+3283632us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0130 02:58:16.766194 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0130 02:58:16.766777 (+ 583us) write_op.cc:432] Acquiring schema lock in shared mode
0130 02:58:16.766801 (+ 24us) write_op.cc:435] Acquired schema lock
0130 02:58:16.766815 (+ 14us) tablet.cc:660] Decoding operations
0130 02:58:16.767157 (+ 342us) write_op.cc:620] Acquiring the partition lock for write op
0130 02:58:16.767195 (+ 38us) write_op.cc:641] Partition lock acquired for write op
0130 02:58:16.767219 (+ 24us) tablet.cc:683] Acquiring locks for 2 operations
0130 02:58:16.767490 (+ 271us) tablet.cc:699] Row locks acquired
0130 02:58:16.767507 (+ 17us) write_op.cc:260] PREPARE: finished
0130 02:58:16.767617 (+ 110us) write_op.cc:270] Start()
0130 02:58:16.767706 (+ 89us) write_op.cc:276] Timestamp: P: 1769741896767569 usec, L: 0
0130 02:58:16.767722 (+ 16us) op_driver.cc:348] REPLICATION: starting
0130 02:58:16.768397 (+ 675us) log.cc:844] Serialized 913 byte log entry
0130 02:58:16.875904 (+107507us) op_driver.cc:464] REPLICATION: finished
0130 02:58:16.878044 (+ 2140us) write_op.cc:301] APPLY: starting
0130 02:58:16.878092 (+ 48us) tablet.cc:1365] starting BulkCheckPresence
0130 02:58:16.878173 (+ 81us) tablet.cc:1368] finished BulkCheckPresence
0130 02:58:16.878344 (+ 171us) tablet.cc:1370] starting ApplyRowOperation cycle
0130 02:58:16.878566 (+ 222us) tablet.cc:1381] finished ApplyRowOperation cycle
0130 02:58:16.878703 (+ 137us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0130 02:58:16.878743 (+ 40us) write_op.cc:312] APPLY: finished
0130 02:58:16.878914 (+ 171us) log.cc:844] Serialized 29 byte log entry
0130 02:58:16.879034 (+ 120us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:58:16.879089 (+ 55us) write_op.cc:454] Released schema lock
0130 02:58:16.879246 (+ 157us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"mutex_wait_us":122322,"reported_tablets":49,"child_traces":[["op",{"apply.queue_time_us":2029,"apply.run_cpu_time_us":1435,"apply.run_wall_time_us":16195,"mutex_wait_us":8266,"num_ops":2,"prepare.queue_time_us":8357,"prepare.run_cpu_time_us":2136,"prepare.run_wall_time_us":3379,"replication_time_us":108109,"thread_start_us":82303,"threads_started":1}]]}
I20260130 02:58:18.963114 19807 tablet_bootstrap.cc:492] T af0f90071a334016aa17fae2e3a97fc8 P 371a36d219164974abdf848841528152: Bootstrap starting.
W20260130 02:58:18.940379 18867 rpcz_store.cc:269] 0130 02:58:14.693433 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:58:15.582412 (+888979us) service_pool.cc:225] Handling call
0130 02:58:16.103330 (+520918us) spinlock_profiling.cc:243] Waited 38.1 ms on lock 0x7b5400001a90. stack: 00007fb0bd3f7373 00007fb0bd3f7270 00007fb0bcf6c574 0000000000508744 00000000005086d9 00007fb0bd2698b1 00007fb0bd3ec880 00007fb0c61dbad6 00007fb0c62e19ce 00007fb0c3efe4a4 00007fb0c3efe432 00007fb0c3efe361 00007fb0c3efe2dc 00007fb0c3efd592 00007fb0c21dd34c 00007fb0c21dc786
0130 02:58:17.493940 (+1390610us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:58:17.493966 (+ 26us) write_op.cc:454] Released schema lock
0130 02:58:18.922147 (+1428181us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0130 02:58:17.473116 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0130 02:58:17.473594 (+ 478us) write_op.cc:432] Acquiring schema lock in shared mode
0130 02:58:17.473624 (+ 30us) write_op.cc:435] Acquired schema lock
0130 02:58:17.473640 (+ 16us) tablet.cc:660] Decoding operations
0130 02:58:17.473925 (+ 285us) write_op.cc:620] Acquiring the partition lock for write op
0130 02:58:17.473970 (+ 45us) write_op.cc:641] Partition lock acquired for write op
0130 02:58:17.473997 (+ 27us) tablet.cc:683] Acquiring locks for 1 operations
0130 02:58:17.474149 (+ 152us) tablet.cc:699] Row locks acquired
0130 02:58:17.474164 (+ 15us) write_op.cc:260] PREPARE: finished
0130 02:58:17.474290 (+ 126us) write_op.cc:270] Start()
0130 02:58:17.474396 (+ 106us) write_op.cc:276] Timestamp: P: 1769741897474222 usec, L: 0
0130 02:58:17.474411 (+ 15us) op_driver.cc:348] REPLICATION: starting
0130 02:58:17.475028 (+ 617us) log.cc:844] Serialized 533 byte log entry
0130 02:58:17.482704 (+ 7676us) op_driver.cc:464] REPLICATION: finished
0130 02:58:17.483145 (+ 441us) write_op.cc:301] APPLY: starting
0130 02:58:17.483196 (+ 51us) tablet.cc:1365] starting BulkCheckPresence
0130 02:58:17.483286 (+ 90us) tablet.cc:1368] finished BulkCheckPresence
0130 02:58:17.483368 (+ 82us) tablet.cc:1370] starting ApplyRowOperation cycle
0130 02:58:17.483517 (+ 149us) tablet.cc:1381] finished ApplyRowOperation cycle
0130 02:58:17.483638 (+ 121us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0130 02:58:17.483681 (+ 43us) write_op.cc:312] APPLY: finished
0130 02:58:17.483841 (+ 160us) log.cc:844] Serialized 23 byte log entry
0130 02:58:17.483975 (+ 134us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:58:17.484054 (+ 79us) write_op.cc:454] Released schema lock
0130 02:58:17.484087 (+ 33us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"mutex_wait_us":77136,"reported_tablets":57,"spinlock_wait_cycles":87661824,"child_traces":[["op",{"apply.queue_time_us":298,"apply.run_cpu_time_us":1145,"apply.run_wall_time_us":1143,"num_ops":1,"prepare.queue_time_us":31060,"prepare.run_cpu_time_us":2206,"prepare.run_wall_time_us":2206,"raft.queue_time_us":914,"raft.run_cpu_time_us":599,"raft.run_wall_time_us":599,"replication_time_us":8192}]]}
W20260130 02:58:18.957334 19159 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.100.66:40767, user_credentials={real_user=slave}} blocked reactor thread for 178010us
I20260130 02:58:18.966652 18867 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50284:
name: "default.Table_f619de75da8e4d23b4d641d30285e0e7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260130 02:58:18.968237 18867 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f619de75da8e4d23b4d641d30285e0e7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260130 02:58:18.974808 18871 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50284:
name: "default.Table_7fe8c354c0dc4cfba0e84ed33e4c2506"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260130 02:58:18.976529 18871 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7fe8c354c0dc4cfba0e84ed33e4c2506 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260130 02:58:18.979779 19222 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3aa76993beb94b12bc706a76db26a03b" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152"
I20260130 02:58:18.980314 19222 raft_consensus.cc:3060] T 3aa76993beb94b12bc706a76db26a03b P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:18.960356 18870 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50284:
name: "default.Table_2414288f0dc54e5f835f876060927a9b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260130 02:58:18.981158 19314 leader_election.cc:343] T 2b0edaad6877460cb7982efc62b00b66 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:18.981544 19314 leader_election.cc:304] T 2b0edaad6877460cb7982efc62b00b66 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78c2c46c7520492db35d51c244fb4f74; no voters: 371a36d219164974abdf848841528152, d870164263bb44b6b780a557daea3656
W20260130 02:58:18.981815 18870 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2414288f0dc54e5f835f876060927a9b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260130 02:58:18.983186 20697 raft_consensus.cc:2749] T 2b0edaad6877460cb7982efc62b00b66 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260130 02:58:18.985153 19204 tablet_service.cc:1552] Processing DeleteTablet for tablet 04f4ac27800c4ac3a79ae12d35479757 with delete_type TABLET_DATA_DELETED (replaced by tablet 3738385130c044e0a5a5803da74c567d) from {username='slave'} at 127.0.0.1:42870
I20260130 02:58:18.988741 19840 tablet_bootstrap.cc:492] T cfd12a72726846e587cff9d6a4d61bd6 P 78c2c46c7520492db35d51c244fb4f74: No bootstrap required, opened a new log
I20260130 02:58:18.989140 19840 ts_tablet_manager.cc:1403] T cfd12a72726846e587cff9d6a4d61bd6 P 78c2c46c7520492db35d51c244fb4f74: Time spent bootstrapping tablet: real 0.119s user 0.017s sys 0.076s
I20260130 02:58:18.992206 19840 raft_consensus.cc:359] T cfd12a72726846e587cff9d6a4d61bd6 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:18.993072 19840 raft_consensus.cc:385] T cfd12a72726846e587cff9d6a4d61bd6 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:18.993312 19840 raft_consensus.cc:740] T cfd12a72726846e587cff9d6a4d61bd6 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78c2c46c7520492db35d51c244fb4f74, State: Initialized, Role: FOLLOWER
I20260130 02:58:18.994038 19840 consensus_queue.cc:260] T cfd12a72726846e587cff9d6a4d61bd6 P 78c2c46c7520492db35d51c244fb4f74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:18.994750 20616 raft_consensus.cc:493] T 13f0ace4d9de43abab08eca32cdb30c1 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:18.995213 20616 raft_consensus.cc:515] T 13f0ace4d9de43abab08eca32cdb30c1 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:18.996452 19840 ts_tablet_manager.cc:1434] T cfd12a72726846e587cff9d6a4d61bd6 P 78c2c46c7520492db35d51c244fb4f74: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260130 02:58:18.996999 20616 leader_election.cc:290] T 13f0ace4d9de43abab08eca32cdb30c1 P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767), d870164263bb44b6b780a557daea3656 (127.18.100.67:39571)
I20260130 02:58:19.001061 19526 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "25fd48e693424537b91ba7c3fcc96878" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656"
I20260130 02:58:19.001571 19526 raft_consensus.cc:3060] T 25fd48e693424537b91ba7c3fcc96878 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:19.009227 19528 raft_consensus.cc:3060] T cf1b66a20fce4c4d96bc0bc32a7bafa8 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:19.023710 19222 raft_consensus.cc:2468] T 3aa76993beb94b12bc706a76db26a03b P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d870164263bb44b6b780a557daea3656 in term 1.
I20260130 02:58:19.047892 19463 leader_election.cc:304] T 3aa76993beb94b12bc706a76db26a03b P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 371a36d219164974abdf848841528152, d870164263bb44b6b780a557daea3656; no voters:
I20260130 02:58:19.057349 19526 raft_consensus.cc:2468] T 25fd48e693424537b91ba7c3fcc96878 P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 371a36d219164974abdf848841528152 in term 1.
I20260130 02:58:19.060166 19516 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "13f0ace4d9de43abab08eca32cdb30c1" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
I20260130 02:58:19.061409 19527 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3aa76993beb94b12bc706a76db26a03b" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
I20260130 02:58:19.062525 19520 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e12d130bcdf34965888753f12c1ff848" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
I20260130 02:58:19.070720 20929 raft_consensus.cc:2804] T 3aa76993beb94b12bc706a76db26a03b P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Leader election won for term 1
I20260130 02:58:19.071518 20929 raft_consensus.cc:697] T 3aa76993beb94b12bc706a76db26a03b P d870164263bb44b6b780a557daea3656 [term 1 LEADER]: Becoming Leader. State: Replica: d870164263bb44b6b780a557daea3656, State: Running, Role: LEADER
I20260130 02:58:19.075311 19368 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3aa76993beb94b12bc706a76db26a03b" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74" is_pre_election: true
I20260130 02:58:19.075760 20704 raft_consensus.cc:493] T e12d130bcdf34965888753f12c1ff848 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:19.075928 19368 raft_consensus.cc:2468] T 3aa76993beb94b12bc706a76db26a03b P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 371a36d219164974abdf848841528152 in term 0.
I20260130 02:58:19.076190 20704 raft_consensus.cc:515] T e12d130bcdf34965888753f12c1ff848 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:19.077970 20929 consensus_queue.cc:237] T 3aa76993beb94b12bc706a76db26a03b P d870164263bb44b6b780a557daea3656 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:19.079000 20704 leader_election.cc:290] T e12d130bcdf34965888753f12c1ff848 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d870164263bb44b6b780a557daea3656 (127.18.100.67:39571), 371a36d219164974abdf848841528152 (127.18.100.65:42021)
I20260130 02:58:19.082005 19370 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e12d130bcdf34965888753f12c1ff848" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74" is_pre_election: true
I20260130 02:58:19.083629 19364 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "13f0ace4d9de43abab08eca32cdb30c1" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74" is_pre_election: true
I20260130 02:58:19.084652 19364 raft_consensus.cc:2468] T 13f0ace4d9de43abab08eca32cdb30c1 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 371a36d219164974abdf848841528152 in term 0.
I20260130 02:58:19.063434 20899 ts_tablet_manager.cc:1929] T c7c621b99deb457998c44ab1644d3939 P 78c2c46c7520492db35d51c244fb4f74: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260130 02:58:19.087647 20899 log.cc:1199] T c7c621b99deb457998c44ab1644d3939 P 78c2c46c7520492db35d51c244fb4f74: Deleting WAL directory at /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-1/wal/wals/c7c621b99deb457998c44ab1644d3939
I20260130 02:58:19.088445 20899 ts_tablet_manager.cc:1950] T c7c621b99deb457998c44ab1644d3939 P 78c2c46c7520492db35d51c244fb4f74: Deleting consensus metadata
I20260130 02:58:19.089669 19374 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "25fd48e693424537b91ba7c3fcc96878" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74"
I20260130 02:58:19.090157 19374 raft_consensus.cc:3060] T 25fd48e693424537b91ba7c3fcc96878 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:19.084676 19379 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3aa76993beb94b12bc706a76db26a03b" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74"
I20260130 02:58:19.092060 19379 raft_consensus.cc:3060] T 3aa76993beb94b12bc706a76db26a03b P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:19.094856 19370 raft_consensus.cc:2468] T e12d130bcdf34965888753f12c1ff848 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 371a36d219164974abdf848841528152 in term 0.
I20260130 02:58:19.097371 19520 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e12d130bcdf34965888753f12c1ff848" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
W20260130 02:58:19.099308 19313 leader_election.cc:343] T e12d130bcdf34965888753f12c1ff848 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:19.102144 19374 raft_consensus.cc:2468] T 25fd48e693424537b91ba7c3fcc96878 P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 371a36d219164974abdf848841528152 in term 1.
I20260130 02:58:19.105659 19840 tablet_bootstrap.cc:492] T c76f0dc7762c49c389fa6fef41f02141 P 78c2c46c7520492db35d51c244fb4f74: Bootstrap starting.
I20260130 02:58:19.109930 19379 raft_consensus.cc:2468] T 3aa76993beb94b12bc706a76db26a03b P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d870164263bb44b6b780a557daea3656 in term 1.
I20260130 02:58:19.112183 19840 tablet_bootstrap.cc:654] T c76f0dc7762c49c389fa6fef41f02141 P 78c2c46c7520492db35d51c244fb4f74: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:19.121937 20704 raft_consensus.cc:493] T cd30d7adcc4545848e3501a205a10293 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:19.122757 20704 raft_consensus.cc:515] T cd30d7adcc4545848e3501a205a10293 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:19.123723 18860 catalog_manager.cc:4985] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767): tablet c7c621b99deb457998c44ab1644d3939 (table default.Table_3413dae2efac41488448e6e5975cf710 [id=7a08fa3986d0410d81de0235d2fb0f92]) successfully deleted
W20260130 02:58:19.125288 18860 catalog_manager.cc:4952] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767): delete failed for tablet ec28ff12927d450ead16d26cf016138d because the tablet was not found. No further retry: Not found: Tablet not found: ec28ff12927d450ead16d26cf016138d
I20260130 02:58:19.126060 20899 tablet_replica.cc:333] T 78e9bc3e55fe4e4a98e8eb01da59d365 P 78c2c46c7520492db35d51c244fb4f74: stopping tablet replica
I20260130 02:58:19.126641 20704 leader_election.cc:290] T cd30d7adcc4545848e3501a205a10293 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d870164263bb44b6b780a557daea3656 (127.18.100.67:39571), 371a36d219164974abdf848841528152 (127.18.100.65:42021)
I20260130 02:58:19.127108 20899 raft_consensus.cc:2243] T 78e9bc3e55fe4e4a98e8eb01da59d365 P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260130 02:58:19.127629 20899 raft_consensus.cc:2272] T 78e9bc3e55fe4e4a98e8eb01da59d365 P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260130 02:58:19.128125 19520 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cd30d7adcc4545848e3501a205a10293" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
W20260130 02:58:19.129324 19313 leader_election.cc:343] T cd30d7adcc4545848e3501a205a10293 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:19.129684 20899 ts_tablet_manager.cc:1916] T 78e9bc3e55fe4e4a98e8eb01da59d365 P 78c2c46c7520492db35d51c244fb4f74: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260130 02:58:19.034559 19807 tablet_bootstrap.cc:654] T af0f90071a334016aa17fae2e3a97fc8 P 371a36d219164974abdf848841528152: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:19.134265 19625 tablet_bootstrap.cc:492] T e12d130bcdf34965888753f12c1ff848 P d870164263bb44b6b780a557daea3656: No bootstrap required, opened a new log
W20260130 02:58:19.036340 19159 leader_election.cc:343] T 3389db58493448aea3cf00aca411047c P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): Not found: Tablet not found: 3389db58493448aea3cf00aca411047c
W20260130 02:58:19.136123 19159 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.100.67:39571, user_credentials={real_user=slave}} blocked reactor thread for 99798.5us
I20260130 02:58:19.083801 19222 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e12d130bcdf34965888753f12c1ff848" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152" is_pre_election: true
I20260130 02:58:19.137843 19222 raft_consensus.cc:2468] T e12d130bcdf34965888753f12c1ff848 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78c2c46c7520492db35d51c244fb4f74 in term 0.
I20260130 02:58:19.140089 19625 ts_tablet_manager.cc:1403] T e12d130bcdf34965888753f12c1ff848 P d870164263bb44b6b780a557daea3656: Time spent bootstrapping tablet: real 0.368s user 0.011s sys 0.000s
I20260130 02:58:19.146281 19159 leader_election.cc:304] T 25fd48e693424537b91ba7c3fcc96878 P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 371a36d219164974abdf848841528152, d870164263bb44b6b780a557daea3656; no voters:
I20260130 02:58:19.133132 19840 tablet_bootstrap.cc:492] T c76f0dc7762c49c389fa6fef41f02141 P 78c2c46c7520492db35d51c244fb4f74: No bootstrap required, opened a new log
I20260130 02:58:19.138834 19314 leader_election.cc:304] T e12d130bcdf34965888753f12c1ff848 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 371a36d219164974abdf848841528152, 78c2c46c7520492db35d51c244fb4f74; no voters: d870164263bb44b6b780a557daea3656
I20260130 02:58:19.147311 19840 ts_tablet_manager.cc:1403] T c76f0dc7762c49c389fa6fef41f02141 P 78c2c46c7520492db35d51c244fb4f74: Time spent bootstrapping tablet: real 0.042s user 0.007s sys 0.003s
W20260130 02:58:19.147284 19159 leader_election.cc:343] T 13f0ace4d9de43abab08eca32cdb30c1 P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): Illegal state: must be running to vote when last-logged opid is not known
W20260130 02:58:19.147794 19159 leader_election.cc:343] T e12d130bcdf34965888753f12c1ff848 P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:19.147951 20697 raft_consensus.cc:2804] T e12d130bcdf34965888753f12c1ff848 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260130 02:58:19.148298 20697 raft_consensus.cc:493] T e12d130bcdf34965888753f12c1ff848 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260130 02:58:19.126410 19221 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cd30d7adcc4545848e3501a205a10293" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152" is_pre_election: true
I20260130 02:58:19.148603 20697 raft_consensus.cc:3060] T e12d130bcdf34965888753f12c1ff848 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:19.149292 20923 raft_consensus.cc:2804] T 25fd48e693424537b91ba7c3fcc96878 P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Leader election won for term 1
I20260130 02:58:19.149655 19840 raft_consensus.cc:359] T c76f0dc7762c49c389fa6fef41f02141 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:19.150308 19840 raft_consensus.cc:385] T c76f0dc7762c49c389fa6fef41f02141 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:19.150619 19840 raft_consensus.cc:740] T c76f0dc7762c49c389fa6fef41f02141 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78c2c46c7520492db35d51c244fb4f74, State: Initialized, Role: FOLLOWER
I20260130 02:58:19.151185 19840 consensus_queue.cc:260] T c76f0dc7762c49c389fa6fef41f02141 P 78c2c46c7520492db35d51c244fb4f74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
W20260130 02:58:19.151598 19314 leader_election.cc:343] T cd30d7adcc4545848e3501a205a10293 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:19.151939 19314 leader_election.cc:304] T cd30d7adcc4545848e3501a205a10293 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78c2c46c7520492db35d51c244fb4f74; no voters: 371a36d219164974abdf848841528152, d870164263bb44b6b780a557daea3656
I20260130 02:58:19.152336 20923 raft_consensus.cc:697] T 25fd48e693424537b91ba7c3fcc96878 P 371a36d219164974abdf848841528152 [term 1 LEADER]: Becoming Leader. State: Replica: 371a36d219164974abdf848841528152, State: Running, Role: LEADER
I20260130 02:58:19.153189 19840 ts_tablet_manager.cc:1434] T c76f0dc7762c49c389fa6fef41f02141 P 78c2c46c7520492db35d51c244fb4f74: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20260130 02:58:19.153133 20898 raft_consensus.cc:2749] T cd30d7adcc4545848e3501a205a10293 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260130 02:58:19.153141 20923 consensus_queue.cc:237] T 25fd48e693424537b91ba7c3fcc96878 P 371a36d219164974abdf848841528152 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:19.154066 19840 tablet_bootstrap.cc:492] T 94355fc1338f479e8798070aff04fcba P 78c2c46c7520492db35d51c244fb4f74: Bootstrap starting.
I20260130 02:58:19.164201 20697 raft_consensus.cc:515] T e12d130bcdf34965888753f12c1ff848 P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:19.175854 19159 leader_election.cc:304] T 3aa76993beb94b12bc706a76db26a03b P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 371a36d219164974abdf848841528152, 78c2c46c7520492db35d51c244fb4f74; no voters:
I20260130 02:58:19.176781 20867 raft_consensus.cc:2764] T 3aa76993beb94b12bc706a76db26a03b P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20260130 02:58:19.167513 19625 raft_consensus.cc:359] T e12d130bcdf34965888753f12c1ff848 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:19.178320 19159 leader_election.cc:304] T 13f0ace4d9de43abab08eca32cdb30c1 P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 371a36d219164974abdf848841528152, 78c2c46c7520492db35d51c244fb4f74; no voters: d870164263bb44b6b780a557daea3656
I20260130 02:58:19.035281 20897 ts_tablet_manager.cc:1929] T c7c621b99deb457998c44ab1644d3939 P 371a36d219164974abdf848841528152: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260130 02:58:19.180287 20897 log.cc:1199] T c7c621b99deb457998c44ab1644d3939 P 371a36d219164974abdf848841528152: Deleting WAL directory at /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-0/wal/wals/c7c621b99deb457998c44ab1644d3939
I20260130 02:58:19.181227 19625 raft_consensus.cc:385] T e12d130bcdf34965888753f12c1ff848 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:19.181954 20897 ts_tablet_manager.cc:1950] T c7c621b99deb457998c44ab1644d3939 P 371a36d219164974abdf848841528152: Deleting consensus metadata
I20260130 02:58:19.183357 18861 catalog_manager.cc:4985] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021): tablet c7c621b99deb457998c44ab1644d3939 (table default.Table_3413dae2efac41488448e6e5975cf710 [id=7a08fa3986d0410d81de0235d2fb0f92]) successfully deleted
I20260130 02:58:19.183851 19625 raft_consensus.cc:740] T e12d130bcdf34965888753f12c1ff848 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d870164263bb44b6b780a557daea3656, State: Initialized, Role: FOLLOWER
I20260130 02:58:19.184442 19625 consensus_queue.cc:260] T e12d130bcdf34965888753f12c1ff848 P d870164263bb44b6b780a557daea3656 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:19.182952 20697 leader_election.cc:290] T e12d130bcdf34965888753f12c1ff848 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 election: Requested vote from peers d870164263bb44b6b780a557daea3656 (127.18.100.67:39571), 371a36d219164974abdf848841528152 (127.18.100.65:42021)
I20260130 02:58:19.186079 19520 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e12d130bcdf34965888753f12c1ff848" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656"
I20260130 02:58:19.186676 19520 raft_consensus.cc:3060] T e12d130bcdf34965888753f12c1ff848 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:19.192955 19625 ts_tablet_manager.cc:1434] T e12d130bcdf34965888753f12c1ff848 P d870164263bb44b6b780a557daea3656: Time spent starting tablet: real 0.051s user 0.004s sys 0.000s
W20260130 02:58:19.207448 19314 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.100.65:42021, user_credentials={real_user=slave}} blocked reactor thread for 55864.4us
I20260130 02:58:19.211021 19625 tablet_bootstrap.cc:492] T af0f90071a334016aa17fae2e3a97fc8 P d870164263bb44b6b780a557daea3656: Bootstrap starting.
I20260130 02:58:19.215291 19840 tablet_bootstrap.cc:654] T 94355fc1338f479e8798070aff04fcba P 78c2c46c7520492db35d51c244fb4f74: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:19.216578 19625 tablet_bootstrap.cc:654] T af0f90071a334016aa17fae2e3a97fc8 P d870164263bb44b6b780a557daea3656: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:19.242522 20697 raft_consensus.cc:493] T cfd12a72726846e587cff9d6a4d61bd6 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:19.243177 20697 raft_consensus.cc:515] T cfd12a72726846e587cff9d6a4d61bd6 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:19.247794 20926 raft_consensus.cc:2804] T 13f0ace4d9de43abab08eca32cdb30c1 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260130 02:58:19.248219 20926 raft_consensus.cc:493] T 13f0ace4d9de43abab08eca32cdb30c1 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260130 02:58:19.248550 20926 raft_consensus.cc:3060] T 13f0ace4d9de43abab08eca32cdb30c1 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:19.249670 19520 raft_consensus.cc:2468] T e12d130bcdf34965888753f12c1ff848 P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78c2c46c7520492db35d51c244fb4f74 in term 1.
I20260130 02:58:19.251139 20899 ts_tablet_manager.cc:1929] T 78e9bc3e55fe4e4a98e8eb01da59d365 P 78c2c46c7520492db35d51c244fb4f74: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260130 02:58:19.252310 20899 log.cc:1199] T 78e9bc3e55fe4e4a98e8eb01da59d365 P 78c2c46c7520492db35d51c244fb4f74: Deleting WAL directory at /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-1/wal/wals/78e9bc3e55fe4e4a98e8eb01da59d365
I20260130 02:58:19.252513 19313 leader_election.cc:304] T e12d130bcdf34965888753f12c1ff848 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78c2c46c7520492db35d51c244fb4f74, d870164263bb44b6b780a557daea3656; no voters:
I20260130 02:58:19.253422 20704 raft_consensus.cc:2804] T e12d130bcdf34965888753f12c1ff848 P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Leader election won for term 1
I20260130 02:58:19.253847 20704 raft_consensus.cc:697] T e12d130bcdf34965888753f12c1ff848 P 78c2c46c7520492db35d51c244fb4f74 [term 1 LEADER]: Becoming Leader. State: Replica: 78c2c46c7520492db35d51c244fb4f74, State: Running, Role: LEADER
I20260130 02:58:19.254755 20704 consensus_queue.cc:237] T e12d130bcdf34965888753f12c1ff848 P 78c2c46c7520492db35d51c244fb4f74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
W20260130 02:58:19.268491 19159 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.100.66:40767, user_credentials={real_user=slave}} blocked reactor thread for 72899us
I20260130 02:58:19.274151 19221 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e12d130bcdf34965888753f12c1ff848" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152"
I20260130 02:58:19.274752 19221 raft_consensus.cc:3060] T e12d130bcdf34965888753f12c1ff848 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:19.274314 20867 consensus_queue.cc:1048] T 6218f2f7d73a42428b57a45858f53afd P 371a36d219164974abdf848841528152 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260130 02:58:19.282837 20899 ts_tablet_manager.cc:1950] T 78e9bc3e55fe4e4a98e8eb01da59d365 P 78c2c46c7520492db35d51c244fb4f74: Deleting consensus metadata
I20260130 02:58:19.284240 18860 catalog_manager.cc:4985] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767): tablet 78e9bc3e55fe4e4a98e8eb01da59d365 (table default.Table_5d29239ce1984d2fad02607a2bdd4dd0 [id=7e271b81bb064a949685b2411b485e3e]) successfully deleted
I20260130 02:58:19.284660 19221 raft_consensus.cc:2468] T e12d130bcdf34965888753f12c1ff848 P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78c2c46c7520492db35d51c244fb4f74 in term 1.
I20260130 02:58:19.289551 20864 consensus_queue.cc:1048] T f9e5f01e69254d658c369f3e21cef523 P d870164263bb44b6b780a557daea3656 [LEADER]: Connected to new peer: Peer: permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260130 02:58:19.297224 19159 leader_election.cc:343] T 3389db58493448aea3cf00aca411047c P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767): Not found: Tablet not found: 3389db58493448aea3cf00aca411047c
I20260130 02:58:19.297648 19159 leader_election.cc:304] T 3389db58493448aea3cf00aca411047c P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 371a36d219164974abdf848841528152; no voters: 78c2c46c7520492db35d51c244fb4f74, d870164263bb44b6b780a557daea3656
W20260130 02:58:19.299513 18861 catalog_manager.cc:4952] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021): delete failed for tablet d55291d7cea64a0d9a345ce712fbfa80 because the tablet was not found. No further retry: Not found: Tablet not found: d55291d7cea64a0d9a345ce712fbfa80
W20260130 02:58:19.300794 18861 catalog_manager.cc:4952] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021): delete failed for tablet 38a7a10e36874183bf9fe55a38d829f6 because the tablet was not found. No further retry: Not found: Tablet not found: 38a7a10e36874183bf9fe55a38d829f6
W20260130 02:58:19.302559 18861 catalog_manager.cc:4952] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021): delete failed for tablet d27ca7dd87f44102b7680e1f4a88954a because the tablet was not found. No further retry: Not found: Tablet not found: d27ca7dd87f44102b7680e1f4a88954a
I20260130 02:58:19.304680 20897 tablet_replica.cc:333] T 3389db58493448aea3cf00aca411047c P 371a36d219164974abdf848841528152: stopping tablet replica
I20260130 02:58:19.305485 18872 catalog_manager.cc:5746] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021) does not have the latest schema for tablet a8a5038e9def44eb8b3b1e4aeae25652 (table default.Table_dee034aec0b24982b8dc2a3fa1b326f1 [id=66033e75b3814a8584311cf7e4a07ad3]). Expected version 1 got 0
I20260130 02:58:19.305770 20925 raft_consensus.cc:2749] T 3389db58493448aea3cf00aca411047c P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260130 02:58:19.309845 19159 leader_election.cc:304] T e12d130bcdf34965888753f12c1ff848 P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 371a36d219164974abdf848841528152, 78c2c46c7520492db35d51c244fb4f74; no voters: d870164263bb44b6b780a557daea3656
I20260130 02:58:19.311748 20924 raft_consensus.cc:2764] T e12d130bcdf34965888753f12c1ff848 P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20260130 02:58:19.318529 20920 raft_consensus.cc:515] T 838e49e5e8874db393bdc15a1c92f87e P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:19.321439 20916 ts_tablet_manager.cc:1929] T c7c621b99deb457998c44ab1644d3939 P d870164263bb44b6b780a557daea3656: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260130 02:58:19.321844 20916 log.cc:1199] T c7c621b99deb457998c44ab1644d3939 P d870164263bb44b6b780a557daea3656: Deleting WAL directory at /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-2/wal/wals/c7c621b99deb457998c44ab1644d3939
I20260130 02:58:19.327916 20916 ts_tablet_manager.cc:1950] T c7c621b99deb457998c44ab1644d3939 P d870164263bb44b6b780a557daea3656: Deleting consensus metadata
I20260130 02:58:19.339135 19625 tablet_bootstrap.cc:492] T af0f90071a334016aa17fae2e3a97fc8 P d870164263bb44b6b780a557daea3656: No bootstrap required, opened a new log
I20260130 02:58:19.339646 19625 ts_tablet_manager.cc:1403] T af0f90071a334016aa17fae2e3a97fc8 P d870164263bb44b6b780a557daea3656: Time spent bootstrapping tablet: real 0.129s user 0.010s sys 0.000s
W20260130 02:58:19.341995 19462 consensus_peers.cc:597] T 78e9bc3e55fe4e4a98e8eb01da59d365 P d870164263bb44b6b780a557daea3656 -> Peer 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767): Couldn't send request to peer 78c2c46c7520492db35d51c244fb4f74. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 78e9bc3e55fe4e4a98e8eb01da59d365. This is attempt 1: this message will repeat every 5th retry.
I20260130 02:58:19.342350 20704 raft_consensus.cc:493] T 0855d88806914a36b4bf6b49036f0ad0 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:19.342903 20704 raft_consensus.cc:515] T 0855d88806914a36b4bf6b49036f0ad0 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:19.345497 20704 leader_election.cc:290] T 0855d88806914a36b4bf6b49036f0ad0 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d870164263bb44b6b780a557daea3656 (127.18.100.67:39571), 371a36d219164974abdf848841528152 (127.18.100.65:42021)
I20260130 02:58:19.347556 18867 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:50284 (request call id 253) took 2187 ms. Trace:
I20260130 02:58:19.347872 18867 rpcz_store.cc:276] 0130 02:58:17.160580 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:58:18.966260 (+1805680us) service_pool.cc:225] Handling call
0130 02:58:19.322331 (+356071us) catalog_manager.cc:2101] Acquired catalog manager lock
0130 02:58:19.323749 (+ 1418us) catalog_manager.cc:2147] Created new table and tablet info
0130 02:58:19.346747 (+ 22998us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:58:19.346772 (+ 25us) write_op.cc:454] Released schema lock
0130 02:58:19.346973 (+ 201us) catalog_manager.cc:2207] Wrote table and tablets to system table
0130 02:58:19.347244 (+ 271us) catalog_manager.cc:2241] Inserted table and tablets into CatalogManager maps
0130 02:58:19.347488 (+ 244us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0130 02:58:19.324976 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0130 02:58:19.325418 (+ 442us) write_op.cc:432] Acquiring schema lock in shared mode
0130 02:58:19.325446 (+ 28us) write_op.cc:435] Acquired schema lock
0130 02:58:19.325462 (+ 16us) tablet.cc:660] Decoding operations
0130 02:58:19.325768 (+ 306us) write_op.cc:620] Acquiring the partition lock for write op
0130 02:58:19.325814 (+ 46us) write_op.cc:641] Partition lock acquired for write op
0130 02:58:19.325839 (+ 25us) tablet.cc:683] Acquiring locks for 2 operations
0130 02:58:19.326044 (+ 205us) tablet.cc:699] Row locks acquired
0130 02:58:19.326061 (+ 17us) write_op.cc:260] PREPARE: finished
0130 02:58:19.326203 (+ 142us) write_op.cc:270] Start()
0130 02:58:19.326289 (+ 86us) write_op.cc:276] Timestamp: P: 1769741899326122 usec, L: 0
0130 02:58:19.326309 (+ 20us) op_driver.cc:348] REPLICATION: starting
0130 02:58:19.326941 (+ 632us) log.cc:844] Serialized 433 byte log entry
0130 02:58:19.335908 (+ 8967us) op_driver.cc:464] REPLICATION: finished
0130 02:58:19.336207 (+ 299us) write_op.cc:301] APPLY: starting
0130 02:58:19.336262 (+ 55us) tablet.cc:1365] starting BulkCheckPresence
0130 02:58:19.336358 (+ 96us) tablet.cc:1368] finished BulkCheckPresence
0130 02:58:19.336456 (+ 98us) tablet.cc:1370] starting ApplyRowOperation cycle
0130 02:58:19.336851 (+ 395us) tablet.cc:1381] finished ApplyRowOperation cycle
0130 02:58:19.336998 (+ 147us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0130 02:58:19.337042 (+ 44us) write_op.cc:312] APPLY: finished
0130 02:58:19.337218 (+ 176us) log.cc:844] Serialized 29 byte log entry
0130 02:58:19.337358 (+ 140us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:58:19.337597 (+ 239us) write_op.cc:454] Released schema lock
0130 02:58:19.337631 (+ 34us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":164,"apply.run_cpu_time_us":1595,"apply.run_wall_time_us":1593,"num_ops":2,"prepare.queue_time_us":95,"prepare.run_cpu_time_us":2243,"prepare.run_wall_time_us":2243,"raft.queue_time_us":1071,"raft.run_cpu_time_us":652,"raft.run_wall_time_us":652,"replication_time_us":9500}]]}
I20260130 02:58:19.353472 20920 leader_election.cc:290] T 838e49e5e8874db393bdc15a1c92f87e P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 election: Requested vote from peers 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767), d870164263bb44b6b780a557daea3656 (127.18.100.67:39571)
I20260130 02:58:19.261669 20926 raft_consensus.cc:515] T 13f0ace4d9de43abab08eca32cdb30c1 P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:19.346616 19221 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0855d88806914a36b4bf6b49036f0ad0" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152" is_pre_election: true
W20260130 02:58:19.354931 19314 leader_election.cc:343] T 0855d88806914a36b4bf6b49036f0ad0 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:19.355680 20926 leader_election.cc:290] T 13f0ace4d9de43abab08eca32cdb30c1 P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 election: Requested vote from peers 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767), d870164263bb44b6b780a557daea3656 (127.18.100.67:39571)
I20260130 02:58:19.348718 20897 raft_consensus.cc:2243] T 3389db58493448aea3cf00aca411047c P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260130 02:58:19.356459 20897 raft_consensus.cc:2272] T 3389db58493448aea3cf00aca411047c P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260130 02:58:19.359141 20897 ts_tablet_manager.cc:1916] T 3389db58493448aea3cf00aca411047c P 371a36d219164974abdf848841528152: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260130 02:58:19.363461 19528 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0855d88806914a36b4bf6b49036f0ad0" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
W20260130 02:58:19.364799 19313 leader_election.cc:343] T 0855d88806914a36b4bf6b49036f0ad0 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:19.365195 19313 leader_election.cc:304] T 0855d88806914a36b4bf6b49036f0ad0 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78c2c46c7520492db35d51c244fb4f74; no voters: 371a36d219164974abdf848841528152, d870164263bb44b6b780a557daea3656
I20260130 02:58:19.365890 19503 tablet_service.cc:1505] Processing CreateTablet for tablet 0d14fb95ace9447f810ccbed4d5befdf (DEFAULT_TABLE table=default.Table_c238448062ad4cef8ddd0bd199d5e7f2 [id=e8d3be5336974867a5d65d1ceb4433d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:19.367370 19270 heartbeater.cc:344] Connected to a master server at 127.18.100.124:34629
I20260130 02:58:19.373238 20913 tablet.cc:1722] T 127a84346b9f473d9e7f739b1e65feb8 P 371a36d219164974abdf848841528152: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 6 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 7
I20260130 02:58:19.376865 20942 raft_consensus.cc:2749] T 0855d88806914a36b4bf6b49036f0ad0 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260130 02:58:19.384665 19360 tablet_service.cc:1505] Processing CreateTablet for tablet 0d14fb95ace9447f810ccbed4d5befdf (DEFAULT_TABLE table=default.Table_c238448062ad4cef8ddd0bd199d5e7f2 [id=e8d3be5336974867a5d65d1ceb4433d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:19.385525 20913 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.AlterSchema from 127.0.0.1:42870 (request call id 323) took 1020 ms. Trace:
I20260130 02:58:19.385721 19360 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0d14fb95ace9447f810ccbed4d5befdf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:19.394202 19496 tablet_service.cc:1505] Processing CreateTablet for tablet 9896ff9364744857968e87e80e776dd2 (DEFAULT_TABLE table=default.Table_5d5e26f93a5a4fb084831add317d90f9 [id=f237b1c90a1743d597d87c5cf1abc89a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:19.397538 19204 tablet_service.cc:1505] Processing CreateTablet for tablet 0d14fb95ace9447f810ccbed4d5befdf (DEFAULT_TABLE table=default.Table_c238448062ad4cef8ddd0bd199d5e7f2 [id=e8d3be5336974867a5d65d1ceb4433d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:19.397828 19202 tablet_service.cc:1505] Processing CreateTablet for tablet 9896ff9364744857968e87e80e776dd2 (DEFAULT_TABLE table=default.Table_5d5e26f93a5a4fb084831add317d90f9 [id=f237b1c90a1743d597d87c5cf1abc89a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:19.398566 19204 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0d14fb95ace9447f810ccbed4d5befdf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:19.399062 19202 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9896ff9364744857968e87e80e776dd2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:19.399030 20899 tablet_replica.cc:333] T e2aaaf00f53d418b87d5c7af286e4d84 P 78c2c46c7520492db35d51c244fb4f74: stopping tablet replica
I20260130 02:58:19.400066 19840 tablet_bootstrap.cc:492] T 94355fc1338f479e8798070aff04fcba P 78c2c46c7520492db35d51c244fb4f74: No bootstrap required, opened a new log
I20260130 02:58:19.400472 19840 ts_tablet_manager.cc:1403] T 94355fc1338f479e8798070aff04fcba P 78c2c46c7520492db35d51c244fb4f74: Time spent bootstrapping tablet: real 0.247s user 0.010s sys 0.011s
I20260130 02:58:19.418046 20864 consensus_queue.cc:1048] T f9e5f01e69254d658c369f3e21cef523 P d870164263bb44b6b780a557daea3656 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260130 02:58:19.423583 19840 raft_consensus.cc:359] T 94355fc1338f479e8798070aff04fcba P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:19.424230 19840 raft_consensus.cc:385] T 94355fc1338f479e8798070aff04fcba P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:19.431566 19520 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "13f0ace4d9de43abab08eca32cdb30c1" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656"
I20260130 02:58:19.433380 19527 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "838e49e5e8874db393bdc15a1c92f87e" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656"
I20260130 02:58:19.433871 19527 raft_consensus.cc:3060] T 838e49e5e8874db393bdc15a1c92f87e P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:19.438048 19197 tablet_service.cc:1505] Processing CreateTablet for tablet dd377fab8c544bf69121dc9e9a060b29 (DEFAULT_TABLE table=default.Table_5e70ab0e32bc4048bd19388f9d7cd302 [id=030c8ac23aac4a1fb7acc954d09434db]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:19.438092 19378 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "13f0ace4d9de43abab08eca32cdb30c1" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74"
I20260130 02:58:19.438517 19356 tablet_service.cc:1505] Processing CreateTablet for tablet 9896ff9364744857968e87e80e776dd2 (DEFAULT_TABLE table=default.Table_5d5e26f93a5a4fb084831add317d90f9 [id=f237b1c90a1743d597d87c5cf1abc89a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:19.439100 19197 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dd377fab8c544bf69121dc9e9a060b29. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:19.439510 19356 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9896ff9364744857968e87e80e776dd2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:19.441953 19375 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "838e49e5e8874db393bdc15a1c92f87e" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74"
I20260130 02:58:19.442505 19375 raft_consensus.cc:3060] T 838e49e5e8874db393bdc15a1c92f87e P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:19.444010 19501 tablet_service.cc:1505] Processing CreateTablet for tablet dd377fab8c544bf69121dc9e9a060b29 (DEFAULT_TABLE table=default.Table_5e70ab0e32bc4048bd19388f9d7cd302 [id=030c8ac23aac4a1fb7acc954d09434db]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:19.446660 19378 raft_consensus.cc:3060] T 13f0ace4d9de43abab08eca32cdb30c1 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:19.448901 19527 raft_consensus.cc:2468] T 838e49e5e8874db393bdc15a1c92f87e P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 371a36d219164974abdf848841528152 in term 1.
I20260130 02:58:19.456943 18870 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:50284 (request call id 252) took 2401 ms. Trace:
I20260130 02:58:19.457520 18870 rpcz_store.cc:276] 0130 02:58:17.056966 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:58:18.959919 (+1902953us) service_pool.cc:225] Handling call
0130 02:58:19.351395 (+391476us) catalog_manager.cc:2101] Acquired catalog manager lock
0130 02:58:19.352868 (+ 1473us) catalog_manager.cc:2147] Created new table and tablet info
0130 02:58:19.390705 (+ 37837us) catalog_manager.cc:2207] Wrote table and tablets to system table
0130 02:58:19.456592 (+ 65887us) catalog_manager.cc:2241] Inserted table and tablets into CatalogManager maps
0130 02:58:19.456879 (+ 287us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0130 02:58:19.354100 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0130 02:58:19.354601 (+ 501us) write_op.cc:432] Acquiring schema lock in shared mode
0130 02:58:19.354627 (+ 26us) write_op.cc:435] Acquired schema lock
0130 02:58:19.354643 (+ 16us) tablet.cc:660] Decoding operations
0130 02:58:19.354939 (+ 296us) write_op.cc:620] Acquiring the partition lock for write op
0130 02:58:19.354987 (+ 48us) write_op.cc:641] Partition lock acquired for write op
0130 02:58:19.355011 (+ 24us) tablet.cc:683] Acquiring locks for 2 operations
0130 02:58:19.355217 (+ 206us) tablet.cc:699] Row locks acquired
0130 02:58:19.355234 (+ 17us) write_op.cc:260] PREPARE: finished
0130 02:58:19.355353 (+ 119us) write_op.cc:270] Start()
0130 02:58:19.355430 (+ 77us) write_op.cc:276] Timestamp: P: 1769741899355294 usec, L: 0
0130 02:58:19.355449 (+ 19us) op_driver.cc:348] REPLICATION: starting
0130 02:58:19.356033 (+ 584us) log.cc:844] Serialized 433 byte log entry
0130 02:58:19.372862 (+ 16829us) op_driver.cc:464] REPLICATION: finished
0130 02:58:19.373144 (+ 282us) write_op.cc:301] APPLY: starting
0130 02:58:19.373199 (+ 55us) tablet.cc:1365] starting BulkCheckPresence
0130 02:58:19.373300 (+ 101us) tablet.cc:1368] finished BulkCheckPresence
0130 02:58:19.373386 (+ 86us) tablet.cc:1370] starting ApplyRowOperation cycle
0130 02:58:19.373802 (+ 416us) tablet.cc:1381] finished ApplyRowOperation cycle
0130 02:58:19.374462 (+ 660us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0130 02:58:19.374510 (+ 48us) write_op.cc:312] APPLY: finished
0130 02:58:19.374715 (+ 205us) log.cc:844] Serialized 29 byte log entry
0130 02:58:19.374853 (+ 138us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:58:19.374920 (+ 67us) write_op.cc:454] Released schema lock
0130 02:58:19.374951 (+ 31us) write_op.cc:341] FINISH: Updating metrics
0130 02:58:19.375169 (+ 218us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:58:19.375192 (+ 23us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":152,"apply.run_cpu_time_us":1986,"apply.run_wall_time_us":1985,"num_ops":2,"prepare.queue_time_us":104,"prepare.run_cpu_time_us":2275,"prepare.run_wall_time_us":6114,"raft.queue_time_us":88,"raft.run_cpu_time_us":724,"raft.run_wall_time_us":724,"replication_time_us":17321}]]}
I20260130 02:58:19.461174 18871 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:50284 (request call id 250) took 2491 ms. Trace:
I20260130 02:58:19.462416 19378 raft_consensus.cc:2468] T 13f0ace4d9de43abab08eca32cdb30c1 P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 371a36d219164974abdf848841528152 in term 1.
I20260130 02:58:19.465402 18868 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:50284 (request call id 251) took 2469 ms. Trace:
I20260130 02:58:19.465866 20942 raft_consensus.cc:493] T c76f0dc7762c49c389fa6fef41f02141 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:19.466596 20942 raft_consensus.cc:515] T c76f0dc7762c49c389fa6fef41f02141 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:19.467250 19375 raft_consensus.cc:2468] T 838e49e5e8874db393bdc15a1c92f87e P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 371a36d219164974abdf848841528152 in term 1.
I20260130 02:58:19.385893 20913 rpcz_store.cc:276] 0130 02:58:18.365337 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:58:18.365809 (+ 472us) service_pool.cc:225] Handling call
0130 02:58:19.385447 (+1019638us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0130 02:58:18.386071 (+ 0us) alter_schema_op.cc:104] PREPARE ALTER-SCHEMA: Starting
0130 02:58:18.386488 (+ 417us) alter_schema_op.cc:63] Acquiring schema lock in exclusive mode
0130 02:58:18.386522 (+ 34us) alter_schema_op.cc:66] Acquired schema lock
0130 02:58:18.386598 (+ 76us) alter_schema_op.cc:119] PREPARE ALTER-SCHEMA: finished
0130 02:58:18.386716 (+ 118us) alter_schema_op.cc:128] START. Timestamp: 1769741898386639
0130 02:58:18.386741 (+ 25us) op_driver.cc:348] REPLICATION: starting
0130 02:58:18.387318 (+ 577us) log.cc:844] Serialized 173 byte log entry
0130 02:58:19.372915 (+985597us) op_driver.cc:464] REPLICATION: finished
0130 02:58:19.373157 (+ 242us) alter_schema_op.cc:133] APPLY ALTER-SCHEMA: Starting
0130 02:58:19.374560 (+ 1403us) log.cc:844] Serialized 5 byte log entry
0130 02:58:19.383569 (+ 9009us) tablet_metadata.cc:641] Metadata flushed
0130 02:58:19.384510 (+ 941us) log.cc:844] Serialized 14 byte log entry
0130 02:58:19.385283 (+ 773us) alter_schema_op.cc:73] Released schema lock
0130 02:58:19.385318 (+ 35us) alter_schema_op.cc:186] AlterSchemaCommitCallback: making alter schema visible
Metrics: {"child_traces":[["op",{"apply.queue_time_us":91,"mutex_wait_us":18513,"prepare.queue_time_us":19397,"prepare.run_cpu_time_us":2097,"prepare.run_wall_time_us":69692,"raft.queue_time_us":18756,"raft.run_cpu_time_us":811,"raft.run_wall_time_us":810,"replication_time_us":986069,"thread_start_us":57400,"threads_started":2,"wal-append.queue_time_us":2544}]]}
I20260130 02:58:19.468750 20942 leader_election.cc:290] T c76f0dc7762c49c389fa6fef41f02141 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), d870164263bb44b6b780a557daea3656 (127.18.100.67:39571)
I20260130 02:58:19.469969 19224 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c76f0dc7762c49c389fa6fef41f02141" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152" is_pre_election: true
W20260130 02:58:19.471094 19314 leader_election.cc:343] T c76f0dc7762c49c389fa6fef41f02141 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:19.472586 19503 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0d14fb95ace9447f810ccbed4d5befdf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:19.475107 19501 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dd377fab8c544bf69121dc9e9a060b29. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:19.478343 20697 leader_election.cc:290] T cfd12a72726846e587cff9d6a4d61bd6 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), d870164263bb44b6b780a557daea3656 (127.18.100.67:39571)
I20260130 02:58:19.479218 19527 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c76f0dc7762c49c389fa6fef41f02141" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
I20260130 02:58:19.481741 19528 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cfd12a72726846e587cff9d6a4d61bd6" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
I20260130 02:58:19.481894 19224 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cfd12a72726846e587cff9d6a4d61bd6" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152" is_pre_election: true
I20260130 02:58:19.483032 19496 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9896ff9364744857968e87e80e776dd2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:19.485438 19807 tablet_bootstrap.cc:492] T af0f90071a334016aa17fae2e3a97fc8 P 371a36d219164974abdf848841528152: No bootstrap required, opened a new log
I20260130 02:58:19.485816 19807 ts_tablet_manager.cc:1403] T af0f90071a334016aa17fae2e3a97fc8 P 371a36d219164974abdf848841528152: Time spent bootstrapping tablet: real 0.528s user 0.010s sys 0.004s
I20260130 02:58:19.488147 18860 catalog_manager.cc:4985] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): tablet c7c621b99deb457998c44ab1644d3939 (table default.Table_3413dae2efac41488448e6e5975cf710 [id=7a08fa3986d0410d81de0235d2fb0f92]) successfully deleted
W20260130 02:58:19.488835 18860 catalog_manager.cc:4952] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): delete failed for tablet ec28ff12927d450ead16d26cf016138d because the tablet was not found. No further retry: Not found: Tablet not found: ec28ff12927d450ead16d26cf016138d
I20260130 02:58:19.488723 19807 raft_consensus.cc:359] T af0f90071a334016aa17fae2e3a97fc8 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:19.489331 19807 raft_consensus.cc:385] T af0f90071a334016aa17fae2e3a97fc8 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:19.489581 19807 raft_consensus.cc:740] T af0f90071a334016aa17fae2e3a97fc8 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 371a36d219164974abdf848841528152, State: Initialized, Role: FOLLOWER
I20260130 02:58:19.490521 20916 tablet_replica.cc:333] T 78e9bc3e55fe4e4a98e8eb01da59d365 P d870164263bb44b6b780a557daea3656: stopping tablet replica
I20260130 02:58:19.491500 20916 raft_consensus.cc:2243] T 78e9bc3e55fe4e4a98e8eb01da59d365 P d870164263bb44b6b780a557daea3656 [term 1 LEADER]: Raft consensus shutting down.
I20260130 02:58:19.498159 20925 consensus_queue.cc:1048] T 6218f2f7d73a42428b57a45858f53afd P 371a36d219164974abdf848841528152 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260130 02:58:19.480305 19313 leader_election.cc:343] T c76f0dc7762c49c389fa6fef41f02141 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:19.499046 19313 leader_election.cc:304] T c76f0dc7762c49c389fa6fef41f02141 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78c2c46c7520492db35d51c244fb4f74; no voters: 371a36d219164974abdf848841528152, d870164263bb44b6b780a557daea3656
I20260130 02:58:19.499679 20942 raft_consensus.cc:2749] T c76f0dc7762c49c389fa6fef41f02141 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260130 02:58:19.424485 19840 raft_consensus.cc:740] T 94355fc1338f479e8798070aff04fcba P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78c2c46c7520492db35d51c244fb4f74, State: Initialized, Role: FOLLOWER
W20260130 02:58:19.496022 19314 leader_election.cc:343] T cfd12a72726846e587cff9d6a4d61bd6 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Illegal state: must be running to vote when last-logged opid is not known
W20260130 02:58:19.501578 19313 leader_election.cc:343] T cfd12a72726846e587cff9d6a4d61bd6 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:19.502333 19313 leader_election.cc:304] T cfd12a72726846e587cff9d6a4d61bd6 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78c2c46c7520492db35d51c244fb4f74; no voters: 371a36d219164974abdf848841528152, d870164263bb44b6b780a557daea3656
I20260130 02:58:19.306460 18872 catalog_manager.cc:5746] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021) does not have the latest schema for tablet 345eaaa5625c48d099280b7fbb2e514b (table default.Table_c1886dd9a7934e7797e0e7e0b065d692 [id=7a2d5367224b492b844199088b01b149]). Expected version 1 got 0
I20260130 02:58:19.504973 20704 raft_consensus.cc:2749] T cfd12a72726846e587cff9d6a4d61bd6 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260130 02:58:19.505414 18872 catalog_manager.cc:5746] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021) does not have the latest schema for tablet b1e8b78eff0d4a0b92059adb2de4a3ce (table default.Table_d0b4516d24384e5386c1debaf2543adb [id=f1342f9497b74e7a9ecad1de8082578d]). Expected version 1 got 0
I20260130 02:58:19.501598 19840 consensus_queue.cc:260] T 94355fc1338f479e8798070aff04fcba P 78c2c46c7520492db35d51c244fb4f74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:19.504729 20952 tablet.cc:1722] T 127a84346b9f473d9e7f739b1e65feb8 P 78c2c46c7520492db35d51c244fb4f74: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 6 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 7
I20260130 02:58:19.507186 19840 ts_tablet_manager.cc:1434] T 94355fc1338f479e8798070aff04fcba P 78c2c46c7520492db35d51c244fb4f74: Time spent starting tablet: real 0.106s user 0.006s sys 0.000s
I20260130 02:58:19.507972 19840 tablet_bootstrap.cc:492] T 2b8034a92b2043e5b57cf02f1ee6296d P 78c2c46c7520492db35d51c244fb4f74: Bootstrap starting.
I20260130 02:58:19.461575 18871 rpcz_store.cc:276] 0130 02:58:16.969908 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:58:18.939660 (+1969752us) service_pool.cc:225] Handling call
0130 02:58:18.974706 (+ 35046us) spinlock_profiling.cc:243] Waited 4.95 ms on lock 0x7b5400030554. stack: 00007fb0bd3f7373 00007fb0bd3f7270 00007fb0bcf6c574 0000000000508744 00000000005086d9 00007fb0c44490c0 00007fb0c44e7bba 00007fb0c61e6eac 00007fb0c62e2656 00007fb0c3f05524 00007fb0c3f054b2 00007fb0c3f053e1 00007fb0c3f0535c 00007fb0c3f04612 00007fb0c21dd34c 00007fb0c21dc786
0130 02:58:19.350635 (+375929us) catalog_manager.cc:2101] Acquired catalog manager lock
0130 02:58:19.361376 (+ 10741us) catalog_manager.cc:2147] Created new table and tablet info
0130 02:58:19.413526 (+ 52150us) catalog_manager.cc:2207] Wrote table and tablets to system table
0130 02:58:19.460849 (+ 47323us) catalog_manager.cc:2241] Inserted table and tablets into CatalogManager maps
0130 02:58:19.461116 (+ 267us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0130 02:58:19.378777 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0130 02:58:19.379337 (+ 560us) write_op.cc:432] Acquiring schema lock in shared mode
0130 02:58:19.379367 (+ 30us) write_op.cc:435] Acquired schema lock
0130 02:58:19.379383 (+ 16us) tablet.cc:660] Decoding operations
0130 02:58:19.379722 (+ 339us) write_op.cc:620] Acquiring the partition lock for write op
0130 02:58:19.379771 (+ 49us) write_op.cc:641] Partition lock acquired for write op
0130 02:58:19.379798 (+ 27us) tablet.cc:683] Acquiring locks for 2 operations
0130 02:58:19.380013 (+ 215us) tablet.cc:699] Row locks acquired
0130 02:58:19.380032 (+ 19us) write_op.cc:260] PREPARE: finished
0130 02:58:19.380166 (+ 134us) write_op.cc:270] Start()
0130 02:58:19.380264 (+ 98us) write_op.cc:276] Timestamp: P: 1769741899380099 usec, L: 0
0130 02:58:19.380279 (+ 15us) op_driver.cc:348] REPLICATION: starting
0130 02:58:19.380830 (+ 551us) log.cc:844] Serialized 433 byte log entry
0130 02:58:19.406284 (+ 25454us) op_driver.cc:464] REPLICATION: finished
0130 02:58:19.406677 (+ 393us) write_op.cc:301] APPLY: starting
0130 02:58:19.406730 (+ 53us) tablet.cc:1365] starting BulkCheckPresence
0130 02:58:19.406829 (+ 99us) tablet.cc:1368] finished BulkCheckPresence
0130 02:58:19.406916 (+ 87us) tablet.cc:1370] starting ApplyRowOperation cycle
0130 02:58:19.407289 (+ 373us) tablet.cc:1381] finished ApplyRowOperation cycle
0130 02:58:19.407449 (+ 160us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0130 02:58:19.407488 (+ 39us) write_op.cc:312] APPLY: finished
0130 02:58:19.407664 (+ 176us) log.cc:844] Serialized 29 byte log entry
0130 02:58:19.407797 (+ 133us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:58:19.407865 (+ 68us) write_op.cc:454] Released schema lock
0130 02:58:19.407895 (+ 30us) write_op.cc:341] FINISH: Updating metrics
0130 02:58:19.408102 (+ 207us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:58:19.408126 (+ 24us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":11376000,"child_traces":[["op",{"apply.queue_time_us":251,"apply.run_cpu_time_us":1397,"apply.run_wall_time_us":1397,"num_ops":2,"prepare.queue_time_us":16286,"prepare.run_cpu_time_us":2281,"prepare.run_wall_time_us":3186,"replication_time_us":25882}]]}
I20260130 02:58:19.510461 19427 heartbeater.cc:344] Connected to a master server at 127.18.100.126:38191
I20260130 02:58:19.510967 18867 master_service.cc:954] ReplaceTablet: received request to replace tablet 8e57ae31ab3941f789ac69627384e3b0 from {username='slave'} at 127.0.0.1:50284
I20260130 02:58:19.511395 20916 raft_consensus.cc:2272] T 78e9bc3e55fe4e4a98e8eb01da59d365 P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260130 02:58:19.512128 20704 raft_consensus.cc:493] T 1fc8f09d6329407c8078542b4123ef4d P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:19.512049 18870 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50284:
table { table_name: "default.Table_84fbe659a98c4a2e9f805bb1efc7b4de" } new_table_name: "default.Table_aa496bf2019748f29823a9a253b1f3ca" modify_external_catalogs: true
I20260130 02:58:19.512511 20704 raft_consensus.cc:515] T 1fc8f09d6329407c8078542b4123ef4d P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:19.513298 20916 ts_tablet_manager.cc:1916] T 78e9bc3e55fe4e4a98e8eb01da59d365 P d870164263bb44b6b780a557daea3656: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260130 02:58:19.516348 18871 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50284:
table { table_name: "default.Table_89d4117da99842719125411e30d10169" } new_table_name: "default.Table_aaf139c1a55d4a1783298ca12ae0d5c0" modify_external_catalogs: true
I20260130 02:58:19.516438 19528 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1fc8f09d6329407c8078542b4123ef4d" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
I20260130 02:58:19.515776 20704 leader_election.cc:290] T 1fc8f09d6329407c8078542b4123ef4d P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), d870164263bb44b6b780a557daea3656 (127.18.100.67:39571)
W20260130 02:58:19.517704 19313 leader_election.cc:343] T 1fc8f09d6329407c8078542b4123ef4d P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:19.525804 19353 tablet_service.cc:1505] Processing CreateTablet for tablet dd377fab8c544bf69121dc9e9a060b29 (DEFAULT_TABLE table=default.Table_5e70ab0e32bc4048bd19388f9d7cd302 [id=030c8ac23aac4a1fb7acc954d09434db]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:19.529052 20697 raft_consensus.cc:493] T 306452e208ca4e719609f67345114a13 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:19.470127 18868 rpcz_store.cc:276] 0130 02:58:16.997043 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:58:18.953849 (+1956806us) service_pool.cc:225] Handling call
0130 02:58:19.320577 (+366728us) catalog_manager.cc:2101] Acquired catalog manager lock
0130 02:58:19.322061 (+ 1484us) catalog_manager.cc:2147] Created new table and tablet info
0130 02:58:19.433012 (+110951us) catalog_manager.cc:2207] Wrote table and tablets to system table
0130 02:58:19.464828 (+ 31816us) catalog_manager.cc:2241] Inserted table and tablets into CatalogManager maps
0130 02:58:19.465340 (+ 512us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0130 02:58:19.414245 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0130 02:58:19.414723 (+ 478us) write_op.cc:432] Acquiring schema lock in shared mode
0130 02:58:19.414752 (+ 29us) write_op.cc:435] Acquired schema lock
0130 02:58:19.414763 (+ 11us) tablet.cc:660] Decoding operations
0130 02:58:19.415055 (+ 292us) write_op.cc:620] Acquiring the partition lock for write op
0130 02:58:19.415097 (+ 42us) write_op.cc:641] Partition lock acquired for write op
0130 02:58:19.415121 (+ 24us) tablet.cc:683] Acquiring locks for 2 operations
0130 02:58:19.415333 (+ 212us) tablet.cc:699] Row locks acquired
0130 02:58:19.415346 (+ 13us) write_op.cc:260] PREPARE: finished
0130 02:58:19.415466 (+ 120us) write_op.cc:270] Start()
0130 02:58:19.415559 (+ 93us) write_op.cc:276] Timestamp: P: 1769741899415402 usec, L: 0
0130 02:58:19.415574 (+ 15us) op_driver.cc:348] REPLICATION: starting
0130 02:58:19.416082 (+ 508us) log.cc:844] Serialized 433 byte log entry
0130 02:58:19.425085 (+ 9003us) op_driver.cc:464] REPLICATION: finished
0130 02:58:19.425545 (+ 460us) write_op.cc:301] APPLY: starting
0130 02:58:19.425605 (+ 60us) tablet.cc:1365] starting BulkCheckPresence
0130 02:58:19.425716 (+ 111us) tablet.cc:1368] finished BulkCheckPresence
0130 02:58:19.425816 (+ 100us) tablet.cc:1370] starting ApplyRowOperation cycle
0130 02:58:19.426241 (+ 425us) tablet.cc:1381] finished ApplyRowOperation cycle
0130 02:58:19.426361 (+ 120us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0130 02:58:19.426403 (+ 42us) write_op.cc:312] APPLY: finished
0130 02:58:19.426633 (+ 230us) log.cc:844] Serialized 29 byte log entry
0130 02:58:19.426767 (+ 134us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:58:19.426828 (+ 61us) write_op.cc:454] Released schema lock
0130 02:58:19.426857 (+ 29us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":285,"apply.run_cpu_time_us":1498,"apply.run_wall_time_us":1498,"num_ops":2,"prepare.queue_time_us":13828,"prepare.run_cpu_time_us":2199,"prepare.run_wall_time_us":94571,"raft.queue_time_us":140,"raft.run_cpu_time_us":603,"raft.run_wall_time_us":603,"replication_time_us":9404}]]}
I20260130 02:58:19.530335 18868 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50284:
table { table_name: "default.Table_4dde1730dacb42728dd8375fdaa37615" } modify_external_catalogs: true
I20260130 02:58:19.490103 19807 consensus_queue.cc:260] T af0f90071a334016aa17fae2e3a97fc8 P 371a36d219164974abdf848841528152 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:19.530992 19353 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dd377fab8c544bf69121dc9e9a060b29. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:19.541257 20899 raft_consensus.cc:2243] T e2aaaf00f53d418b87d5c7af286e4d84 P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260130 02:58:19.548486 20899 raft_consensus.cc:2272] T e2aaaf00f53d418b87d5c7af286e4d84 P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260130 02:58:19.548223 20697 raft_consensus.cc:515] T 306452e208ca4e719609f67345114a13 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
W20260130 02:58:19.549073 19159 leader_election.cc:343] T 13f0ace4d9de43abab08eca32cdb30c1 P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:19.551245 20899 ts_tablet_manager.cc:1916] T e2aaaf00f53d418b87d5c7af286e4d84 P 78c2c46c7520492db35d51c244fb4f74: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260130 02:58:19.551179 19159 leader_election.cc:304] T 838e49e5e8874db393bdc15a1c92f87e P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 371a36d219164974abdf848841528152, d870164263bb44b6b780a557daea3656; no voters:
I20260130 02:58:19.551653 18868 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50284:
table { table_name: "default.Table_4dde1730dacb42728dd8375fdaa37615" } modify_external_catalogs: true
I20260130 02:58:19.551805 20920 raft_consensus.cc:2804] T 838e49e5e8874db393bdc15a1c92f87e P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Leader election won for term 1
I20260130 02:58:19.552205 20920 raft_consensus.cc:697] T 838e49e5e8874db393bdc15a1c92f87e P 371a36d219164974abdf848841528152 [term 1 LEADER]: Becoming Leader. State: Replica: 371a36d219164974abdf848841528152, State: Running, Role: LEADER
W20260130 02:58:19.543450 18860 catalog_manager.cc:4952] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): delete failed for tablet 38a7a10e36874183bf9fe55a38d829f6 because the tablet was not found. No further retry: Not found: Tablet not found: 38a7a10e36874183bf9fe55a38d829f6
W20260130 02:58:19.553932 18860 catalog_manager.cc:4952] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): delete failed for tablet 3389db58493448aea3cf00aca411047c because the tablet was not found. No further retry: Not found: Tablet not found: 3389db58493448aea3cf00aca411047c
I20260130 02:58:19.552953 20920 consensus_queue.cc:237] T 838e49e5e8874db393bdc15a1c92f87e P 371a36d219164974abdf848841528152 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
W20260130 02:58:19.554525 18860 catalog_manager.cc:4952] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): delete failed for tablet 1ca34772ef484c1f809a99d126e28eff because the tablet was not found. No further retry: Not found: Tablet not found: 1ca34772ef484c1f809a99d126e28eff
I20260130 02:58:19.554929 19528 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "306452e208ca4e719609f67345114a13" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
W20260130 02:58:19.555593 18860 catalog_manager.cc:4952] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): delete failed for tablet d27ca7dd87f44102b7680e1f4a88954a because the tablet was not found. No further retry: Not found: Tablet not found: d27ca7dd87f44102b7680e1f4a88954a
W20260130 02:58:19.555995 19313 leader_election.cc:343] T 306452e208ca4e719609f67345114a13 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): Illegal state: must be running to vote when last-logged opid is not known
W20260130 02:58:19.556115 18860 catalog_manager.cc:4952] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): delete failed for tablet d55291d7cea64a0d9a345ce712fbfa80 because the tablet was not found. No further retry: Not found: Tablet not found: d55291d7cea64a0d9a345ce712fbfa80
W20260130 02:58:19.556665 18860 catalog_manager.cc:4952] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): delete failed for tablet e364f4e70e5749e083e14e7fa7416ff0 because the tablet was not found. No further retry: Not found: Tablet not found: e364f4e70e5749e083e14e7fa7416ff0
W20260130 02:58:19.557165 18860 catalog_manager.cc:4952] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): delete failed for tablet f54addb1a4304ce69c711ae0cc3b452c because the tablet was not found. No further retry: Not found: Tablet not found: f54addb1a4304ce69c711ae0cc3b452c
W20260130 02:58:19.557674 18860 catalog_manager.cc:4952] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): delete failed for tablet 8bd3e1c06b89426da57e2b259e7983af because the tablet was not found. No further retry: Not found: Tablet not found: 8bd3e1c06b89426da57e2b259e7983af
W20260130 02:58:19.558224 18860 catalog_manager.cc:4952] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): delete failed for tablet b2ac91da01954c2390b6b7dcc06c394f because the tablet was not found. No further retry: Not found: Tablet not found: b2ac91da01954c2390b6b7dcc06c394f
I20260130 02:58:19.563195 20697 leader_election.cc:290] T 306452e208ca4e719609f67345114a13 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), d870164263bb44b6b780a557daea3656 (127.18.100.67:39571)
I20260130 02:58:19.564718 19224 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1fc8f09d6329407c8078542b4123ef4d" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152" is_pre_election: true
W20260130 02:58:19.567554 19314 leader_election.cc:343] T 1fc8f09d6329407c8078542b4123ef4d P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:19.567939 19314 leader_election.cc:304] T 1fc8f09d6329407c8078542b4123ef4d P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78c2c46c7520492db35d51c244fb4f74; no voters: 371a36d219164974abdf848841528152, d870164263bb44b6b780a557daea3656
I20260130 02:58:19.568990 19224 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "306452e208ca4e719609f67345114a13" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152" is_pre_election: true
I20260130 02:58:19.570730 19275 heartbeater.cc:344] Connected to a master server at 127.18.100.126:38191
I20260130 02:58:19.577564 20697 raft_consensus.cc:2749] T 1fc8f09d6329407c8078542b4123ef4d P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260130 02:58:19.577075 19807 ts_tablet_manager.cc:1434] T af0f90071a334016aa17fae2e3a97fc8 P 371a36d219164974abdf848841528152: Time spent starting tablet: real 0.091s user 0.007s sys 0.000s
I20260130 02:58:19.579747 19807 tablet_bootstrap.cc:492] T 320dbe98a32a43b9899bbeed80b20f55 P 371a36d219164974abdf848841528152: Bootstrap starting.
I20260130 02:58:19.580163 18872 catalog_manager.cc:5746] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021) does not have the latest schema for tablet 8e57ae31ab3941f789ac69627384e3b0 (table default.Table_1ae68ae3f0404843aca3c72f02678ac6 [id=8a9182f49e18489fafec8dca190c1b2f]). Expected version 1 got 0
I20260130 02:58:19.581190 18872 catalog_manager.cc:5746] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021) does not have the latest schema for tablet 127a84346b9f473d9e7f739b1e65feb8 (table default.Table_805e07e3546e447d8c8b8029477013cc [id=1515db389f284e2f9ed1a3cd0061d0f1]). Expected version 7 got 6
I20260130 02:58:19.581871 18872 catalog_manager.cc:5654] T 6218f2f7d73a42428b57a45858f53afd P 371a36d219164974abdf848841528152 reported cstate change: term changed from 0 to 1, leader changed from <none> to 371a36d219164974abdf848841528152 (127.18.100.65). New cstate: current_term: 1 leader_uuid: "371a36d219164974abdf848841528152" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } health_report { overall_health: UNKNOWN } } }
I20260130 02:58:19.584380 18872 catalog_manager.cc:5746] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021) does not have the latest schema for tablet 9fe0448bcc4a4346b3c67fc3f626ed3c (table default.Table_5c1c447e85524ac5a61ffd8a48509d36 [id=70aaf21ad9bd4d959f46939c6b4a4880]). Expected version 1 got 0
I20260130 02:58:19.584522 19356 tablet_service.cc:1505] Processing CreateTablet for tablet 6d1f121f49cd47c2b8247daaf6874e01 (DEFAULT_TABLE table=default.Table_0480357296ac4a27812d802ee4566351 [id=0da6197ab32e4645aa8d8787662ee5e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:19.585269 18872 catalog_manager.cc:5746] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021) does not have the latest schema for tablet 274530359b174ff79f046c2d4a6afa76 (table default.Table_c4f4904b6fac4b9880979cf0314e71c1 [id=c456c6418c93499a9d08f652720854ca]). Expected version 1 got 0
I20260130 02:58:19.585719 19356 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6d1f121f49cd47c2b8247daaf6874e01. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:19.580664 19202 tablet_service.cc:1505] Processing CreateTablet for tablet 6d1f121f49cd47c2b8247daaf6874e01 (DEFAULT_TABLE table=default.Table_0480357296ac4a27812d802ee4566351 [id=0da6197ab32e4645aa8d8787662ee5e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:19.595265 19202 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6d1f121f49cd47c2b8247daaf6874e01. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:19.597132 19501 tablet_service.cc:1505] Processing CreateTablet for tablet 6d1f121f49cd47c2b8247daaf6874e01 (DEFAULT_TABLE table=default.Table_0480357296ac4a27812d802ee4566351 [id=0da6197ab32e4645aa8d8787662ee5e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:19.598163 19501 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6d1f121f49cd47c2b8247daaf6874e01. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:19.342005 19625 raft_consensus.cc:359] T af0f90071a334016aa17fae2e3a97fc8 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:19.601670 19625 raft_consensus.cc:385] T af0f90071a334016aa17fae2e3a97fc8 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:19.601902 19625 raft_consensus.cc:740] T af0f90071a334016aa17fae2e3a97fc8 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d870164263bb44b6b780a557daea3656, State: Initialized, Role: FOLLOWER
I20260130 02:58:19.602476 19625 consensus_queue.cc:260] T af0f90071a334016aa17fae2e3a97fc8 P d870164263bb44b6b780a557daea3656 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:19.602427 18872 catalog_manager.cc:5654] T dcba6b534f9d406dad4fb502be40921c P 371a36d219164974abdf848841528152 reported cstate change: term changed from 0 to 1, leader changed from <none> to 371a36d219164974abdf848841528152 (127.18.100.65). New cstate: current_term: 1 leader_uuid: "371a36d219164974abdf848841528152" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } health_report { overall_health: UNKNOWN } } }
I20260130 02:58:19.604146 18872 catalog_manager.cc:5746] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021) does not have the latest schema for tablet 25fd48e693424537b91ba7c3fcc96878 (table default.Table_ebc8b8379c274c18b0a06a919dbe0ee0 [id=28156e0843674fce88f5914a3e4195ae]). Expected version 1 got 0
I20260130 02:58:19.604925 18872 catalog_manager.cc:5746] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021) does not have the latest schema for tablet 838e49e5e8874db393bdc15a1c92f87e (table default.Table_da3d581b15a94840898777f74611994e [id=b7e241ced53e4160ac6fa1f65ef3674e]). Expected version 1 got 0
I20260130 02:58:19.617501 19840 tablet_bootstrap.cc:654] T 2b8034a92b2043e5b57cf02f1ee6296d P 78c2c46c7520492db35d51c244fb4f74: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:19.614545 19807 tablet_bootstrap.cc:654] T 320dbe98a32a43b9899bbeed80b20f55 P 371a36d219164974abdf848841528152: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:19.626322 19505 tablet_service.cc:1505] Processing CreateTablet for tablet 1129f72158ec4bae89da3c94cdb70839 (DEFAULT_TABLE table=default.Table_4dde1730dacb42728dd8375fdaa37615 [id=22dd8f2c7e01431f97cbab7b8d94ea32]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:19.627390 19505 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1129f72158ec4bae89da3c94cdb70839. 1 dirs total, 0 dirs full, 0 dirs failed
W20260130 02:58:19.631026 19314 leader_election.cc:343] T 306452e208ca4e719609f67345114a13 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:19.631405 19314 leader_election.cc:304] T 306452e208ca4e719609f67345114a13 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78c2c46c7520492db35d51c244fb4f74; no voters: 371a36d219164974abdf848841528152, d870164263bb44b6b780a557daea3656
I20260130 02:58:19.634195 20704 raft_consensus.cc:2749] T 306452e208ca4e719609f67345114a13 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260130 02:58:19.636332 19159 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.100.67:39571, user_credentials={real_user=slave}} blocked reactor thread for 85207.1us
I20260130 02:58:19.641292 19625 ts_tablet_manager.cc:1434] T af0f90071a334016aa17fae2e3a97fc8 P d870164263bb44b6b780a557daea3656: Time spent starting tablet: real 0.301s user 0.004s sys 0.000s
I20260130 02:58:19.642460 19625 tablet_bootstrap.cc:492] T 13f0ace4d9de43abab08eca32cdb30c1 P d870164263bb44b6b780a557daea3656: Bootstrap starting.
I20260130 02:58:19.655272 20899 ts_tablet_manager.cc:1929] T e2aaaf00f53d418b87d5c7af286e4d84 P 78c2c46c7520492db35d51c244fb4f74: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260130 02:58:19.658967 20899 log.cc:1199] T e2aaaf00f53d418b87d5c7af286e4d84 P 78c2c46c7520492db35d51c244fb4f74: Deleting WAL directory at /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-1/wal/wals/e2aaaf00f53d418b87d5c7af286e4d84
I20260130 02:58:19.662742 20899 ts_tablet_manager.cc:1950] T e2aaaf00f53d418b87d5c7af286e4d84 P 78c2c46c7520492db35d51c244fb4f74: Deleting consensus metadata
I20260130 02:58:19.668480 20899 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60110 (request call id 292) took 1015 ms. Trace:
I20260130 02:58:19.670713 20899 rpcz_store.cc:276] 0130 02:58:18.653381 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:58:18.653483 (+ 102us) service_pool.cc:225] Handling call
0130 02:58:19.398862 (+745379us) ts_tablet_manager.cc:1151] Deleting tablet e2aaaf00f53d418b87d5c7af286e4d84
0130 02:58:19.398915 (+ 53us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0130 02:58:19.561552 (+162637us) tablet_metadata.cc:641] Metadata flushed
0130 02:58:19.633900 (+ 72348us) tablet_metadata.cc:641] Metadata flushed
0130 02:58:19.668422 (+ 34522us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":141259,"tablet-delete.queue_time_us":744861}
I20260130 02:58:19.672601 18860 catalog_manager.cc:4985] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767): tablet e2aaaf00f53d418b87d5c7af286e4d84 (table default.Table_e84c6277dfc944059f62cbe0d6b937ee [id=747af828147149efb7b7b8d33c49286d]) successfully deleted
I20260130 02:58:19.673415 20899 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60110 (request call id 293) took 1019 ms. Trace:
I20260130 02:58:19.676676 20899 rpcz_store.cc:276] 0130 02:58:18.654402 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:58:18.654510 (+ 108us) service_pool.cc:225] Handling call
0130 02:58:19.673151 (+1018641us) ts_tablet_manager.cc:1151] Deleting tablet 3389db58493448aea3cf00aca411047c
0130 02:58:19.673201 (+ 50us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0130 02:58:19.673362 (+ 161us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1018152}
I20260130 02:58:19.684924 20960 tablet.cc:1722] T 127a84346b9f473d9e7f739b1e65feb8 P d870164263bb44b6b780a557daea3656: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 6 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 7
W20260130 02:58:19.694231 18860 catalog_manager.cc:4952] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767): delete failed for tablet 3389db58493448aea3cf00aca411047c because the tablet was not found. No further retry: Not found: Tablet not found: 3389db58493448aea3cf00aca411047c
W20260130 02:58:19.695889 18860 catalog_manager.cc:4952] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767): delete failed for tablet 1ca34772ef484c1f809a99d126e28eff because the tablet was not found. No further retry: Not found: Tablet not found: 1ca34772ef484c1f809a99d126e28eff
W20260130 02:58:19.699990 18860 catalog_manager.cc:4952] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767): delete failed for tablet d27ca7dd87f44102b7680e1f4a88954a because the tablet was not found. No further retry: Not found: Tablet not found: d27ca7dd87f44102b7680e1f4a88954a
W20260130 02:58:19.702803 18860 catalog_manager.cc:4952] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767): delete failed for tablet 38a7a10e36874183bf9fe55a38d829f6 because the tablet was not found. No further retry: Not found: Tablet not found: 38a7a10e36874183bf9fe55a38d829f6
W20260130 02:58:19.705646 18860 catalog_manager.cc:4952] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767): delete failed for tablet d55291d7cea64a0d9a345ce712fbfa80 because the tablet was not found. No further retry: Not found: Tablet not found: d55291d7cea64a0d9a345ce712fbfa80
I20260130 02:58:19.737262 18868 catalog_manager.cc:5941] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4: Sending DeleteTablet for 3 replicas of tablet 1129f72158ec4bae89da3c94cdb70839
I20260130 02:58:19.737524 20916 ts_tablet_manager.cc:1929] T 78e9bc3e55fe4e4a98e8eb01da59d365 P d870164263bb44b6b780a557daea3656: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260130 02:58:19.737870 20916 log.cc:1199] T 78e9bc3e55fe4e4a98e8eb01da59d365 P d870164263bb44b6b780a557daea3656: Deleting WAL directory at /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-2/wal/wals/78e9bc3e55fe4e4a98e8eb01da59d365
I20260130 02:58:19.739306 18868 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50284:
name: "default.Table_6c46e6995d4741dcaaa6ed7bdc765e29"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260130 02:58:19.740494 20916 ts_tablet_manager.cc:1950] T 78e9bc3e55fe4e4a98e8eb01da59d365 P d870164263bb44b6b780a557daea3656: Deleting consensus metadata
W20260130 02:58:19.740738 18868 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6c46e6995d4741dcaaa6ed7bdc765e29 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260130 02:58:19.741637 20864 raft_consensus.cc:493] T af0f90071a334016aa17fae2e3a97fc8 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:19.742063 20864 raft_consensus.cc:515] T af0f90071a334016aa17fae2e3a97fc8 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:19.744524 20864 leader_election.cc:290] T af0f90071a334016aa17fae2e3a97fc8 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767)
I20260130 02:58:19.746333 19376 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "af0f90071a334016aa17fae2e3a97fc8" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74" is_pre_election: true
I20260130 02:58:19.746946 19376 raft_consensus.cc:2468] T af0f90071a334016aa17fae2e3a97fc8 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d870164263bb44b6b780a557daea3656 in term 0.
I20260130 02:58:19.748021 19462 leader_election.cc:304] T af0f90071a334016aa17fae2e3a97fc8 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78c2c46c7520492db35d51c244fb4f74, d870164263bb44b6b780a557daea3656; no voters:
I20260130 02:58:19.748783 19360 tablet_service.cc:1552] Processing DeleteTablet for tablet 1129f72158ec4bae89da3c94cdb70839 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:58:19 UTC) from {username='slave'} at 127.0.0.1:60110
I20260130 02:58:19.749320 20864 raft_consensus.cc:2804] T af0f90071a334016aa17fae2e3a97fc8 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260130 02:58:19.749622 20864 raft_consensus.cc:493] T af0f90071a334016aa17fae2e3a97fc8 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260130 02:58:19.750655 20864 raft_consensus.cc:3060] T af0f90071a334016aa17fae2e3a97fc8 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:19.752301 18867 catalog_manager.cc:5941] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4: Sending DeleteTablet for 3 replicas of tablet 8e57ae31ab3941f789ac69627384e3b0
I20260130 02:58:19.753499 18867 catalog_manager.cc:6724] ReplaceTablet: tablet 8e57ae31ab3941f789ac69627384e3b0 deleted and replaced by tablet 8b47653c7c3945a68e9bf4680bfc0496
I20260130 02:58:19.753705 20897 ts_tablet_manager.cc:1929] T 3389db58493448aea3cf00aca411047c P 371a36d219164974abdf848841528152: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260130 02:58:19.753819 18867 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:50284 (request call id 254) took 2103 ms. Trace:
I20260130 02:58:19.754081 20897 log.cc:1199] T 3389db58493448aea3cf00aca411047c P 371a36d219164974abdf848841528152: Deleting WAL directory at /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-0/wal/wals/3389db58493448aea3cf00aca411047c
I20260130 02:58:19.754120 18867 rpcz_store.cc:276] 0130 02:58:17.650045 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:58:19.510857 (+1860812us) service_pool.cc:225] Handling call
0130 02:58:19.753770 (+242913us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0130 02:58:19.695488 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0130 02:58:19.695881 (+ 393us) write_op.cc:432] Acquiring schema lock in shared mode
0130 02:58:19.695905 (+ 24us) write_op.cc:435] Acquired schema lock
0130 02:58:19.695916 (+ 11us) tablet.cc:660] Decoding operations
0130 02:58:19.696234 (+ 318us) write_op.cc:620] Acquiring the partition lock for write op
0130 02:58:19.696272 (+ 38us) write_op.cc:641] Partition lock acquired for write op
0130 02:58:19.696294 (+ 22us) tablet.cc:683] Acquiring locks for 2 operations
0130 02:58:19.696480 (+ 186us) tablet.cc:699] Row locks acquired
0130 02:58:19.696493 (+ 13us) write_op.cc:260] PREPARE: finished
0130 02:58:19.696606 (+ 113us) write_op.cc:270] Start()
0130 02:58:19.696682 (+ 76us) write_op.cc:276] Timestamp: P: 1769741899696512 usec, L: 0
0130 02:58:19.696697 (+ 15us) op_driver.cc:348] REPLICATION: starting
0130 02:58:19.722225 (+ 25528us) log.cc:844] Serialized 662 byte log entry
0130 02:58:19.724012 (+ 1787us) spinlock_profiling.cc:243] Waited 25 ms on lock 0x7b5400030554. stack: 00007fb0bd3f7373 00007fb0bd3f7270 00007fb0bcf6c574 0000000000508744 00000000005086d9 00007fb0c44485a1 00007fb0c44daaa8 00007fb0c494c4d1 00007fb0c494bca8 00007fb0c4950941 00007fb0c49508f9 00007fb0c4950889 00007fb0c4950851 00007fb0c494fa7d 00000000005aa2b4 00000000005aa0e9
0130 02:58:19.749068 (+ 25056us) op_driver.cc:464] REPLICATION: finished
0130 02:58:19.749575 (+ 507us) write_op.cc:301] APPLY: starting
0130 02:58:19.749628 (+ 53us) tablet.cc:1365] starting BulkCheckPresence
0130 02:58:19.749722 (+ 94us) tablet.cc:1368] finished BulkCheckPresence
0130 02:58:19.749743 (+ 21us) tablet.cc:1370] starting ApplyRowOperation cycle
0130 02:58:19.750043 (+ 300us) tablet.cc:1381] finished ApplyRowOperation cycle
0130 02:58:19.750237 (+ 194us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0130 02:58:19.750278 (+ 41us) write_op.cc:312] APPLY: finished
0130 02:58:19.750446 (+ 168us) log.cc:844] Serialized 29 byte log entry
0130 02:58:19.750617 (+ 171us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:58:19.750678 (+ 61us) write_op.cc:454] Released schema lock
0130 02:58:19.750711 (+ 33us) write_op.cc:341] FINISH: Updating metrics
0130 02:58:19.750972 (+ 261us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:58:19.750995 (+ 23us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":366,"apply.run_cpu_time_us":1361,"apply.run_wall_time_us":1360,"num_ops":2,"prepare.queue_time_us":18140,"prepare.run_cpu_time_us":2813,"prepare.run_wall_time_us":28699,"raft.queue_time_us":2878,"raft.run_cpu_time_us":717,"raft.run_wall_time_us":717,"replication_time_us":52286,"spinlock_wait_cycles":57416064}]]}
I20260130 02:58:19.757519 18867 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50284:
name: "default.Table_dabf999abe834151a0a5b6747a05cac3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260130 02:58:19.757959 20864 raft_consensus.cc:515] T af0f90071a334016aa17fae2e3a97fc8 P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:19.760798 19376 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "af0f90071a334016aa17fae2e3a97fc8" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74"
I20260130 02:58:19.761345 19376 raft_consensus.cc:3060] T af0f90071a334016aa17fae2e3a97fc8 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Advancing to term 1
W20260130 02:58:19.766433 18867 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_dabf999abe834151a0a5b6747a05cac3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260130 02:58:19.767764 19376 raft_consensus.cc:2468] T af0f90071a334016aa17fae2e3a97fc8 P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d870164263bb44b6b780a557daea3656 in term 1.
I20260130 02:58:19.768906 19462 leader_election.cc:304] T af0f90071a334016aa17fae2e3a97fc8 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78c2c46c7520492db35d51c244fb4f74, d870164263bb44b6b780a557daea3656; no voters:
I20260130 02:58:19.769914 20957 raft_consensus.cc:2804] T af0f90071a334016aa17fae2e3a97fc8 P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Leader election won for term 1
I20260130 02:58:19.770606 20957 raft_consensus.cc:697] T af0f90071a334016aa17fae2e3a97fc8 P d870164263bb44b6b780a557daea3656 [term 1 LEADER]: Becoming Leader. State: Replica: d870164263bb44b6b780a557daea3656, State: Running, Role: LEADER
I20260130 02:58:19.771919 19353 tablet_service.cc:1505] Processing CreateTablet for tablet 1129f72158ec4bae89da3c94cdb70839 (DEFAULT_TABLE table=default.Table_4dde1730dacb42728dd8375fdaa37615 [id=22dd8f2c7e01431f97cbab7b8d94ea32]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:19.772173 20957 consensus_queue.cc:237] T af0f90071a334016aa17fae2e3a97fc8 P d870164263bb44b6b780a557daea3656 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:19.773748 19353 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1129f72158ec4bae89da3c94cdb70839. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:19.775141 19503 tablet_service.cc:1552] Processing DeleteTablet for tablet 1129f72158ec4bae89da3c94cdb70839 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:58:19 UTC) from {username='slave'} at 127.0.0.1:59068
I20260130 02:58:19.783751 19503 tablet_service.cc:1552] Processing DeleteTablet for tablet 8e57ae31ab3941f789ac69627384e3b0 with delete_type TABLET_DATA_DELETED (replaced by tablet 8b47653c7c3945a68e9bf4680bfc0496) from {username='slave'} at 127.0.0.1:59068
W20260130 02:58:19.784255 19159 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.100.66:40767, user_credentials={real_user=slave}} blocked reactor thread for 82972.4us
I20260130 02:58:19.789664 19224 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "af0f90071a334016aa17fae2e3a97fc8" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152" is_pre_election: true
I20260130 02:58:19.790212 19224 raft_consensus.cc:2468] T af0f90071a334016aa17fae2e3a97fc8 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d870164263bb44b6b780a557daea3656 in term 0.
I20260130 02:58:19.792256 19159 leader_election.cc:304] T 13f0ace4d9de43abab08eca32cdb30c1 P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 371a36d219164974abdf848841528152, 78c2c46c7520492db35d51c244fb4f74; no voters: d870164263bb44b6b780a557daea3656
I20260130 02:58:19.799077 20864 leader_election.cc:290] T af0f90071a334016aa17fae2e3a97fc8 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 election: Requested vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767)
I20260130 02:58:19.804976 18874 catalog_manager.cc:5746] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571) does not have the latest schema for tablet 345eaaa5625c48d099280b7fbb2e514b (table default.Table_c1886dd9a7934e7797e0e7e0b065d692 [id=7a2d5367224b492b844199088b01b149]). Expected version 1 got 0
I20260130 02:58:19.805612 20916 tablet_replica.cc:333] T 04f4ac27800c4ac3a79ae12d35479757 P d870164263bb44b6b780a557daea3656: stopping tablet replica
I20260130 02:58:19.806474 20916 raft_consensus.cc:2243] T 04f4ac27800c4ac3a79ae12d35479757 P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260130 02:58:19.807008 20916 raft_consensus.cc:2272] T 04f4ac27800c4ac3a79ae12d35479757 P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260130 02:58:19.808996 20916 ts_tablet_manager.cc:1916] T 04f4ac27800c4ac3a79ae12d35479757 P d870164263bb44b6b780a557daea3656: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260130 02:58:19.810869 19356 tablet_service.cc:1552] Processing DeleteTablet for tablet 8e57ae31ab3941f789ac69627384e3b0 with delete_type TABLET_DATA_DELETED (replaced by tablet 8b47653c7c3945a68e9bf4680bfc0496) from {username='slave'} at 127.0.0.1:60110
I20260130 02:58:19.793231 19224 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "af0f90071a334016aa17fae2e3a97fc8" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152"
I20260130 02:58:19.813701 19224 raft_consensus.cc:3060] T af0f90071a334016aa17fae2e3a97fc8 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:19.815310 19625 tablet_bootstrap.cc:654] T 13f0ace4d9de43abab08eca32cdb30c1 P d870164263bb44b6b780a557daea3656: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:19.805936 18860 catalog_manager.cc:4985] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): tablet 78e9bc3e55fe4e4a98e8eb01da59d365 (table default.Table_5d29239ce1984d2fad02607a2bdd4dd0 [id=7e271b81bb064a949685b2411b485e3e]) successfully deleted
I20260130 02:58:19.819872 19224 raft_consensus.cc:2468] T af0f90071a334016aa17fae2e3a97fc8 P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d870164263bb44b6b780a557daea3656 in term 1.
I20260130 02:58:19.823837 18868 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50284:
name: "default.Table_78b080efddd94878b9d8b77b5ef3af35"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260130 02:58:19.825285 18868 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_78b080efddd94878b9d8b77b5ef3af35 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260130 02:58:19.825366 20926 raft_consensus.cc:2804] T 13f0ace4d9de43abab08eca32cdb30c1 P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Leader election won for term 1
I20260130 02:58:19.826007 20926 raft_consensus.cc:697] T 13f0ace4d9de43abab08eca32cdb30c1 P 371a36d219164974abdf848841528152 [term 1 LEADER]: Becoming Leader. State: Replica: 371a36d219164974abdf848841528152, State: Running, Role: LEADER
I20260130 02:58:19.826779 20926 consensus_queue.cc:237] T 13f0ace4d9de43abab08eca32cdb30c1 P 371a36d219164974abdf848841528152 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
W20260130 02:58:19.829329 18860 catalog_manager.cc:4952] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767): delete failed for tablet e364f4e70e5749e083e14e7fa7416ff0 because the tablet was not found. No further retry: Not found: Tablet not found: e364f4e70e5749e083e14e7fa7416ff0
I20260130 02:58:19.830983 18874 catalog_manager.cc:5746] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571) does not have the latest schema for tablet b1e8b78eff0d4a0b92059adb2de4a3ce (table default.Table_d0b4516d24384e5386c1debaf2543adb [id=f1342f9497b74e7a9ecad1de8082578d]). Expected version 1 got 0
W20260130 02:58:19.835848 19159 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.100.66:40767, user_credentials={real_user=slave}} blocked reactor thread for 43673.6us
I20260130 02:58:19.841568 18874 catalog_manager.cc:5746] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571) does not have the latest schema for tablet 7caf7cbad5054396b5617094e4d536a0 (table default.Table_007e969f70d345a7bcb3dfbaabbc025d [id=77d4fa8705a3419cb0a84d3ca1bb4ce3]). Expected version 1 got 0
I20260130 02:58:19.849601 20899 tablet_replica.cc:333] T 04f4ac27800c4ac3a79ae12d35479757 P 78c2c46c7520492db35d51c244fb4f74: stopping tablet replica
I20260130 02:58:19.855005 18874 catalog_manager.cc:5746] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571) does not have the latest schema for tablet 274530359b174ff79f046c2d4a6afa76 (table default.Table_c4f4904b6fac4b9880979cf0314e71c1 [id=c456c6418c93499a9d08f652720854ca]). Expected version 1 got 0
I20260130 02:58:19.858234 20916 ts_tablet_manager.cc:1929] T 04f4ac27800c4ac3a79ae12d35479757 P d870164263bb44b6b780a557daea3656: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260130 02:58:19.858870 20916 log.cc:1199] T 04f4ac27800c4ac3a79ae12d35479757 P d870164263bb44b6b780a557daea3656: Deleting WAL directory at /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-2/wal/wals/04f4ac27800c4ac3a79ae12d35479757
I20260130 02:58:19.860323 20916 ts_tablet_manager.cc:1950] T 04f4ac27800c4ac3a79ae12d35479757 P d870164263bb44b6b780a557daea3656: Deleting consensus metadata
I20260130 02:58:19.860805 18867 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50284:
name: "default.Table_6f53c17cfd144031b2d4b112fb2d88eb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260130 02:58:19.863411 19625 tablet_bootstrap.cc:492] T 13f0ace4d9de43abab08eca32cdb30c1 P d870164263bb44b6b780a557daea3656: No bootstrap required, opened a new log
I20260130 02:58:19.865060 19625 ts_tablet_manager.cc:1403] T 13f0ace4d9de43abab08eca32cdb30c1 P d870164263bb44b6b780a557daea3656: Time spent bootstrapping tablet: real 0.223s user 0.010s sys 0.001s
I20260130 02:58:19.867324 19625 raft_consensus.cc:359] T 13f0ace4d9de43abab08eca32cdb30c1 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:19.867962 19625 raft_consensus.cc:385] T 13f0ace4d9de43abab08eca32cdb30c1 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:19.868244 19625 raft_consensus.cc:740] T 13f0ace4d9de43abab08eca32cdb30c1 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d870164263bb44b6b780a557daea3656, State: Initialized, Role: FOLLOWER
I20260130 02:58:19.864568 20916 tablet_replica.cc:333] T 8e57ae31ab3941f789ac69627384e3b0 P d870164263bb44b6b780a557daea3656: stopping tablet replica
I20260130 02:58:19.869544 20916 raft_consensus.cc:2243] T 8e57ae31ab3941f789ac69627384e3b0 P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260130 02:58:19.870527 20916 raft_consensus.cc:2272] T 8e57ae31ab3941f789ac69627384e3b0 P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260130 02:58:19.871857 18860 catalog_manager.cc:4985] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): tablet 04f4ac27800c4ac3a79ae12d35479757 (table default.Table_ddfb5df90fdb40a087ac434d088bbf0e [id=1e72b7dd669d4539b61884ba63ff7183]) successfully deleted
W20260130 02:58:19.873081 18860 catalog_manager.cc:4952] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): delete failed for tablet 7d96e93d95f14af79cfb268626559b1e because the tablet was not found. No further retry: Not found: Tablet not found: 7d96e93d95f14af79cfb268626559b1e
W20260130 02:58:19.873689 18860 catalog_manager.cc:4965] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): delete failed for tablet 1129f72158ec4bae89da3c94cdb70839 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1129f72158ec4bae89da3c94cdb70839 already in progress: creating tablet
I20260130 02:58:19.873361 19625 consensus_queue.cc:260] T 13f0ace4d9de43abab08eca32cdb30c1 P d870164263bb44b6b780a557daea3656 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
W20260130 02:58:19.874440 18867 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6f53c17cfd144031b2d4b112fb2d88eb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260130 02:58:19.874670 20916 ts_tablet_manager.cc:1916] T 8e57ae31ab3941f789ac69627384e3b0 P d870164263bb44b6b780a557daea3656: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260130 02:58:19.876526 19807 tablet_bootstrap.cc:492] T 320dbe98a32a43b9899bbeed80b20f55 P 371a36d219164974abdf848841528152: No bootstrap required, opened a new log
I20260130 02:58:19.876957 19807 ts_tablet_manager.cc:1403] T 320dbe98a32a43b9899bbeed80b20f55 P 371a36d219164974abdf848841528152: Time spent bootstrapping tablet: real 0.297s user 0.018s sys 0.002s
I20260130 02:58:19.869683 19840 tablet_bootstrap.cc:492] T 2b8034a92b2043e5b57cf02f1ee6296d P 78c2c46c7520492db35d51c244fb4f74: No bootstrap required, opened a new log
I20260130 02:58:19.878986 19840 ts_tablet_manager.cc:1403] T 2b8034a92b2043e5b57cf02f1ee6296d P 78c2c46c7520492db35d51c244fb4f74: Time spent bootstrapping tablet: real 0.371s user 0.012s sys 0.002s
I20260130 02:58:19.881286 19840 raft_consensus.cc:359] T 2b8034a92b2043e5b57cf02f1ee6296d P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:19.881963 19840 raft_consensus.cc:385] T 2b8034a92b2043e5b57cf02f1ee6296d P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:19.882272 19840 raft_consensus.cc:740] T 2b8034a92b2043e5b57cf02f1ee6296d P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78c2c46c7520492db35d51c244fb4f74, State: Initialized, Role: FOLLOWER
I20260130 02:58:19.883008 19840 consensus_queue.cc:260] T 2b8034a92b2043e5b57cf02f1ee6296d P 78c2c46c7520492db35d51c244fb4f74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
W20260130 02:58:19.884541 19159 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.100.66:40767, user_credentials={real_user=slave}} blocked reactor thread for 47697.1us
I20260130 02:58:19.887341 19807 raft_consensus.cc:359] T 320dbe98a32a43b9899bbeed80b20f55 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:19.888015 19807 raft_consensus.cc:385] T 320dbe98a32a43b9899bbeed80b20f55 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:19.888306 19807 raft_consensus.cc:740] T 320dbe98a32a43b9899bbeed80b20f55 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 371a36d219164974abdf848841528152, State: Initialized, Role: FOLLOWER
I20260130 02:58:19.888906 18874 catalog_manager.cc:5746] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571) does not have the latest schema for tablet 127a84346b9f473d9e7f739b1e65feb8 (table default.Table_805e07e3546e447d8c8b8029477013cc [id=1515db389f284e2f9ed1a3cd0061d0f1]). Expected version 7 got 6
I20260130 02:58:19.888983 19807 consensus_queue.cc:260] T 320dbe98a32a43b9899bbeed80b20f55 P 371a36d219164974abdf848841528152 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:19.894515 20897 ts_tablet_manager.cc:1950] T 3389db58493448aea3cf00aca411047c P 371a36d219164974abdf848841528152: Deleting consensus metadata
I20260130 02:58:19.894433 18868 master_service.cc:954] ReplaceTablet: received request to replace tablet a8a5038e9def44eb8b3b1e4aeae25652 from {username='slave'} at 127.0.0.1:50284
I20260130 02:58:19.909147 19625 ts_tablet_manager.cc:1434] T 13f0ace4d9de43abab08eca32cdb30c1 P d870164263bb44b6b780a557daea3656: Time spent starting tablet: real 0.044s user 0.005s sys 0.000s
I20260130 02:58:19.910144 19625 tablet_bootstrap.cc:492] T 12c3f3ec523040e7b7cd356e3da57e0d P d870164263bb44b6b780a557daea3656: Bootstrap starting.
I20260130 02:58:19.910707 18874 catalog_manager.cc:5746] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571) does not have the latest schema for tablet 9fe0448bcc4a4346b3c67fc3f626ed3c (table default.Table_5c1c447e85524ac5a61ffd8a48509d36 [id=70aaf21ad9bd4d959f46939c6b4a4880]). Expected version 1 got 0
I20260130 02:58:19.912935 20897 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42870 (request call id 328) took 1320 ms. Trace:
I20260130 02:58:19.917917 19625 tablet_bootstrap.cc:654] T 12c3f3ec523040e7b7cd356e3da57e0d P d870164263bb44b6b780a557daea3656: Neither blocks nor log segments found. Creating new log.
W20260130 02:58:19.923406 20909 lock_manager.cc:585] Waited 1 seconds to obtain row lock on key '<redacted>' (key hash 16383881659362995755) tablet 00000000000000000000000000000000 cur holder 0x7b5800064e00
I20260130 02:58:19.926278 18874 catalog_manager.cc:5746] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571) does not have the latest schema for tablet a8a5038e9def44eb8b3b1e4aeae25652 (table default.Table_dee034aec0b24982b8dc2a3fa1b326f1 [id=66033e75b3814a8584311cf7e4a07ad3]). Expected version 1 got 0
W20260130 02:58:19.927877 18860 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet 25fd48e693424537b91ba7c3fcc96878 (table default.Table_ebc8b8379c274c18b0a06a919dbe0ee0 [id=28156e0843674fce88f5914a3e4195ae], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 25fd48e693424537b91ba7c3fcc96878 consensus state has no leader
I20260130 02:58:19.903288 19425 heartbeater.cc:344] Connected to a master server at 127.18.100.124:34629
I20260130 02:58:19.936262 19840 ts_tablet_manager.cc:1434] T 2b8034a92b2043e5b57cf02f1ee6296d P 78c2c46c7520492db35d51c244fb4f74: Time spent starting tablet: real 0.057s user 0.004s sys 0.000s
I20260130 02:58:19.939927 19807 ts_tablet_manager.cc:1434] T 320dbe98a32a43b9899bbeed80b20f55 P 371a36d219164974abdf848841528152: Time spent starting tablet: real 0.063s user 0.006s sys 0.000s
I20260130 02:58:19.962937 20864 consensus_queue.cc:1048] T 3aa76993beb94b12bc706a76db26a03b P d870164263bb44b6b780a557daea3656 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260130 02:58:19.976256 20916 ts_tablet_manager.cc:1929] T 8e57ae31ab3941f789ac69627384e3b0 P d870164263bb44b6b780a557daea3656: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260130 02:58:19.976668 20916 log.cc:1199] T 8e57ae31ab3941f789ac69627384e3b0 P d870164263bb44b6b780a557daea3656: Deleting WAL directory at /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-2/wal/wals/8e57ae31ab3941f789ac69627384e3b0
I20260130 02:58:19.977365 20916 ts_tablet_manager.cc:1950] T 8e57ae31ab3941f789ac69627384e3b0 P d870164263bb44b6b780a557daea3656: Deleting consensus metadata
I20260130 02:58:19.982791 19202 tablet_service.cc:1505] Processing CreateTablet for tablet 1129f72158ec4bae89da3c94cdb70839 (DEFAULT_TABLE table=default.Table_4dde1730dacb42728dd8375fdaa37615 [id=22dd8f2c7e01431f97cbab7b8d94ea32]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:19.983795 19202 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1129f72158ec4bae89da3c94cdb70839. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:19.984683 19204 tablet_service.cc:1552] Processing DeleteTablet for tablet 1129f72158ec4bae89da3c94cdb70839 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:58:19 UTC) from {username='slave'} at 127.0.0.1:42870
I20260130 02:58:19.989148 20899 raft_consensus.cc:2243] T 04f4ac27800c4ac3a79ae12d35479757 P 78c2c46c7520492db35d51c244fb4f74 [term 1 LEADER]: Raft consensus shutting down.
I20260130 02:58:19.992877 19840 tablet_bootstrap.cc:492] T 39bcc41f10ae4241b370529364454586 P 78c2c46c7520492db35d51c244fb4f74: Bootstrap starting.
I20260130 02:58:19.989955 20899 raft_consensus.cc:2272] T 04f4ac27800c4ac3a79ae12d35479757 P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260130 02:58:19.997195 20899 ts_tablet_manager.cc:1916] T 04f4ac27800c4ac3a79ae12d35479757 P 78c2c46c7520492db35d51c244fb4f74: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260130 02:58:20.000277 19625 tablet_bootstrap.cc:492] T 12c3f3ec523040e7b7cd356e3da57e0d P d870164263bb44b6b780a557daea3656: No bootstrap required, opened a new log
I20260130 02:58:20.001361 19625 ts_tablet_manager.cc:1403] T 12c3f3ec523040e7b7cd356e3da57e0d P d870164263bb44b6b780a557daea3656: Time spent bootstrapping tablet: real 0.091s user 0.009s sys 0.006s
I20260130 02:58:20.003718 19625 raft_consensus.cc:359] T 12c3f3ec523040e7b7cd356e3da57e0d P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:20.005703 18860 catalog_manager.cc:4985] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): tablet 8e57ae31ab3941f789ac69627384e3b0 (table default.Table_1ae68ae3f0404843aca3c72f02678ac6 [id=8a9182f49e18489fafec8dca190c1b2f]) successfully deleted
I20260130 02:58:20.008937 19625 raft_consensus.cc:385] T 12c3f3ec523040e7b7cd356e3da57e0d P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:20.009250 19625 raft_consensus.cc:740] T 12c3f3ec523040e7b7cd356e3da57e0d P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d870164263bb44b6b780a557daea3656, State: Initialized, Role: FOLLOWER
I20260130 02:58:20.009932 19625 consensus_queue.cc:260] T 12c3f3ec523040e7b7cd356e3da57e0d P d870164263bb44b6b780a557daea3656 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:20.015167 20697 raft_consensus.cc:493] T 94355fc1338f479e8798070aff04fcba P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:20.015637 20697 raft_consensus.cc:515] T 94355fc1338f479e8798070aff04fcba P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:20.016763 20920 raft_consensus.cc:493] T 320dbe98a32a43b9899bbeed80b20f55 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:20.017179 20920 raft_consensus.cc:515] T 320dbe98a32a43b9899bbeed80b20f55 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:20.018082 19204 tablet_service.cc:1552] Processing DeleteTablet for tablet 8e57ae31ab3941f789ac69627384e3b0 with delete_type TABLET_DATA_DELETED (replaced by tablet 8b47653c7c3945a68e9bf4680bfc0496) from {username='slave'} at 127.0.0.1:42870
I20260130 02:58:20.018911 20920 leader_election.cc:290] T 320dbe98a32a43b9899bbeed80b20f55 P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d870164263bb44b6b780a557daea3656 (127.18.100.67:39571), 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767)
I20260130 02:58:20.021147 19527 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "94355fc1338f479e8798070aff04fcba" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
W20260130 02:58:20.022365 19313 leader_election.cc:343] T 94355fc1338f479e8798070aff04fcba P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:20.018826 20697 leader_election.cc:290] T 94355fc1338f479e8798070aff04fcba P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), d870164263bb44b6b780a557daea3656 (127.18.100.67:39571)
I20260130 02:58:20.026032 19625 ts_tablet_manager.cc:1434] T 12c3f3ec523040e7b7cd356e3da57e0d P d870164263bb44b6b780a557daea3656: Time spent starting tablet: real 0.024s user 0.004s sys 0.000s
I20260130 02:58:20.027041 19625 tablet_bootstrap.cc:492] T 320dbe98a32a43b9899bbeed80b20f55 P d870164263bb44b6b780a557daea3656: Bootstrap starting.
I20260130 02:58:20.033461 18868 catalog_manager.cc:5941] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4: Sending DeleteTablet for 3 replicas of tablet a8a5038e9def44eb8b3b1e4aeae25652
I20260130 02:58:20.035323 19356 tablet_service.cc:1552] Processing DeleteTablet for tablet a8a5038e9def44eb8b3b1e4aeae25652 with delete_type TABLET_DATA_DELETED (replaced by tablet 475b435b2967418f927fa172d2ff04ea) from {username='slave'} at 127.0.0.1:60110
I20260130 02:58:20.036815 18868 catalog_manager.cc:6724] ReplaceTablet: tablet a8a5038e9def44eb8b3b1e4aeae25652 deleted and replaced by tablet 475b435b2967418f927fa172d2ff04ea
I20260130 02:58:20.036861 19501 tablet_service.cc:1552] Processing DeleteTablet for tablet a8a5038e9def44eb8b3b1e4aeae25652 with delete_type TABLET_DATA_DELETED (replaced by tablet 475b435b2967418f927fa172d2ff04ea) from {username='slave'} at 127.0.0.1:59068
I20260130 02:58:20.037638 20916 tablet_replica.cc:333] T a8a5038e9def44eb8b3b1e4aeae25652 P d870164263bb44b6b780a557daea3656: stopping tablet replica
I20260130 02:58:20.043290 19225 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "94355fc1338f479e8798070aff04fcba" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152" is_pre_election: true
W20260130 02:58:20.044157 20972 log.cc:927] Time spent T 3aa76993beb94b12bc706a76db26a03b P 78c2c46c7520492db35d51c244fb4f74: Append to log took a long time: real 0.065s user 0.001s sys 0.000s
I20260130 02:58:20.044682 20916 raft_consensus.cc:2243] T a8a5038e9def44eb8b3b1e4aeae25652 P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260130 02:58:20.046831 20916 raft_consensus.cc:2272] T a8a5038e9def44eb8b3b1e4aeae25652 P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260130 02:58:20.049793 19807 tablet_bootstrap.cc:492] T 12c3f3ec523040e7b7cd356e3da57e0d P 371a36d219164974abdf848841528152: Bootstrap starting.
I20260130 02:58:20.049849 20916 ts_tablet_manager.cc:1916] T a8a5038e9def44eb8b3b1e4aeae25652 P d870164263bb44b6b780a557daea3656: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260130 02:58:19.999433 19840 tablet_bootstrap.cc:654] T 39bcc41f10ae4241b370529364454586 P 78c2c46c7520492db35d51c244fb4f74: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:20.083254 19501 tablet_service.cc:1505] Processing CreateTablet for tablet 532e867fd5944d468d7672d2a5c92f3c (DEFAULT_TABLE table=default.Table_0ca1d7370d2f416782b386c5b47111ab [id=3ffe1a273d4f452c835278a8e7f10102]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:20.084244 19501 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 532e867fd5944d468d7672d2a5c92f3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:20.091149 19356 tablet_service.cc:1505] Processing CreateTablet for tablet 532e867fd5944d468d7672d2a5c92f3c (DEFAULT_TABLE table=default.Table_0ca1d7370d2f416782b386c5b47111ab [id=3ffe1a273d4f452c835278a8e7f10102]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:19.946480 20897 rpcz_store.cc:276] 0130 02:58:18.592455 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:58:18.592571 (+ 116us) service_pool.cc:225] Handling call
0130 02:58:19.304526 (+711955us) ts_tablet_manager.cc:1151] Deleting tablet 3389db58493448aea3cf00aca411047c
0130 02:58:19.304566 (+ 40us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0130 02:58:19.559359 (+254793us) tablet_metadata.cc:641] Metadata flushed
0130 02:58:19.753466 (+194107us) tablet_metadata.cc:641] Metadata flushed
0130 02:58:19.912861 (+159395us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":40515,"tablet-delete.queue_time_us":683890}
I20260130 02:58:20.099263 20897 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42870 (request call id 329) took 1332 ms. Trace:
I20260130 02:58:20.099743 20897 rpcz_store.cc:276] 0130 02:58:18.766413 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:58:18.766527 (+ 114us) service_pool.cc:225] Handling call
0130 02:58:20.098976 (+1332449us) ts_tablet_manager.cc:1151] Deleting tablet 1ca34772ef484c1f809a99d126e28eff
0130 02:58:20.099026 (+ 50us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0130 02:58:20.099203 (+ 177us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1331852}
I20260130 02:58:20.100973 20897 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42870 (request call id 330) took 1312 ms. Trace:
I20260130 02:58:20.101444 20897 rpcz_store.cc:276] 0130 02:58:18.788188 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:58:18.788312 (+ 124us) service_pool.cc:225] Handling call
0130 02:58:20.100716 (+1312404us) ts_tablet_manager.cc:1151] Deleting tablet ec28ff12927d450ead16d26cf016138d
0130 02:58:20.100758 (+ 42us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0130 02:58:20.100925 (+ 167us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1311819}
I20260130 02:58:20.103550 19840 tablet_bootstrap.cc:492] T 39bcc41f10ae4241b370529364454586 P 78c2c46c7520492db35d51c244fb4f74: No bootstrap required, opened a new log
I20260130 02:58:20.104005 19840 ts_tablet_manager.cc:1403] T 39bcc41f10ae4241b370529364454586 P 78c2c46c7520492db35d51c244fb4f74: Time spent bootstrapping tablet: real 0.111s user 0.009s sys 0.001s
I20260130 02:58:20.084245 19625 tablet_bootstrap.cc:654] T 320dbe98a32a43b9899bbeed80b20f55 P d870164263bb44b6b780a557daea3656: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:20.114646 19360 tablet_service.cc:1505] Processing CreateTablet for tablet 30cc52136bb3422895d95c0fed0c6798 (DEFAULT_TABLE table=default.Table_f619de75da8e4d23b4d641d30285e0e7 [id=2dc3a57f750e4104b99f4589fdb29e3c]), partition=RANGE (key) PARTITION UNBOUNDED
W20260130 02:58:20.117653 19314 leader_election.cc:343] T 94355fc1338f479e8798070aff04fcba P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:20.117962 19358 tablet_service.cc:1505] Processing CreateTablet for tablet bc04be1284414a8699d3c4b88bf62813 (DEFAULT_TABLE table=default.Table_2414288f0dc54e5f835f876060927a9b [id=b0c6f2d9483c47da8ff3e65f0400a593]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:20.118120 19314 leader_election.cc:304] T 94355fc1338f479e8798070aff04fcba P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78c2c46c7520492db35d51c244fb4f74; no voters: 371a36d219164974abdf848841528152, d870164263bb44b6b780a557daea3656
I20260130 02:58:20.119305 20942 raft_consensus.cc:2749] T 94355fc1338f479e8798070aff04fcba P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260130 02:58:20.120718 19354 tablet_service.cc:1505] Processing CreateTablet for tablet 8b47653c7c3945a68e9bf4680bfc0496 (DEFAULT_TABLE table=default.Table_1ae68ae3f0404843aca3c72f02678ac6 [id=8a9182f49e18489fafec8dca190c1b2f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:20.106312 19840 raft_consensus.cc:359] T 39bcc41f10ae4241b370529364454586 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:20.122438 19840 raft_consensus.cc:385] T 39bcc41f10ae4241b370529364454586 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:20.124130 20916 ts_tablet_manager.cc:1929] T a8a5038e9def44eb8b3b1e4aeae25652 P d870164263bb44b6b780a557daea3656: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260130 02:58:20.116987 19505 tablet_service.cc:1505] Processing CreateTablet for tablet bc04be1284414a8699d3c4b88bf62813 (DEFAULT_TABLE table=default.Table_2414288f0dc54e5f835f876060927a9b [id=b0c6f2d9483c47da8ff3e65f0400a593]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:20.124502 20916 log.cc:1199] T a8a5038e9def44eb8b3b1e4aeae25652 P d870164263bb44b6b780a557daea3656: Deleting WAL directory at /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-2/wal/wals/a8a5038e9def44eb8b3b1e4aeae25652
I20260130 02:58:20.124959 19505 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bc04be1284414a8699d3c4b88bf62813. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:20.125196 20916 ts_tablet_manager.cc:1950] T a8a5038e9def44eb8b3b1e4aeae25652 P d870164263bb44b6b780a557daea3656: Deleting consensus metadata
I20260130 02:58:20.125535 19625 tablet_bootstrap.cc:492] T 320dbe98a32a43b9899bbeed80b20f55 P d870164263bb44b6b780a557daea3656: No bootstrap required, opened a new log
I20260130 02:58:20.125937 19625 ts_tablet_manager.cc:1403] T 320dbe98a32a43b9899bbeed80b20f55 P d870164263bb44b6b780a557daea3656: Time spent bootstrapping tablet: real 0.099s user 0.009s sys 0.000s
I20260130 02:58:20.127701 19202 tablet_service.cc:1552] Processing DeleteTablet for tablet a8a5038e9def44eb8b3b1e4aeae25652 with delete_type TABLET_DATA_DELETED (replaced by tablet 475b435b2967418f927fa172d2ff04ea) from {username='slave'} at 127.0.0.1:42870
I20260130 02:58:20.127775 18860 catalog_manager.cc:4985] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): tablet a8a5038e9def44eb8b3b1e4aeae25652 (table default.Table_dee034aec0b24982b8dc2a3fa1b326f1 [id=66033e75b3814a8584311cf7e4a07ad3]) successfully deleted
I20260130 02:58:20.128786 19625 raft_consensus.cc:359] T 320dbe98a32a43b9899bbeed80b20f55 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:20.129400 19625 raft_consensus.cc:385] T 320dbe98a32a43b9899bbeed80b20f55 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:20.129693 19625 raft_consensus.cc:740] T 320dbe98a32a43b9899bbeed80b20f55 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d870164263bb44b6b780a557daea3656, State: Initialized, Role: FOLLOWER
I20260130 02:58:20.131052 19351 tablet_service.cc:1505] Processing CreateTablet for tablet 3738385130c044e0a5a5803da74c567d (DEFAULT_TABLE table=default.Table_ddfb5df90fdb40a087ac434d088bbf0e [id=1e72b7dd669d4539b61884ba63ff7183]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:20.120333 19503 tablet_service.cc:1505] Processing CreateTablet for tablet 8b47653c7c3945a68e9bf4680bfc0496 (DEFAULT_TABLE table=default.Table_1ae68ae3f0404843aca3c72f02678ac6 [id=8a9182f49e18489fafec8dca190c1b2f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:20.131873 19204 tablet_service.cc:1505] Processing CreateTablet for tablet 532e867fd5944d468d7672d2a5c92f3c (DEFAULT_TABLE table=default.Table_0ca1d7370d2f416782b386c5b47111ab [id=3ffe1a273d4f452c835278a8e7f10102]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:20.132799 19503 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8b47653c7c3945a68e9bf4680bfc0496. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:20.132808 19204 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 532e867fd5944d468d7672d2a5c92f3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:20.134996 18861 catalog_manager.cc:4985] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021): tablet 3389db58493448aea3cf00aca411047c (table default.Table_22d10332a83f4700ac21892bab67f46b [id=da3a9cb1f6d5419abc1afc93659b5324]) successfully deleted
I20260130 02:58:20.134716 19625 consensus_queue.cc:260] T 320dbe98a32a43b9899bbeed80b20f55 P d870164263bb44b6b780a557daea3656 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
W20260130 02:58:20.135576 18861 catalog_manager.cc:4952] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021): delete failed for tablet 1ca34772ef484c1f809a99d126e28eff because the tablet was not found. No further retry: Not found: Tablet not found: 1ca34772ef484c1f809a99d126e28eff
I20260130 02:58:20.137075 19496 tablet_service.cc:1505] Processing CreateTablet for tablet 3738385130c044e0a5a5803da74c567d (DEFAULT_TABLE table=default.Table_ddfb5df90fdb40a087ac434d088bbf0e [id=1e72b7dd669d4539b61884ba63ff7183]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:20.137979 19496 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3738385130c044e0a5a5803da74c567d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:20.138867 19197 tablet_service.cc:1505] Processing CreateTablet for tablet bc04be1284414a8699d3c4b88bf62813 (DEFAULT_TABLE table=default.Table_2414288f0dc54e5f835f876060927a9b [id=b0c6f2d9483c47da8ff3e65f0400a593]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:20.139784 19197 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bc04be1284414a8699d3c4b88bf62813. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:20.140400 19502 tablet_service.cc:1505] Processing CreateTablet for tablet e691c238199b4914bcc3fd248f31420c (DEFAULT_TABLE table=default.Table_7fe8c354c0dc4cfba0e84ed33e4c2506 [id=7523d671e9ac4814acff5060fd282eea]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:20.141311 19502 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e691c238199b4914bcc3fd248f31420c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:20.122735 19840 raft_consensus.cc:740] T 39bcc41f10ae4241b370529364454586 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78c2c46c7520492db35d51c244fb4f74, State: Initialized, Role: FOLLOWER
I20260130 02:58:20.142973 19350 tablet_service.cc:1505] Processing CreateTablet for tablet e691c238199b4914bcc3fd248f31420c (DEFAULT_TABLE table=default.Table_7fe8c354c0dc4cfba0e84ed33e4c2506 [id=7523d671e9ac4814acff5060fd282eea]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:20.146293 19625 ts_tablet_manager.cc:1434] T 320dbe98a32a43b9899bbeed80b20f55 P d870164263bb44b6b780a557daea3656: Time spent starting tablet: real 0.020s user 0.002s sys 0.003s
I20260130 02:58:20.147161 19625 tablet_bootstrap.cc:492] T 2c79a3aecab844698e70834999e1caac P d870164263bb44b6b780a557daea3656: Bootstrap starting.
I20260130 02:58:20.150903 19840 consensus_queue.cc:260] T 39bcc41f10ae4241b370529364454586 P 78c2c46c7520492db35d51c244fb4f74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:20.156883 19807 tablet_bootstrap.cc:654] T 12c3f3ec523040e7b7cd356e3da57e0d P 371a36d219164974abdf848841528152: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:20.163797 19202 tablet_service.cc:1505] Processing CreateTablet for tablet 30cc52136bb3422895d95c0fed0c6798 (DEFAULT_TABLE table=default.Table_f619de75da8e4d23b4d641d30285e0e7 [id=2dc3a57f750e4104b99f4589fdb29e3c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:20.164815 19202 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 30cc52136bb3422895d95c0fed0c6798. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:20.113253 19501 tablet_service.cc:1505] Processing CreateTablet for tablet 30cc52136bb3422895d95c0fed0c6798 (DEFAULT_TABLE table=default.Table_f619de75da8e4d23b4d641d30285e0e7 [id=2dc3a57f750e4104b99f4589fdb29e3c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:20.166883 19501 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 30cc52136bb3422895d95c0fed0c6798. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:20.167706 19840 ts_tablet_manager.cc:1434] T 39bcc41f10ae4241b370529364454586 P 78c2c46c7520492db35d51c244fb4f74: Time spent starting tablet: real 0.063s user 0.006s sys 0.000s
I20260130 02:58:20.173676 20899 ts_tablet_manager.cc:1929] T 04f4ac27800c4ac3a79ae12d35479757 P 78c2c46c7520492db35d51c244fb4f74: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260130 02:58:20.174132 20899 log.cc:1199] T 04f4ac27800c4ac3a79ae12d35479757 P 78c2c46c7520492db35d51c244fb4f74: Deleting WAL directory at /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-1/wal/wals/04f4ac27800c4ac3a79ae12d35479757
I20260130 02:58:20.177104 20897 tablet_replica.cc:333] T 04f4ac27800c4ac3a79ae12d35479757 P 371a36d219164974abdf848841528152: stopping tablet replica
I20260130 02:58:20.177407 19625 tablet_bootstrap.cc:654] T 2c79a3aecab844698e70834999e1caac P d870164263bb44b6b780a557daea3656: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:20.178097 20897 raft_consensus.cc:2243] T 04f4ac27800c4ac3a79ae12d35479757 P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260130 02:58:20.175015 20899 ts_tablet_manager.cc:1950] T 04f4ac27800c4ac3a79ae12d35479757 P 78c2c46c7520492db35d51c244fb4f74: Deleting consensus metadata
I20260130 02:58:20.189498 20897 raft_consensus.cc:2272] T 04f4ac27800c4ac3a79ae12d35479757 P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260130 02:58:20.194245 19205 tablet_service.cc:1505] Processing CreateTablet for tablet 8b47653c7c3945a68e9bf4680bfc0496 (DEFAULT_TABLE table=default.Table_1ae68ae3f0404843aca3c72f02678ac6 [id=8a9182f49e18489fafec8dca190c1b2f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:20.194643 19199 tablet_service.cc:1505] Processing CreateTablet for tablet e691c238199b4914bcc3fd248f31420c (DEFAULT_TABLE table=default.Table_7fe8c354c0dc4cfba0e84ed33e4c2506 [id=7523d671e9ac4814acff5060fd282eea]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:20.194859 19197 tablet_service.cc:1505] Processing CreateTablet for tablet 3738385130c044e0a5a5803da74c567d (DEFAULT_TABLE table=default.Table_ddfb5df90fdb40a087ac434d088bbf0e [id=1e72b7dd669d4539b61884ba63ff7183]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:20.195351 19205 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8b47653c7c3945a68e9bf4680bfc0496. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:20.196132 19199 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e691c238199b4914bcc3fd248f31420c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:20.197395 19197 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3738385130c044e0a5a5803da74c567d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:20.232007 20956 consensus_queue.cc:1048] T 3aa76993beb94b12bc706a76db26a03b P d870164263bb44b6b780a557daea3656 [LEADER]: Connected to new peer: Peer: permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260130 02:58:20.243713 19625 tablet_bootstrap.cc:492] T 2c79a3aecab844698e70834999e1caac P d870164263bb44b6b780a557daea3656: No bootstrap required, opened a new log
I20260130 02:58:20.244351 19625 ts_tablet_manager.cc:1403] T 2c79a3aecab844698e70834999e1caac P d870164263bb44b6b780a557daea3656: Time spent bootstrapping tablet: real 0.097s user 0.005s sys 0.006s
I20260130 02:58:20.246791 19625 raft_consensus.cc:359] T 2c79a3aecab844698e70834999e1caac P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:20.247598 19625 raft_consensus.cc:385] T 2c79a3aecab844698e70834999e1caac P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:20.248189 19625 raft_consensus.cc:740] T 2c79a3aecab844698e70834999e1caac P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d870164263bb44b6b780a557daea3656, State: Initialized, Role: FOLLOWER
I20260130 02:58:20.250167 20942 consensus_queue.cc:1048] T e12d130bcdf34965888753f12c1ff848 P 78c2c46c7520492db35d51c244fb4f74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260130 02:58:20.255525 18861 catalog_manager.cc:4952] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021): delete failed for tablet ec28ff12927d450ead16d26cf016138d because the tablet was not found. No further retry: Not found: Tablet not found: ec28ff12927d450ead16d26cf016138d
I20260130 02:58:20.259650 19625 consensus_queue.cc:260] T 2c79a3aecab844698e70834999e1caac P d870164263bb44b6b780a557daea3656 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:20.264573 19625 ts_tablet_manager.cc:1434] T 2c79a3aecab844698e70834999e1caac P d870164263bb44b6b780a557daea3656: Time spent starting tablet: real 0.020s user 0.004s sys 0.001s
I20260130 02:58:20.269409 19625 tablet_bootstrap.cc:492] T 133e9bfe175f44aab1a53114abe67e4d P d870164263bb44b6b780a557daea3656: Bootstrap starting.
I20260130 02:58:20.286967 19807 tablet_bootstrap.cc:492] T 12c3f3ec523040e7b7cd356e3da57e0d P 371a36d219164974abdf848841528152: No bootstrap required, opened a new log
I20260130 02:58:20.286899 20697 consensus_queue.cc:1048] T e12d130bcdf34965888753f12c1ff848 P 78c2c46c7520492db35d51c244fb4f74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260130 02:58:20.287626 19807 ts_tablet_manager.cc:1403] T 12c3f3ec523040e7b7cd356e3da57e0d P 371a36d219164974abdf848841528152: Time spent bootstrapping tablet: real 0.262s user 0.011s sys 0.001s
I20260130 02:58:20.289880 19807 raft_consensus.cc:359] T 12c3f3ec523040e7b7cd356e3da57e0d P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:20.290457 19807 raft_consensus.cc:385] T 12c3f3ec523040e7b7cd356e3da57e0d P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:20.290731 19807 raft_consensus.cc:740] T 12c3f3ec523040e7b7cd356e3da57e0d P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 371a36d219164974abdf848841528152, State: Initialized, Role: FOLLOWER
W20260130 02:58:20.291519 19159 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.100.67:39571, user_credentials={real_user=slave}} blocked reactor thread for 66274us
I20260130 02:58:20.293514 19625 tablet_bootstrap.cc:654] T 133e9bfe175f44aab1a53114abe67e4d P d870164263bb44b6b780a557daea3656: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:20.297403 18875 catalog_manager.cc:5746] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767) does not have the latest schema for tablet 345eaaa5625c48d099280b7fbb2e514b (table default.Table_c1886dd9a7934e7797e0e7e0b065d692 [id=7a2d5367224b492b844199088b01b149]). Expected version 1 got 0
I20260130 02:58:20.298468 18875 catalog_manager.cc:5746] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767) does not have the latest schema for tablet b1e8b78eff0d4a0b92059adb2de4a3ce (table default.Table_d0b4516d24384e5386c1debaf2543adb [id=f1342f9497b74e7a9ecad1de8082578d]). Expected version 1 got 0
I20260130 02:58:20.299401 18875 catalog_manager.cc:5746] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767) does not have the latest schema for tablet 127a84346b9f473d9e7f739b1e65feb8 (table default.Table_805e07e3546e447d8c8b8029477013cc [id=1515db389f284e2f9ed1a3cd0061d0f1]). Expected version 7 got 6
I20260130 02:58:20.300014 19525 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "320dbe98a32a43b9899bbeed80b20f55" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
I20260130 02:58:20.300539 19525 raft_consensus.cc:2468] T 320dbe98a32a43b9899bbeed80b20f55 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 371a36d219164974abdf848841528152 in term 0.
I20260130 02:58:20.301054 18875 catalog_manager.cc:5746] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767) does not have the latest schema for tablet 2b0edaad6877460cb7982efc62b00b66 (table default.Table_d95381c590c94802bdc83acef0f0df96 [id=64a8e0e079f34806aa68795f3d548c5e]). Expected version 1 got 0
I20260130 02:58:20.302141 19376 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "320dbe98a32a43b9899bbeed80b20f55" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74" is_pre_election: true
I20260130 02:58:20.302681 19376 raft_consensus.cc:2468] T 320dbe98a32a43b9899bbeed80b20f55 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 371a36d219164974abdf848841528152 in term 0.
I20260130 02:58:20.324404 18875 catalog_manager.cc:5746] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767) does not have the latest schema for tablet 9fe0448bcc4a4346b3c67fc3f626ed3c (table default.Table_5c1c447e85524ac5a61ffd8a48509d36 [id=70aaf21ad9bd4d959f46939c6b4a4880]). Expected version 1 got 0
I20260130 02:58:20.326134 18875 catalog_manager.cc:5746] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767) does not have the latest schema for tablet 274530359b174ff79f046c2d4a6afa76 (table default.Table_c4f4904b6fac4b9880979cf0314e71c1 [id=c456c6418c93499a9d08f652720854ca]). Expected version 1 got 0
I20260130 02:58:20.326756 18875 catalog_manager.cc:5746] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767) does not have the latest schema for tablet 49efcfced0b74c48962393149ac2a500 (table default.Table_2e81f0a0cbc449a397141b717db17025 [id=a9c919f57c0c437cb35a6921103e45a6]). Expected version 1 got 0
I20260130 02:58:20.327587 18875 catalog_manager.cc:5746] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767) does not have the latest schema for tablet 657e9b0d8c05413f904129dce745710b (table default.Table_526677c8ef4144bd9de7a54a5e78e2be [id=b16908b75587459597a978c71756451c]). Expected version 1 got 0
I20260130 02:58:20.330155 18875 catalog_manager.cc:5746] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767) does not have the latest schema for tablet 25fd48e693424537b91ba7c3fcc96878 (table default.Table_ebc8b8379c274c18b0a06a919dbe0ee0 [id=28156e0843674fce88f5914a3e4195ae]). Expected version 1 got 0
I20260130 02:58:20.330763 20897 ts_tablet_manager.cc:1916] T 04f4ac27800c4ac3a79ae12d35479757 P 371a36d219164974abdf848841528152: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260130 02:58:20.365190 18875 catalog_manager.cc:5746] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767) does not have the latest schema for tablet 838e49e5e8874db393bdc15a1c92f87e (table default.Table_da3d581b15a94840898777f74611994e [id=b7e241ced53e4160ac6fa1f65ef3674e]). Expected version 1 got 0
I20260130 02:58:20.365991 20956 raft_consensus.cc:493] T 2c79a3aecab844698e70834999e1caac P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:20.366436 20956 raft_consensus.cc:515] T 2c79a3aecab844698e70834999e1caac P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:20.368250 20956 leader_election.cc:290] T 2c79a3aecab844698e70834999e1caac P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767)
I20260130 02:58:20.369712 19221 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2c79a3aecab844698e70834999e1caac" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152" is_pre_election: true
W20260130 02:58:20.371017 19463 leader_election.cc:343] T 2c79a3aecab844698e70834999e1caac P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:20.383653 19625 tablet_bootstrap.cc:492] T 133e9bfe175f44aab1a53114abe67e4d P d870164263bb44b6b780a557daea3656: No bootstrap required, opened a new log
I20260130 02:58:20.384133 19625 ts_tablet_manager.cc:1403] T 133e9bfe175f44aab1a53114abe67e4d P d870164263bb44b6b780a557daea3656: Time spent bootstrapping tablet: real 0.115s user 0.030s sys 0.040s
I20260130 02:58:20.411935 19625 raft_consensus.cc:359] T 133e9bfe175f44aab1a53114abe67e4d P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:20.434721 19625 raft_consensus.cc:385] T 133e9bfe175f44aab1a53114abe67e4d P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:20.435043 19625 raft_consensus.cc:740] T 133e9bfe175f44aab1a53114abe67e4d P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d870164263bb44b6b780a557daea3656, State: Initialized, Role: FOLLOWER
I20260130 02:58:20.435708 19625 consensus_queue.cc:260] T 133e9bfe175f44aab1a53114abe67e4d P d870164263bb44b6b780a557daea3656 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:20.442294 19807 consensus_queue.cc:260] T 12c3f3ec523040e7b7cd356e3da57e0d P 371a36d219164974abdf848841528152 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:20.449887 19807 ts_tablet_manager.cc:1434] T 12c3f3ec523040e7b7cd356e3da57e0d P 371a36d219164974abdf848841528152: Time spent starting tablet: real 0.162s user 0.005s sys 0.000s
I20260130 02:58:20.450774 19807 tablet_bootstrap.cc:492] T 2c79a3aecab844698e70834999e1caac P 371a36d219164974abdf848841528152: Bootstrap starting.
I20260130 02:58:20.450924 19625 ts_tablet_manager.cc:1434] T 133e9bfe175f44aab1a53114abe67e4d P d870164263bb44b6b780a557daea3656: Time spent starting tablet: real 0.066s user 0.005s sys 0.001s
I20260130 02:58:20.454116 19625 tablet_bootstrap.cc:492] T f7607df5efd543438fb2dff3667cc8c8 P d870164263bb44b6b780a557daea3656: Bootstrap starting.
I20260130 02:58:20.463686 19625 tablet_bootstrap.cc:654] T f7607df5efd543438fb2dff3667cc8c8 P d870164263bb44b6b780a557daea3656: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:20.465234 19159 leader_election.cc:304] T 320dbe98a32a43b9899bbeed80b20f55 P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 371a36d219164974abdf848841528152, 78c2c46c7520492db35d51c244fb4f74; no voters:
I20260130 02:58:20.477563 19807 tablet_bootstrap.cc:654] T 2c79a3aecab844698e70834999e1caac P 371a36d219164974abdf848841528152: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:20.489241 20920 raft_consensus.cc:2804] T 320dbe98a32a43b9899bbeed80b20f55 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260130 02:58:20.489650 20920 raft_consensus.cc:493] T 320dbe98a32a43b9899bbeed80b20f55 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260130 02:58:20.489971 20920 raft_consensus.cc:3060] T 320dbe98a32a43b9899bbeed80b20f55 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:20.524062 20956 raft_consensus.cc:493] T 13f0ace4d9de43abab08eca32cdb30c1 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:20.524528 20956 raft_consensus.cc:515] T 13f0ace4d9de43abab08eca32cdb30c1 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:20.526337 20956 leader_election.cc:290] T 13f0ace4d9de43abab08eca32cdb30c1 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767)
I20260130 02:58:20.529227 20920 raft_consensus.cc:515] T 320dbe98a32a43b9899bbeed80b20f55 P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:20.530956 20920 leader_election.cc:290] T 320dbe98a32a43b9899bbeed80b20f55 P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 election: Requested vote from peers d870164263bb44b6b780a557daea3656 (127.18.100.67:39571), 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767)
W20260130 02:58:20.531890 19159 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.100.66:40767, user_credentials={real_user=slave}} blocked reactor thread for 66727.2us
W20260130 02:58:20.532507 19159 consensus_peers.cc:597] T e2aaaf00f53d418b87d5c7af286e4d84 P 371a36d219164974abdf848841528152 -> Peer 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767): Couldn't send request to peer 78c2c46c7520492db35d51c244fb4f74. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e2aaaf00f53d418b87d5c7af286e4d84. This is attempt 1: this message will repeat every 5th retry.
I20260130 02:58:20.549242 19221 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "13f0ace4d9de43abab08eca32cdb30c1" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152" is_pre_election: true
I20260130 02:58:20.563272 19202 tablet_service.cc:1505] Processing CreateTablet for tablet 155ff0c35fc44cb292a28407e6ed937d (DEFAULT_TABLE table=default.Table_dabf999abe834151a0a5b6747a05cac3 [id=038ba9aa798741f6858fa09002d6c3d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:20.564321 19202 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 155ff0c35fc44cb292a28407e6ed937d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:20.571237 19501 tablet_service.cc:1505] Processing CreateTablet for tablet 155ff0c35fc44cb292a28407e6ed937d (DEFAULT_TABLE table=default.Table_dabf999abe834151a0a5b6747a05cac3 [id=038ba9aa798741f6858fa09002d6c3d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:20.572181 19501 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 155ff0c35fc44cb292a28407e6ed937d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:20.574776 19807 tablet_bootstrap.cc:492] T 2c79a3aecab844698e70834999e1caac P 371a36d219164974abdf848841528152: No bootstrap required, opened a new log
I20260130 02:58:20.574672 19525 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "320dbe98a32a43b9899bbeed80b20f55" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656"
I20260130 02:58:20.575170 19807 ts_tablet_manager.cc:1403] T 2c79a3aecab844698e70834999e1caac P 371a36d219164974abdf848841528152: Time spent bootstrapping tablet: real 0.125s user 0.013s sys 0.002s
I20260130 02:58:20.575142 19525 raft_consensus.cc:3060] T 320dbe98a32a43b9899bbeed80b20f55 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:20.579185 19505 tablet_service.cc:1505] Processing CreateTablet for tablet 920f52b975ab480fbf42656171222f09 (DEFAULT_TABLE table=default.Table_78b080efddd94878b9d8b77b5ef3af35 [id=b654822c58224a859ad2dcdd8d0123c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:20.580089 19505 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 920f52b975ab480fbf42656171222f09. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:20.581041 19525 raft_consensus.cc:2468] T 320dbe98a32a43b9899bbeed80b20f55 P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 371a36d219164974abdf848841528152 in term 1.
I20260130 02:58:20.584450 19503 tablet_service.cc:1505] Processing CreateTablet for tablet 66a7487a7ab343a4abca5312afe78862 (DEFAULT_TABLE table=default.Table_6c46e6995d4741dcaaa6ed7bdc765e29 [id=6d91a161cf484239967af58db160c952]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:20.585434 19503 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 66a7487a7ab343a4abca5312afe78862. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:20.585304 19197 tablet_service.cc:1505] Processing CreateTablet for tablet 920f52b975ab480fbf42656171222f09 (DEFAULT_TABLE table=default.Table_78b080efddd94878b9d8b77b5ef3af35 [id=b654822c58224a859ad2dcdd8d0123c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:20.586159 19197 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 920f52b975ab480fbf42656171222f09. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:20.597609 19625 tablet_bootstrap.cc:492] T f7607df5efd543438fb2dff3667cc8c8 P d870164263bb44b6b780a557daea3656: No bootstrap required, opened a new log
I20260130 02:58:20.598018 19625 ts_tablet_manager.cc:1403] T f7607df5efd543438fb2dff3667cc8c8 P d870164263bb44b6b780a557daea3656: Time spent bootstrapping tablet: real 0.144s user 0.008s sys 0.005s
I20260130 02:58:20.600006 19502 tablet_service.cc:1505] Processing CreateTablet for tablet 475b435b2967418f927fa172d2ff04ea (DEFAULT_TABLE table=default.Table_dee034aec0b24982b8dc2a3fa1b326f1 [id=66033e75b3814a8584311cf7e4a07ad3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:20.600044 19199 tablet_service.cc:1505] Processing CreateTablet for tablet 475b435b2967418f927fa172d2ff04ea (DEFAULT_TABLE table=default.Table_dee034aec0b24982b8dc2a3fa1b326f1 [id=66033e75b3814a8584311cf7e4a07ad3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:20.600937 19199 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 475b435b2967418f927fa172d2ff04ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:20.600920 19502 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 475b435b2967418f927fa172d2ff04ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:20.604578 19197 tablet_service.cc:1505] Processing CreateTablet for tablet 66a7487a7ab343a4abca5312afe78862 (DEFAULT_TABLE table=default.Table_6c46e6995d4741dcaaa6ed7bdc765e29 [id=6d91a161cf484239967af58db160c952]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:20.610167 20929 consensus_queue.cc:1048] T af0f90071a334016aa17fae2e3a97fc8 P d870164263bb44b6b780a557daea3656 [LEADER]: Connected to new peer: Peer: permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260130 02:58:20.611860 19625 raft_consensus.cc:359] T f7607df5efd543438fb2dff3667cc8c8 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:20.612517 19625 raft_consensus.cc:385] T f7607df5efd543438fb2dff3667cc8c8 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:20.612753 19625 raft_consensus.cc:740] T f7607df5efd543438fb2dff3667cc8c8 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d870164263bb44b6b780a557daea3656, State: Initialized, Role: FOLLOWER
I20260130 02:58:20.613332 19625 consensus_queue.cc:260] T f7607df5efd543438fb2dff3667cc8c8 P d870164263bb44b6b780a557daea3656 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:20.629974 19496 tablet_service.cc:1505] Processing CreateTablet for tablet 6f5a90b75a0f4851b5db672c41253f7b (DEFAULT_TABLE table=default.Table_6f53c17cfd144031b2d4b112fb2d88eb [id=443cbad5510d463fa8ca878b8628dbda]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:20.631044 19496 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6f5a90b75a0f4851b5db672c41253f7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:20.634538 19625 ts_tablet_manager.cc:1434] T f7607df5efd543438fb2dff3667cc8c8 P d870164263bb44b6b780a557daea3656: Time spent starting tablet: real 0.036s user 0.003s sys 0.004s
I20260130 02:58:20.635442 19625 tablet_bootstrap.cc:492] T ca06f07bb98f45a7bf63aebc28348e08 P d870164263bb44b6b780a557daea3656: Bootstrap starting.
I20260130 02:58:20.626282 18869 catalog_manager.cc:5746] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571) does not have the latest schema for tablet 345eaaa5625c48d099280b7fbb2e514b (table default.Table_c1886dd9a7934e7797e0e7e0b065d692 [id=7a2d5367224b492b844199088b01b149]). Expected version 1 got 0
I20260130 02:58:20.656894 19625 tablet_bootstrap.cc:654] T ca06f07bb98f45a7bf63aebc28348e08 P d870164263bb44b6b780a557daea3656: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:20.663301 19356 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 532e867fd5944d468d7672d2a5c92f3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:20.663754 19349 tablet_service.cc:1505] Processing CreateTablet for tablet 475b435b2967418f927fa172d2ff04ea (DEFAULT_TABLE table=default.Table_dee034aec0b24982b8dc2a3fa1b326f1 [id=66033e75b3814a8584311cf7e4a07ad3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:20.672778 19625 tablet_bootstrap.cc:492] T ca06f07bb98f45a7bf63aebc28348e08 P d870164263bb44b6b780a557daea3656: No bootstrap required, opened a new log
I20260130 02:58:20.673182 19625 ts_tablet_manager.cc:1403] T ca06f07bb98f45a7bf63aebc28348e08 P d870164263bb44b6b780a557daea3656: Time spent bootstrapping tablet: real 0.038s user 0.009s sys 0.001s
I20260130 02:58:20.676083 19625 raft_consensus.cc:359] T ca06f07bb98f45a7bf63aebc28348e08 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:20.676725 19625 raft_consensus.cc:385] T ca06f07bb98f45a7bf63aebc28348e08 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:20.676970 19625 raft_consensus.cc:740] T ca06f07bb98f45a7bf63aebc28348e08 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d870164263bb44b6b780a557daea3656, State: Initialized, Role: FOLLOWER
I20260130 02:58:20.677645 19625 consensus_queue.cc:260] T ca06f07bb98f45a7bf63aebc28348e08 P d870164263bb44b6b780a557daea3656 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:20.682868 19357 tablet_service.cc:1505] Processing CreateTablet for tablet 66a7487a7ab343a4abca5312afe78862 (DEFAULT_TABLE table=default.Table_6c46e6995d4741dcaaa6ed7bdc765e29 [id=6d91a161cf484239967af58db160c952]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:20.684242 19355 tablet_service.cc:1505] Processing CreateTablet for tablet 155ff0c35fc44cb292a28407e6ed937d (DEFAULT_TABLE table=default.Table_dabf999abe834151a0a5b6747a05cac3 [id=038ba9aa798741f6858fa09002d6c3d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:20.686261 19352 tablet_service.cc:1505] Processing CreateTablet for tablet 920f52b975ab480fbf42656171222f09 (DEFAULT_TABLE table=default.Table_78b080efddd94878b9d8b77b5ef3af35 [id=b654822c58224a859ad2dcdd8d0123c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:20.624929 20987 consensus_queue.cc:1048] T 25fd48e693424537b91ba7c3fcc96878 P 371a36d219164974abdf848841528152 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260130 02:58:20.626600 19159 leader_election.cc:304] T 320dbe98a32a43b9899bbeed80b20f55 P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 371a36d219164974abdf848841528152, d870164263bb44b6b780a557daea3656; no voters:
I20260130 02:58:20.628404 19807 raft_consensus.cc:359] T 2c79a3aecab844698e70834999e1caac P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:20.687750 19807 raft_consensus.cc:385] T 2c79a3aecab844698e70834999e1caac P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:20.687498 19376 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2c79a3aecab844698e70834999e1caac" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74" is_pre_election: true
W20260130 02:58:20.687882 19159 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.100.67:39571, user_credentials={real_user=slave}} blocked reactor thread for 61378.5us
I20260130 02:58:20.688022 19807 raft_consensus.cc:740] T 2c79a3aecab844698e70834999e1caac P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 371a36d219164974abdf848841528152, State: Initialized, Role: FOLLOWER
I20260130 02:58:20.688225 19376 raft_consensus.cc:2468] T 2c79a3aecab844698e70834999e1caac P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d870164263bb44b6b780a557daea3656 in term 0.
I20260130 02:58:20.688583 19807 consensus_queue.cc:260] T 2c79a3aecab844698e70834999e1caac P 371a36d219164974abdf848841528152 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:20.690184 19807 ts_tablet_manager.cc:1434] T 2c79a3aecab844698e70834999e1caac P 371a36d219164974abdf848841528152: Time spent starting tablet: real 0.115s user 0.004s sys 0.001s
I20260130 02:58:20.690832 19351 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3738385130c044e0a5a5803da74c567d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:20.690829 20926 raft_consensus.cc:2804] T 320dbe98a32a43b9899bbeed80b20f55 P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Leader election won for term 1
I20260130 02:58:20.691246 20926 raft_consensus.cc:697] T 320dbe98a32a43b9899bbeed80b20f55 P 371a36d219164974abdf848841528152 [term 1 LEADER]: Becoming Leader. State: Replica: 371a36d219164974abdf848841528152, State: Running, Role: LEADER
I20260130 02:58:20.692129 19625 ts_tablet_manager.cc:1434] T ca06f07bb98f45a7bf63aebc28348e08 P d870164263bb44b6b780a557daea3656: Time spent starting tablet: real 0.019s user 0.007s sys 0.000s
I20260130 02:58:20.691959 20926 consensus_queue.cc:237] T 320dbe98a32a43b9899bbeed80b20f55 P 371a36d219164974abdf848841528152 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:20.692978 19625 tablet_bootstrap.cc:492] T a3258852aed24ed582a79f77860e6f0d P d870164263bb44b6b780a557daea3656: Bootstrap starting.
I20260130 02:58:20.687475 19197 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 66a7487a7ab343a4abca5312afe78862. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:20.695261 19350 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e691c238199b4914bcc3fd248f31420c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:20.698160 19625 tablet_bootstrap.cc:654] T a3258852aed24ed582a79f77860e6f0d P d870164263bb44b6b780a557daea3656: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:20.688959 20897 ts_tablet_manager.cc:1929] T 04f4ac27800c4ac3a79ae12d35479757 P 371a36d219164974abdf848841528152: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260130 02:58:20.698963 20897 log.cc:1199] T 04f4ac27800c4ac3a79ae12d35479757 P 371a36d219164974abdf848841528152: Deleting WAL directory at /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-0/wal/wals/04f4ac27800c4ac3a79ae12d35479757
I20260130 02:58:20.699692 20897 ts_tablet_manager.cc:1950] T 04f4ac27800c4ac3a79ae12d35479757 P 371a36d219164974abdf848841528152: Deleting consensus metadata
I20260130 02:58:20.700716 19354 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8b47653c7c3945a68e9bf4680bfc0496. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:20.701890 20897 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42870 (request call id 331) took 1724 ms. Trace:
I20260130 02:58:20.702194 20897 rpcz_store.cc:276] 0130 02:58:18.977397 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:58:18.977564 (+ 167us) service_pool.cc:225] Handling call
0130 02:58:20.176938 (+1199374us) ts_tablet_manager.cc:1151] Deleting tablet 04f4ac27800c4ac3a79ae12d35479757
0130 02:58:20.176983 (+ 45us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0130 02:58:20.501201 (+324218us) tablet_metadata.cc:641] Metadata flushed
0130 02:58:20.688680 (+187479us) tablet_metadata.cc:641] Metadata flushed
0130 02:58:20.701831 (+ 13151us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":10739,"tablet-delete.queue_time_us":1191371}
I20260130 02:58:20.703316 19358 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bc04be1284414a8699d3c4b88bf62813. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:20.704937 18861 catalog_manager.cc:4985] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021): tablet 04f4ac27800c4ac3a79ae12d35479757 (table default.Table_ddfb5df90fdb40a087ac434d088bbf0e [id=1e72b7dd669d4539b61884ba63ff7183]) successfully deleted
I20260130 02:58:20.707221 19360 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 30cc52136bb3422895d95c0fed0c6798. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:20.711751 19840 tablet_bootstrap.cc:492] T 00ddbfe6218c44d788fbf65245a31dc6 P 78c2c46c7520492db35d51c244fb4f74: Bootstrap starting.
I20260130 02:58:20.713778 20897 tablet_replica.cc:333] T 8e57ae31ab3941f789ac69627384e3b0 P 371a36d219164974abdf848841528152: stopping tablet replica
W20260130 02:58:20.715023 18861 catalog_manager.cc:4965] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021): delete failed for tablet 1129f72158ec4bae89da3c94cdb70839 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1129f72158ec4bae89da3c94cdb70839 already in progress: creating tablet
I20260130 02:58:20.716858 20985 consensus_queue.cc:1048] T 25fd48e693424537b91ba7c3fcc96878 P 371a36d219164974abdf848841528152 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260130 02:58:20.719185 20899 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60110 (request call id 311) took 1772 ms. Trace:
I20260130 02:58:20.719604 20899 rpcz_store.cc:276] 0130 02:58:18.946763 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:58:18.946892 (+ 129us) service_pool.cc:225] Handling call
0130 02:58:19.849427 (+902535us) ts_tablet_manager.cc:1151] Deleting tablet 04f4ac27800c4ac3a79ae12d35479757
0130 02:58:19.849470 (+ 43us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0130 02:58:20.069861 (+220391us) tablet_metadata.cc:641] Metadata flushed
0130 02:58:20.173414 (+103553us) tablet_metadata.cc:641] Metadata flushed
0130 02:58:20.719015 (+545601us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":112692,"tablet-delete.queue_time_us":901552}
I20260130 02:58:20.721720 20899 tablet_replica.cc:333] T 8e57ae31ab3941f789ac69627384e3b0 P 78c2c46c7520492db35d51c244fb4f74: stopping tablet replica
I20260130 02:58:20.722631 19349 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 475b435b2967418f927fa172d2ff04ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:20.691975 19807 tablet_bootstrap.cc:492] T f7607df5efd543438fb2dff3667cc8c8 P 371a36d219164974abdf848841528152: Bootstrap starting.
I20260130 02:58:20.729040 19807 tablet_bootstrap.cc:654] T f7607df5efd543438fb2dff3667cc8c8 P 371a36d219164974abdf848841528152: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:20.744024 20986 consensus_queue.cc:1048] T 838e49e5e8874db393bdc15a1c92f87e P 371a36d219164974abdf848841528152 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260130 02:58:20.747399 19159 consensus_peers.cc:597] T a8a5038e9def44eb8b3b1e4aeae25652 P 371a36d219164974abdf848841528152 -> Peer d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): Couldn't send request to peer d870164263bb44b6b780a557daea3656. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a8a5038e9def44eb8b3b1e4aeae25652. This is attempt 1: this message will repeat every 5th retry.
I20260130 02:58:20.756413 20697 raft_consensus.cc:493] T 12c3f3ec523040e7b7cd356e3da57e0d P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:20.756892 20697 raft_consensus.cc:515] T 12c3f3ec523040e7b7cd356e3da57e0d P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:20.761757 19462 leader_election.cc:304] T 2c79a3aecab844698e70834999e1caac P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 78c2c46c7520492db35d51c244fb4f74, d870164263bb44b6b780a557daea3656; no voters: 371a36d219164974abdf848841528152
I20260130 02:58:20.762503 20929 raft_consensus.cc:2804] T 2c79a3aecab844698e70834999e1caac P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260130 02:58:20.762889 20929 raft_consensus.cc:493] T 2c79a3aecab844698e70834999e1caac P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260130 02:58:20.763197 20929 raft_consensus.cc:3060] T 2c79a3aecab844698e70834999e1caac P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:20.763967 19270 heartbeater.cc:344] Connected to a master server at 127.18.100.124:34629
I20260130 02:58:20.764639 19370 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "320dbe98a32a43b9899bbeed80b20f55" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74"
I20260130 02:58:20.765187 19370 raft_consensus.cc:3060] T 320dbe98a32a43b9899bbeed80b20f55 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:20.770192 20929 raft_consensus.cc:515] T 2c79a3aecab844698e70834999e1caac P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:20.772766 19352 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 920f52b975ab480fbf42656171222f09. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:20.772692 19221 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2c79a3aecab844698e70834999e1caac" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152"
I20260130 02:58:20.773164 19221 raft_consensus.cc:3060] T 2c79a3aecab844698e70834999e1caac P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:20.775653 19355 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 155ff0c35fc44cb292a28407e6ed937d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:20.779024 19221 raft_consensus.cc:2468] T 2c79a3aecab844698e70834999e1caac P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d870164263bb44b6b780a557daea3656 in term 1.
I20260130 02:58:20.780090 20929 leader_election.cc:290] T 2c79a3aecab844698e70834999e1caac P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 election: Requested vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767)
I20260130 02:58:20.780045 19463 leader_election.cc:304] T 2c79a3aecab844698e70834999e1caac P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 371a36d219164974abdf848841528152, d870164263bb44b6b780a557daea3656; no voters:
I20260130 02:58:20.782377 20956 consensus_queue.cc:1048] T af0f90071a334016aa17fae2e3a97fc8 P d870164263bb44b6b780a557daea3656 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260130 02:58:20.783612 20956 raft_consensus.cc:2804] T 2c79a3aecab844698e70834999e1caac P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Leader election won for term 1
I20260130 02:58:20.783995 20956 raft_consensus.cc:697] T 2c79a3aecab844698e70834999e1caac P d870164263bb44b6b780a557daea3656 [term 1 LEADER]: Becoming Leader. State: Replica: d870164263bb44b6b780a557daea3656, State: Running, Role: LEADER
I20260130 02:58:20.784703 20956 consensus_queue.cc:237] T 2c79a3aecab844698e70834999e1caac P d870164263bb44b6b780a557daea3656 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:20.801580 19353 tablet_service.cc:1505] Processing CreateTablet for tablet 6f5a90b75a0f4851b5db672c41253f7b (DEFAULT_TABLE table=default.Table_6f53c17cfd144031b2d4b112fb2d88eb [id=443cbad5510d463fa8ca878b8628dbda]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:20.802874 19353 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6f5a90b75a0f4851b5db672c41253f7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:20.806919 19357 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 66a7487a7ab343a4abca5312afe78862. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:20.814257 18860 catalog_manager.cc:4985] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767): tablet 04f4ac27800c4ac3a79ae12d35479757 (table default.Table_ddfb5df90fdb40a087ac434d088bbf0e [id=1e72b7dd669d4539b61884ba63ff7183]) successfully deleted
W20260130 02:58:20.814958 18860 catalog_manager.cc:4965] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767): delete failed for tablet 1129f72158ec4bae89da3c94cdb70839 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1129f72158ec4bae89da3c94cdb70839 already in progress: creating tablet
I20260130 02:58:20.829911 19375 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "13f0ace4d9de43abab08eca32cdb30c1" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74" is_pre_election: true
I20260130 02:58:20.830765 19375 raft_consensus.cc:2393] T 13f0ace4d9de43abab08eca32cdb30c1 P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d870164263bb44b6b780a557daea3656 in current term 1: Already voted for candidate 371a36d219164974abdf848841528152 in this term.
I20260130 02:58:20.831290 19840 tablet_bootstrap.cc:654] T 00ddbfe6218c44d788fbf65245a31dc6 P 78c2c46c7520492db35d51c244fb4f74: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:20.825462 19807 tablet_bootstrap.cc:492] T f7607df5efd543438fb2dff3667cc8c8 P 371a36d219164974abdf848841528152: No bootstrap required, opened a new log
I20260130 02:58:20.839849 19807 ts_tablet_manager.cc:1403] T f7607df5efd543438fb2dff3667cc8c8 P 371a36d219164974abdf848841528152: Time spent bootstrapping tablet: real 0.148s user 0.006s sys 0.004s
I20260130 02:58:20.850210 19625 tablet_bootstrap.cc:492] T a3258852aed24ed582a79f77860e6f0d P d870164263bb44b6b780a557daea3656: No bootstrap required, opened a new log
I20260130 02:58:20.850646 19625 ts_tablet_manager.cc:1403] T a3258852aed24ed582a79f77860e6f0d P d870164263bb44b6b780a557daea3656: Time spent bootstrapping tablet: real 0.158s user 0.020s sys 0.031s
I20260130 02:58:20.851778 20697 leader_election.cc:290] T 12c3f3ec523040e7b7cd356e3da57e0d P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d870164263bb44b6b780a557daea3656 (127.18.100.67:39571), 371a36d219164974abdf848841528152 (127.18.100.65:42021)
I20260130 02:58:20.852833 19625 raft_consensus.cc:359] T a3258852aed24ed582a79f77860e6f0d P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:20.853374 19807 raft_consensus.cc:359] T f7607df5efd543438fb2dff3667cc8c8 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:20.859236 19807 raft_consensus.cc:385] T f7607df5efd543438fb2dff3667cc8c8 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:20.859476 19807 raft_consensus.cc:740] T f7607df5efd543438fb2dff3667cc8c8 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 371a36d219164974abdf848841528152, State: Initialized, Role: FOLLOWER
I20260130 02:58:20.860080 19807 consensus_queue.cc:260] T f7607df5efd543438fb2dff3667cc8c8 P 371a36d219164974abdf848841528152 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:20.861833 19807 ts_tablet_manager.cc:1434] T f7607df5efd543438fb2dff3667cc8c8 P 371a36d219164974abdf848841528152: Time spent starting tablet: real 0.022s user 0.005s sys 0.000s
I20260130 02:58:20.853448 19625 raft_consensus.cc:385] T a3258852aed24ed582a79f77860e6f0d P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:20.863076 19625 raft_consensus.cc:740] T a3258852aed24ed582a79f77860e6f0d P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d870164263bb44b6b780a557daea3656, State: Initialized, Role: FOLLOWER
I20260130 02:58:20.863844 19625 consensus_queue.cc:260] T a3258852aed24ed582a79f77860e6f0d P d870164263bb44b6b780a557daea3656 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:20.865298 19625 ts_tablet_manager.cc:1434] T a3258852aed24ed582a79f77860e6f0d P d870164263bb44b6b780a557daea3656: Time spent starting tablet: real 0.014s user 0.003s sys 0.000s
I20260130 02:58:20.866080 19625 tablet_bootstrap.cc:492] T aae68390dbbe4dc58ce3d3bd01725ae6 P d870164263bb44b6b780a557daea3656: Bootstrap starting.
I20260130 02:58:20.871650 19625 tablet_bootstrap.cc:654] T aae68390dbbe4dc58ce3d3bd01725ae6 P d870164263bb44b6b780a557daea3656: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:20.883565 19376 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2c79a3aecab844698e70834999e1caac" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74"
I20260130 02:58:20.884191 19376 raft_consensus.cc:3060] T 2c79a3aecab844698e70834999e1caac P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:20.886672 19625 tablet_bootstrap.cc:492] T aae68390dbbe4dc58ce3d3bd01725ae6 P d870164263bb44b6b780a557daea3656: No bootstrap required, opened a new log
I20260130 02:58:20.887035 19625 ts_tablet_manager.cc:1403] T aae68390dbbe4dc58ce3d3bd01725ae6 P d870164263bb44b6b780a557daea3656: Time spent bootstrapping tablet: real 0.021s user 0.008s sys 0.001s
I20260130 02:58:20.889263 19625 raft_consensus.cc:359] T aae68390dbbe4dc58ce3d3bd01725ae6 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } }
I20260130 02:58:20.889848 19625 raft_consensus.cc:385] T aae68390dbbe4dc58ce3d3bd01725ae6 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:20.890092 19625 raft_consensus.cc:740] T aae68390dbbe4dc58ce3d3bd01725ae6 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d870164263bb44b6b780a557daea3656, State: Initialized, Role: FOLLOWER
I20260130 02:58:20.890648 19625 consensus_queue.cc:260] T aae68390dbbe4dc58ce3d3bd01725ae6 P d870164263bb44b6b780a557daea3656 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } }
I20260130 02:58:20.892290 19376 raft_consensus.cc:2468] T 2c79a3aecab844698e70834999e1caac P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d870164263bb44b6b780a557daea3656 in term 1.
I20260130 02:58:20.904001 19625 ts_tablet_manager.cc:1434] T aae68390dbbe4dc58ce3d3bd01725ae6 P d870164263bb44b6b780a557daea3656: Time spent starting tablet: real 0.017s user 0.005s sys 0.000s
I20260130 02:58:20.905022 19625 tablet_bootstrap.cc:492] T 49efcfced0b74c48962393149ac2a500 P d870164263bb44b6b780a557daea3656: Bootstrap starting.
I20260130 02:58:20.909996 19625 tablet_bootstrap.cc:654] T 49efcfced0b74c48962393149ac2a500 P d870164263bb44b6b780a557daea3656: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:20.915423 19462 leader_election.cc:304] T 13f0ace4d9de43abab08eca32cdb30c1 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d870164263bb44b6b780a557daea3656; no voters: 371a36d219164974abdf848841528152, 78c2c46c7520492db35d51c244fb4f74
I20260130 02:58:20.916213 20998 raft_consensus.cc:3060] T 13f0ace4d9de43abab08eca32cdb30c1 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:20.923539 20998 raft_consensus.cc:2749] T 13f0ace4d9de43abab08eca32cdb30c1 P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260130 02:58:20.924042 20909 lock_manager.cc:585] Waited 2 seconds to obtain row lock on key '<redacted>' (key hash 16383881659362995755) tablet 00000000000000000000000000000000 cur holder 0x7b5800064e00
I20260130 02:58:20.925590 19625 tablet_bootstrap.cc:492] T 49efcfced0b74c48962393149ac2a500 P d870164263bb44b6b780a557daea3656: No bootstrap required, opened a new log
I20260130 02:58:20.925997 19625 ts_tablet_manager.cc:1403] T 49efcfced0b74c48962393149ac2a500 P d870164263bb44b6b780a557daea3656: Time spent bootstrapping tablet: real 0.021s user 0.008s sys 0.000s
I20260130 02:58:20.928227 19625 raft_consensus.cc:359] T 49efcfced0b74c48962393149ac2a500 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } }
I20260130 02:58:20.928505 20920 consensus_queue.cc:1048] T 838e49e5e8874db393bdc15a1c92f87e P 371a36d219164974abdf848841528152 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260130 02:58:20.929033 19625 raft_consensus.cc:385] T 49efcfced0b74c48962393149ac2a500 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:20.929334 19625 raft_consensus.cc:740] T 49efcfced0b74c48962393149ac2a500 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d870164263bb44b6b780a557daea3656, State: Initialized, Role: FOLLOWER
I20260130 02:58:20.929862 19625 consensus_queue.cc:260] T 49efcfced0b74c48962393149ac2a500 P d870164263bb44b6b780a557daea3656 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } }
I20260130 02:58:20.931588 19625 ts_tablet_manager.cc:1434] T 49efcfced0b74c48962393149ac2a500 P d870164263bb44b6b780a557daea3656: Time spent starting tablet: real 0.005s user 0.003s sys 0.003s
I20260130 02:58:20.932353 19625 tablet_bootstrap.cc:492] T d480bf18140245e3870a62582d5a8318 P d870164263bb44b6b780a557daea3656: Bootstrap starting.
I20260130 02:58:20.937245 19625 tablet_bootstrap.cc:654] T d480bf18140245e3870a62582d5a8318 P d870164263bb44b6b780a557daea3656: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:20.938560 21004 raft_consensus.cc:493] T 133e9bfe175f44aab1a53114abe67e4d P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:20.939073 21004 raft_consensus.cc:515] T 133e9bfe175f44aab1a53114abe67e4d P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:20.940990 21004 leader_election.cc:290] T 133e9bfe175f44aab1a53114abe67e4d P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767)
I20260130 02:58:20.942667 19221 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "133e9bfe175f44aab1a53114abe67e4d" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152" is_pre_election: true
W20260130 02:58:20.943976 19463 leader_election.cc:343] T 133e9bfe175f44aab1a53114abe67e4d P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:20.981793 19625 tablet_bootstrap.cc:492] T d480bf18140245e3870a62582d5a8318 P d870164263bb44b6b780a557daea3656: No bootstrap required, opened a new log
I20260130 02:58:20.982349 19625 ts_tablet_manager.cc:1403] T d480bf18140245e3870a62582d5a8318 P d870164263bb44b6b780a557daea3656: Time spent bootstrapping tablet: real 0.050s user 0.010s sys 0.015s
W20260130 02:58:20.982992 18860 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet 25fd48e693424537b91ba7c3fcc96878 (table default.Table_ebc8b8379c274c18b0a06a919dbe0ee0 [id=28156e0843674fce88f5914a3e4195ae], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 25fd48e693424537b91ba7c3fcc96878 consensus state has no leader
I20260130 02:58:20.986016 19625 raft_consensus.cc:359] T d480bf18140245e3870a62582d5a8318 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } }
I20260130 02:58:20.986653 19625 raft_consensus.cc:385] T d480bf18140245e3870a62582d5a8318 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:20.986896 19625 raft_consensus.cc:740] T d480bf18140245e3870a62582d5a8318 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d870164263bb44b6b780a557daea3656, State: Initialized, Role: FOLLOWER
I20260130 02:58:20.987651 19625 consensus_queue.cc:260] T d480bf18140245e3870a62582d5a8318 P d870164263bb44b6b780a557daea3656 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } }
I20260130 02:58:20.990136 19625 ts_tablet_manager.cc:1434] T d480bf18140245e3870a62582d5a8318 P d870164263bb44b6b780a557daea3656: Time spent starting tablet: real 0.007s user 0.004s sys 0.000s
I20260130 02:58:20.997447 19625 tablet_bootstrap.cc:492] T 657e9b0d8c05413f904129dce745710b P d870164263bb44b6b780a557daea3656: Bootstrap starting.
I20260130 02:58:20.999492 18869 catalog_manager.cc:5746] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571) does not have the latest schema for tablet b1e8b78eff0d4a0b92059adb2de4a3ce (table default.Table_d0b4516d24384e5386c1debaf2543adb [id=f1342f9497b74e7a9ecad1de8082578d]). Expected version 1 got 0
W20260130 02:58:21.001811 19462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.100.66:40767, user_credentials={real_user=slave}} blocked reactor thread for 86455.8us
I20260130 02:58:21.009819 19625 tablet_bootstrap.cc:654] T 657e9b0d8c05413f904129dce745710b P d870164263bb44b6b780a557daea3656: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:21.010107 20920 raft_consensus.cc:493] T 12c3f3ec523040e7b7cd356e3da57e0d P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:20.971372 19202 tablet_service.cc:1505] Processing CreateTablet for tablet 6f5a90b75a0f4851b5db672c41253f7b (DEFAULT_TABLE table=default.Table_6f53c17cfd144031b2d4b112fb2d88eb [id=443cbad5510d463fa8ca878b8628dbda]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:21.010550 20920 raft_consensus.cc:515] T 12c3f3ec523040e7b7cd356e3da57e0d P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:21.012318 20920 leader_election.cc:290] T 12c3f3ec523040e7b7cd356e3da57e0d P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d870164263bb44b6b780a557daea3656 (127.18.100.67:39571), 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767)
I20260130 02:58:21.016204 19520 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "12c3f3ec523040e7b7cd356e3da57e0d" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
I20260130 02:58:21.016991 19520 raft_consensus.cc:2468] T 12c3f3ec523040e7b7cd356e3da57e0d P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 371a36d219164974abdf848841528152 in term 0.
I20260130 02:58:21.018262 19159 leader_election.cc:304] T 12c3f3ec523040e7b7cd356e3da57e0d P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 371a36d219164974abdf848841528152, d870164263bb44b6b780a557daea3656; no voters:
I20260130 02:58:21.019156 20920 raft_consensus.cc:2804] T 12c3f3ec523040e7b7cd356e3da57e0d P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260130 02:58:21.019475 20920 raft_consensus.cc:493] T 12c3f3ec523040e7b7cd356e3da57e0d P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260130 02:58:21.019817 20920 raft_consensus.cc:3060] T 12c3f3ec523040e7b7cd356e3da57e0d P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:21.024205 19625 tablet_bootstrap.cc:492] T 657e9b0d8c05413f904129dce745710b P d870164263bb44b6b780a557daea3656: No bootstrap required, opened a new log
I20260130 02:58:21.024621 19625 ts_tablet_manager.cc:1403] T 657e9b0d8c05413f904129dce745710b P d870164263bb44b6b780a557daea3656: Time spent bootstrapping tablet: real 0.027s user 0.010s sys 0.002s
I20260130 02:58:21.027060 19625 raft_consensus.cc:359] T 657e9b0d8c05413f904129dce745710b P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:21.027653 19625 raft_consensus.cc:385] T 657e9b0d8c05413f904129dce745710b P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:21.027886 19625 raft_consensus.cc:740] T 657e9b0d8c05413f904129dce745710b P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d870164263bb44b6b780a557daea3656, State: Initialized, Role: FOLLOWER
I20260130 02:58:21.028564 19625 consensus_queue.cc:260] T 657e9b0d8c05413f904129dce745710b P d870164263bb44b6b780a557daea3656 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:21.030839 19625 ts_tablet_manager.cc:1434] T 657e9b0d8c05413f904129dce745710b P d870164263bb44b6b780a557daea3656: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260130 02:58:21.031649 19625 tablet_bootstrap.cc:492] T dfbbe9045e7c4aa5b520218687e7e4c1 P d870164263bb44b6b780a557daea3656: Bootstrap starting.
I20260130 02:58:21.038343 19625 tablet_bootstrap.cc:654] T dfbbe9045e7c4aa5b520218687e7e4c1 P d870164263bb44b6b780a557daea3656: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:21.056197 19625 tablet_bootstrap.cc:492] T dfbbe9045e7c4aa5b520218687e7e4c1 P d870164263bb44b6b780a557daea3656: No bootstrap required, opened a new log
I20260130 02:58:21.056600 19625 ts_tablet_manager.cc:1403] T dfbbe9045e7c4aa5b520218687e7e4c1 P d870164263bb44b6b780a557daea3656: Time spent bootstrapping tablet: real 0.025s user 0.013s sys 0.005s
I20260130 02:58:20.952811 19370 raft_consensus.cc:2468] T 320dbe98a32a43b9899bbeed80b20f55 P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 371a36d219164974abdf848841528152 in term 1.
W20260130 02:58:20.985891 19314 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.518s user 0.021s sys 0.014s
W20260130 02:58:21.057336 19314 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.589s user 0.022s sys 0.014s
W20260130 02:58:21.057595 19314 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.100.65:42021, user_credentials={real_user=slave}} blocked reactor thread for 589660us
I20260130 02:58:21.008561 19378 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "133e9bfe175f44aab1a53114abe67e4d" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74" is_pre_election: true
I20260130 02:58:21.060141 19625 raft_consensus.cc:359] T dfbbe9045e7c4aa5b520218687e7e4c1 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:21.060775 19625 raft_consensus.cc:385] T dfbbe9045e7c4aa5b520218687e7e4c1 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:21.061022 19625 raft_consensus.cc:740] T dfbbe9045e7c4aa5b520218687e7e4c1 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d870164263bb44b6b780a557daea3656, State: Initialized, Role: FOLLOWER
I20260130 02:58:21.061570 19625 consensus_queue.cc:260] T dfbbe9045e7c4aa5b520218687e7e4c1 P d870164263bb44b6b780a557daea3656 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:21.063719 19625 ts_tablet_manager.cc:1434] T dfbbe9045e7c4aa5b520218687e7e4c1 P d870164263bb44b6b780a557daea3656: Time spent starting tablet: real 0.007s user 0.005s sys 0.002s
I20260130 02:58:21.064520 19625 tablet_bootstrap.cc:492] T e34d5da0ad864b2eaea0d575c982e447 P d870164263bb44b6b780a557daea3656: Bootstrap starting.
I20260130 02:58:21.068823 19225 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "12c3f3ec523040e7b7cd356e3da57e0d" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152" is_pre_election: true
I20260130 02:58:21.070621 19625 tablet_bootstrap.cc:654] T e34d5da0ad864b2eaea0d575c982e447 P d870164263bb44b6b780a557daea3656: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:21.073431 20985 consensus_queue.cc:1048] T 13f0ace4d9de43abab08eca32cdb30c1 P 371a36d219164974abdf848841528152 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260130 02:58:21.079804 20920 raft_consensus.cc:515] T 12c3f3ec523040e7b7cd356e3da57e0d P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:21.081034 19225 raft_consensus.cc:2393] T 12c3f3ec523040e7b7cd356e3da57e0d P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 78c2c46c7520492db35d51c244fb4f74 in current term 1: Already voted for candidate 371a36d219164974abdf848841528152 in this term.
I20260130 02:58:21.081207 19625 tablet_bootstrap.cc:492] T e34d5da0ad864b2eaea0d575c982e447 P d870164263bb44b6b780a557daea3656: No bootstrap required, opened a new log
I20260130 02:58:21.081601 19625 ts_tablet_manager.cc:1403] T e34d5da0ad864b2eaea0d575c982e447 P d870164263bb44b6b780a557daea3656: Time spent bootstrapping tablet: real 0.017s user 0.006s sys 0.008s
I20260130 02:58:21.081899 20920 leader_election.cc:290] T 12c3f3ec523040e7b7cd356e3da57e0d P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 election: Requested vote from peers d870164263bb44b6b780a557daea3656 (127.18.100.67:39571), 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767)
I20260130 02:58:21.084568 19522 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "12c3f3ec523040e7b7cd356e3da57e0d" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656"
I20260130 02:58:21.085151 19522 raft_consensus.cc:3060] T 12c3f3ec523040e7b7cd356e3da57e0d P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:21.091663 19522 raft_consensus.cc:2468] T 12c3f3ec523040e7b7cd356e3da57e0d P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 371a36d219164974abdf848841528152 in term 1.
I20260130 02:58:21.083964 19625 raft_consensus.cc:359] T e34d5da0ad864b2eaea0d575c982e447 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:21.093076 19625 raft_consensus.cc:385] T e34d5da0ad864b2eaea0d575c982e447 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:21.093379 19625 raft_consensus.cc:740] T e34d5da0ad864b2eaea0d575c982e447 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d870164263bb44b6b780a557daea3656, State: Initialized, Role: FOLLOWER
I20260130 02:58:21.094007 19625 consensus_queue.cc:260] T e34d5da0ad864b2eaea0d575c982e447 P d870164263bb44b6b780a557daea3656 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:21.096099 19625 ts_tablet_manager.cc:1434] T e34d5da0ad864b2eaea0d575c982e447 P d870164263bb44b6b780a557daea3656: Time spent starting tablet: real 0.014s user 0.002s sys 0.003s
I20260130 02:58:21.097288 19625 tablet_bootstrap.cc:492] T 2b0edaad6877460cb7982efc62b00b66 P d870164263bb44b6b780a557daea3656: Bootstrap starting.
I20260130 02:58:21.100998 19378 raft_consensus.cc:2468] T 133e9bfe175f44aab1a53114abe67e4d P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d870164263bb44b6b780a557daea3656 in term 0.
I20260130 02:58:21.102910 19625 tablet_bootstrap.cc:654] T 2b0edaad6877460cb7982efc62b00b66 P d870164263bb44b6b780a557daea3656: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:21.106459 20942 raft_consensus.cc:493] T 2b8034a92b2043e5b57cf02f1ee6296d P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:21.106964 20942 raft_consensus.cc:515] T 2b8034a92b2043e5b57cf02f1ee6296d P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:21.106477 20984 raft_consensus.cc:493] T 39bcc41f10ae4241b370529364454586 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:21.107918 20984 raft_consensus.cc:515] T 39bcc41f10ae4241b370529364454586 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:21.108785 20942 leader_election.cc:290] T 2b8034a92b2043e5b57cf02f1ee6296d P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), d870164263bb44b6b780a557daea3656 (127.18.100.67:39571)
I20260130 02:58:21.109754 20984 leader_election.cc:290] T 39bcc41f10ae4241b370529364454586 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), d870164263bb44b6b780a557daea3656 (127.18.100.67:39571)
I20260130 02:58:21.113382 19625 tablet_bootstrap.cc:492] T 2b0edaad6877460cb7982efc62b00b66 P d870164263bb44b6b780a557daea3656: No bootstrap required, opened a new log
I20260130 02:58:21.113757 19625 ts_tablet_manager.cc:1403] T 2b0edaad6877460cb7982efc62b00b66 P d870164263bb44b6b780a557daea3656: Time spent bootstrapping tablet: real 0.017s user 0.007s sys 0.007s
I20260130 02:58:21.115897 19625 raft_consensus.cc:359] T 2b0edaad6877460cb7982efc62b00b66 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:21.116494 19625 raft_consensus.cc:385] T 2b0edaad6877460cb7982efc62b00b66 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:21.116729 19625 raft_consensus.cc:740] T 2b0edaad6877460cb7982efc62b00b66 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d870164263bb44b6b780a557daea3656, State: Initialized, Role: FOLLOWER
I20260130 02:58:21.117424 19625 consensus_queue.cc:260] T 2b0edaad6877460cb7982efc62b00b66 P d870164263bb44b6b780a557daea3656 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:21.119419 19625 ts_tablet_manager.cc:1434] T 2b0edaad6877460cb7982efc62b00b66 P d870164263bb44b6b780a557daea3656: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260130 02:58:21.120159 19625 tablet_bootstrap.cc:492] T eb7a755ab62b4552bb8588264c5f7b75 P d870164263bb44b6b780a557daea3656: Bootstrap starting.
I20260130 02:58:21.125610 19625 tablet_bootstrap.cc:654] T eb7a755ab62b4552bb8588264c5f7b75 P d870164263bb44b6b780a557daea3656: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:21.134721 19522 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "12c3f3ec523040e7b7cd356e3da57e0d" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
I20260130 02:58:21.135507 19522 raft_consensus.cc:2393] T 12c3f3ec523040e7b7cd356e3da57e0d P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 78c2c46c7520492db35d51c244fb4f74 in current term 1: Already voted for candidate 371a36d219164974abdf848841528152 in this term.
W20260130 02:58:21.142697 19462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.100.66:40767, user_credentials={real_user=slave}} blocked reactor thread for 73711.4us
I20260130 02:58:21.156600 19625 tablet_bootstrap.cc:492] T eb7a755ab62b4552bb8588264c5f7b75 P d870164263bb44b6b780a557daea3656: No bootstrap required, opened a new log
I20260130 02:58:21.157014 19625 ts_tablet_manager.cc:1403] T eb7a755ab62b4552bb8588264c5f7b75 P d870164263bb44b6b780a557daea3656: Time spent bootstrapping tablet: real 0.037s user 0.010s sys 0.004s
I20260130 02:58:21.159247 19625 raft_consensus.cc:359] T eb7a755ab62b4552bb8588264c5f7b75 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:21.159888 19625 raft_consensus.cc:385] T eb7a755ab62b4552bb8588264c5f7b75 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:21.160131 19625 raft_consensus.cc:740] T eb7a755ab62b4552bb8588264c5f7b75 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d870164263bb44b6b780a557daea3656, State: Initialized, Role: FOLLOWER
I20260130 02:58:21.160895 19625 consensus_queue.cc:260] T eb7a755ab62b4552bb8588264c5f7b75 P d870164263bb44b6b780a557daea3656 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:21.163244 19625 ts_tablet_manager.cc:1434] T eb7a755ab62b4552bb8588264c5f7b75 P d870164263bb44b6b780a557daea3656: Time spent starting tablet: real 0.006s user 0.006s sys 0.001s
I20260130 02:58:21.164011 19625 tablet_bootstrap.cc:492] T ab6afa40408a4f458495abf20290e55b P d870164263bb44b6b780a557daea3656: Bootstrap starting.
I20260130 02:58:21.182047 20897 raft_consensus.cc:2243] T 8e57ae31ab3941f789ac69627384e3b0 P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260130 02:58:21.182639 20897 raft_consensus.cc:2272] T 8e57ae31ab3941f789ac69627384e3b0 P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260130 02:58:21.147513 19159 leader_election.cc:304] T 12c3f3ec523040e7b7cd356e3da57e0d P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 371a36d219164974abdf848841528152, d870164263bb44b6b780a557daea3656; no voters:
I20260130 02:58:21.184044 19807 tablet_bootstrap.cc:492] T a3258852aed24ed582a79f77860e6f0d P 371a36d219164974abdf848841528152: Bootstrap starting.
I20260130 02:58:21.184713 20897 ts_tablet_manager.cc:1916] T 8e57ae31ab3941f789ac69627384e3b0 P 371a36d219164974abdf848841528152: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260130 02:58:21.186353 20985 raft_consensus.cc:2804] T 12c3f3ec523040e7b7cd356e3da57e0d P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Leader election won for term 1
I20260130 02:58:21.187034 20985 raft_consensus.cc:697] T 12c3f3ec523040e7b7cd356e3da57e0d P 371a36d219164974abdf848841528152 [term 1 LEADER]: Becoming Leader. State: Replica: 371a36d219164974abdf848841528152, State: Running, Role: LEADER
I20260130 02:58:21.189746 19807 tablet_bootstrap.cc:654] T a3258852aed24ed582a79f77860e6f0d P 371a36d219164974abdf848841528152: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:21.196873 19202 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6f5a90b75a0f4851b5db672c41253f7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:21.202925 19462 leader_election.cc:304] T 133e9bfe175f44aab1a53114abe67e4d P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 78c2c46c7520492db35d51c244fb4f74, d870164263bb44b6b780a557daea3656; no voters: 371a36d219164974abdf848841528152
I20260130 02:58:21.204217 21000 raft_consensus.cc:2804] T 133e9bfe175f44aab1a53114abe67e4d P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260130 02:58:21.204836 21000 raft_consensus.cc:493] T 133e9bfe175f44aab1a53114abe67e4d P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260130 02:58:21.205140 21000 raft_consensus.cc:3060] T 133e9bfe175f44aab1a53114abe67e4d P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:21.205724 20985 consensus_queue.cc:237] T 12c3f3ec523040e7b7cd356e3da57e0d P 371a36d219164974abdf848841528152 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:21.223274 20897 ts_tablet_manager.cc:1929] T 8e57ae31ab3941f789ac69627384e3b0 P 371a36d219164974abdf848841528152: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260130 02:58:21.231989 20897 log.cc:1199] T 8e57ae31ab3941f789ac69627384e3b0 P 371a36d219164974abdf848841528152: Deleting WAL directory at /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-0/wal/wals/8e57ae31ab3941f789ac69627384e3b0
I20260130 02:58:21.233398 20897 ts_tablet_manager.cc:1950] T 8e57ae31ab3941f789ac69627384e3b0 P 371a36d219164974abdf848841528152: Deleting consensus metadata
I20260130 02:58:21.235169 20897 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42870 (request call id 338) took 1249 ms. Trace:
I20260130 02:58:21.236608 20897 rpcz_store.cc:276] 0130 02:58:19.985412 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:58:19.985498 (+ 86us) service_pool.cc:225] Handling call
0130 02:58:20.713610 (+728112us) ts_tablet_manager.cc:1151] Deleting tablet 8e57ae31ab3941f789ac69627384e3b0
0130 02:58:20.713659 (+ 49us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0130 02:58:21.199307 (+485648us) tablet_metadata.cc:641] Metadata flushed
0130 02:58:21.222990 (+ 23683us) tablet_metadata.cc:641] Metadata flushed
0130 02:58:21.235105 (+ 12115us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":467293,"tablet-delete.queue_time_us":695124}
I20260130 02:58:21.142530 18869 catalog_manager.cc:5746] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571) does not have the latest schema for tablet 127a84346b9f473d9e7f739b1e65feb8 (table default.Table_805e07e3546e447d8c8b8029477013cc [id=1515db389f284e2f9ed1a3cd0061d0f1]). Expected version 7 got 6
I20260130 02:58:21.238976 20897 tablet_replica.cc:333] T a8a5038e9def44eb8b3b1e4aeae25652 P 371a36d219164974abdf848841528152: stopping tablet replica
I20260130 02:58:21.238988 18861 catalog_manager.cc:4985] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021): tablet 8e57ae31ab3941f789ac69627384e3b0 (table default.Table_1ae68ae3f0404843aca3c72f02678ac6 [id=8a9182f49e18489fafec8dca190c1b2f]) successfully deleted
I20260130 02:58:21.240677 20897 raft_consensus.cc:2243] T a8a5038e9def44eb8b3b1e4aeae25652 P 371a36d219164974abdf848841528152 [term 1 LEADER]: Raft consensus shutting down.
I20260130 02:58:21.241539 20897 raft_consensus.cc:2272] T a8a5038e9def44eb8b3b1e4aeae25652 P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260130 02:58:21.242002 19350 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:60110 (request call id 325) took 1104 ms. Trace:
I20260130 02:58:21.242442 19350 rpcz_store.cc:276] 0130 02:58:20.137155 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:58:20.142365 (+ 5210us) service_pool.cc:225] Handling call
0130 02:58:20.694680 (+552315us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0130 02:58:20.694799 (+ 119us) ts_tablet_manager.cc:682] Creating new metadata...
0130 02:58:20.896297 (+201498us) tablet_metadata.cc:641] Metadata flushed
0130 02:58:21.241393 (+345096us) tablet_replica.cc:206] Creating consensus instance
0130 02:58:21.241927 (+ 534us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260130 02:58:21.243577 20897 ts_tablet_manager.cc:1916] T a8a5038e9def44eb8b3b1e4aeae25652 P 371a36d219164974abdf848841528152: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260130 02:58:21.249197 18869 catalog_manager.cc:5746] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571) does not have the latest schema for tablet 9fe0448bcc4a4346b3c67fc3f626ed3c (table default.Table_5c1c447e85524ac5a61ffd8a48509d36 [id=70aaf21ad9bd4d959f46939c6b4a4880]). Expected version 1 got 0
I20260130 02:58:21.252713 19527 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "39bcc41f10ae4241b370529364454586" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
I20260130 02:58:21.254482 19520 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2b8034a92b2043e5b57cf02f1ee6296d" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
I20260130 02:58:21.257594 18869 catalog_manager.cc:5746] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571) does not have the latest schema for tablet 25fd48e693424537b91ba7c3fcc96878 (table default.Table_ebc8b8379c274c18b0a06a919dbe0ee0 [id=28156e0843674fce88f5914a3e4195ae]). Expected version 1 got 0
I20260130 02:58:21.269057 19356 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:60110 (request call id 320) took 1178 ms. Trace:
I20260130 02:58:21.269423 19356 rpcz_store.cc:276] 0130 02:58:20.090404 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:58:20.090541 (+ 137us) service_pool.cc:225] Handling call
0130 02:58:20.662142 (+571601us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0130 02:58:20.662315 (+ 173us) ts_tablet_manager.cc:682] Creating new metadata...
0130 02:58:21.115224 (+452909us) tablet_metadata.cc:641] Metadata flushed
0130 02:58:21.253378 (+138154us) tablet_replica.cc:206] Creating consensus instance
0130 02:58:21.268997 (+ 15619us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260130 02:58:21.287250 19360 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:60110 (request call id 321) took 1173 ms. Trace:
I20260130 02:58:21.287719 19360 rpcz_store.cc:276] 0130 02:58:20.113911 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:58:20.114049 (+ 138us) service_pool.cc:225] Handling call
0130 02:58:20.706684 (+592635us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0130 02:58:20.706793 (+ 109us) ts_tablet_manager.cc:682] Creating new metadata...
0130 02:58:20.894308 (+187515us) tablet_metadata.cc:641] Metadata flushed
0130 02:58:21.253255 (+358947us) tablet_replica.cc:206] Creating consensus instance
0130 02:58:21.287186 (+ 33931us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260130 02:58:21.258247 18869 catalog_manager.cc:5746] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571) does not have the latest schema for tablet 274530359b174ff79f046c2d4a6afa76 (table default.Table_c4f4904b6fac4b9880979cf0314e71c1 [id=c456c6418c93499a9d08f652720854ca]). Expected version 1 got 0
W20260130 02:58:21.301553 19314 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.100.65:42021, user_credentials={real_user=slave}} blocked reactor thread for 195473us
I20260130 02:58:21.307113 19807 tablet_bootstrap.cc:492] T a3258852aed24ed582a79f77860e6f0d P 371a36d219164974abdf848841528152: No bootstrap required, opened a new log
I20260130 02:58:21.307538 19807 ts_tablet_manager.cc:1403] T a3258852aed24ed582a79f77860e6f0d P 371a36d219164974abdf848841528152: Time spent bootstrapping tablet: real 0.124s user 0.010s sys 0.027s
I20260130 02:58:21.307116 18869 catalog_manager.cc:5654] T f9e5f01e69254d658c369f3e21cef523 P d870164263bb44b6b780a557daea3656 reported cstate change: term changed from 0 to 1, leader changed from <none> to d870164263bb44b6b780a557daea3656 (127.18.100.67). New cstate: current_term: 1 leader_uuid: "d870164263bb44b6b780a557daea3656" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } health_report { overall_health: UNKNOWN } } }
I20260130 02:58:21.308779 18869 catalog_manager.cc:5746] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571) does not have the latest schema for tablet 838e49e5e8874db393bdc15a1c92f87e (table default.Table_da3d581b15a94840898777f74611994e [id=b7e241ced53e4160ac6fa1f65ef3674e]). Expected version 1 got 0
I20260130 02:58:21.312855 19625 tablet_bootstrap.cc:654] T ab6afa40408a4f458495abf20290e55b P d870164263bb44b6b780a557daea3656: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:21.316625 20897 ts_tablet_manager.cc:1929] T a8a5038e9def44eb8b3b1e4aeae25652 P 371a36d219164974abdf848841528152: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260130 02:58:21.317011 20897 log.cc:1199] T a8a5038e9def44eb8b3b1e4aeae25652 P 371a36d219164974abdf848841528152: Deleting WAL directory at /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-0/wal/wals/a8a5038e9def44eb8b3b1e4aeae25652
I20260130 02:58:21.318086 20998 raft_consensus.cc:493] T 657e9b0d8c05413f904129dce745710b P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:21.318534 20998 raft_consensus.cc:515] T 657e9b0d8c05413f904129dce745710b P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:21.319746 19840 tablet_bootstrap.cc:492] T 00ddbfe6218c44d788fbf65245a31dc6 P 78c2c46c7520492db35d51c244fb4f74: No bootstrap required, opened a new log
I20260130 02:58:21.320185 19840 ts_tablet_manager.cc:1403] T 00ddbfe6218c44d788fbf65245a31dc6 P 78c2c46c7520492db35d51c244fb4f74: Time spent bootstrapping tablet: real 0.609s user 0.004s sys 0.012s
I20260130 02:58:21.320343 20998 leader_election.cc:290] T 657e9b0d8c05413f904129dce745710b P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767)
I20260130 02:58:21.317698 20897 ts_tablet_manager.cc:1950] T a8a5038e9def44eb8b3b1e4aeae25652 P 371a36d219164974abdf848841528152: Deleting consensus metadata
I20260130 02:58:21.321861 19225 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "657e9b0d8c05413f904129dce745710b" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152" is_pre_election: true
W20260130 02:58:21.323166 19463 leader_election.cc:343] T 657e9b0d8c05413f904129dce745710b P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:21.325922 21000 raft_consensus.cc:515] T 133e9bfe175f44aab1a53114abe67e4d P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:21.328761 19225 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "133e9bfe175f44aab1a53114abe67e4d" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152"
W20260130 02:58:21.329983 19463 leader_election.cc:343] T 133e9bfe175f44aab1a53114abe67e4d P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Illegal state: must be running to vote when last-logged opid is not known
W20260130 02:58:21.337047 19159 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.100.67:39571, user_credentials={real_user=slave}} blocked reactor thread for 60649.6us
I20260130 02:58:21.338303 19807 raft_consensus.cc:359] T a3258852aed24ed582a79f77860e6f0d P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:21.343490 19807 raft_consensus.cc:385] T a3258852aed24ed582a79f77860e6f0d P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:21.343775 19807 raft_consensus.cc:740] T a3258852aed24ed582a79f77860e6f0d P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 371a36d219164974abdf848841528152, State: Initialized, Role: FOLLOWER
I20260130 02:58:21.344360 19807 consensus_queue.cc:260] T a3258852aed24ed582a79f77860e6f0d P 371a36d219164974abdf848841528152 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:21.346493 19807 ts_tablet_manager.cc:1434] T a3258852aed24ed582a79f77860e6f0d P 371a36d219164974abdf848841528152: Time spent starting tablet: real 0.039s user 0.006s sys 0.004s
I20260130 02:58:21.355317 21000 leader_election.cc:290] T 133e9bfe175f44aab1a53114abe67e4d P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 election: Requested vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767)
I20260130 02:58:21.356472 19371 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "12c3f3ec523040e7b7cd356e3da57e0d" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74"
I20260130 02:58:21.357265 19371 raft_consensus.cc:3060] T 12c3f3ec523040e7b7cd356e3da57e0d P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Advancing to term 1
W20260130 02:58:21.360303 19313 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.100.67:39571, user_credentials={real_user=slave}} blocked reactor thread for 89145us
W20260130 02:58:21.363480 19313 leader_election.cc:343] T 39bcc41f10ae4241b370529364454586 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): Illegal state: must be running to vote when last-logged opid is not known
W20260130 02:58:21.364068 19313 leader_election.cc:343] T 2b8034a92b2043e5b57cf02f1ee6296d P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:21.367412 19351 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:60110 (request call id 324) took 1237 ms. Trace:
I20260130 02:58:21.367743 19807 tablet_bootstrap.cc:492] T dfbbe9045e7c4aa5b520218687e7e4c1 P 371a36d219164974abdf848841528152: Bootstrap starting.
I20260130 02:58:21.367869 19351 rpcz_store.cc:276] 0130 02:58:20.130243 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:58:20.130389 (+ 146us) service_pool.cc:225] Handling call
0130 02:58:20.690197 (+559808us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0130 02:58:20.690317 (+ 120us) ts_tablet_manager.cc:682] Creating new metadata...
0130 02:58:21.088422 (+398105us) tablet_metadata.cc:641] Metadata flushed
0130 02:58:21.271049 (+182627us) tablet_replica.cc:206] Creating consensus instance
0130 02:58:21.367238 (+ 96189us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260130 02:58:21.368829 20897 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42870 (request call id 339) took 1241 ms. Trace:
I20260130 02:58:21.369346 20897 rpcz_store.cc:276] 0130 02:58:20.127464 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:58:20.127575 (+ 111us) service_pool.cc:225] Handling call
0130 02:58:21.238826 (+1111251us) ts_tablet_manager.cc:1151] Deleting tablet a8a5038e9def44eb8b3b1e4aeae25652
0130 02:58:21.238865 (+ 39us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0130 02:58:21.275185 (+ 36320us) tablet_metadata.cc:641] Metadata flushed
0130 02:58:21.316390 (+ 41205us) tablet_metadata.cc:641] Metadata flushed
0130 02:58:21.368758 (+ 52368us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1102515}
I20260130 02:58:21.371968 19840 raft_consensus.cc:359] T 00ddbfe6218c44d788fbf65245a31dc6 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:21.372714 19840 raft_consensus.cc:385] T 00ddbfe6218c44d788fbf65245a31dc6 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:21.372980 19807 tablet_bootstrap.cc:654] T dfbbe9045e7c4aa5b520218687e7e4c1 P 371a36d219164974abdf848841528152: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:21.373328 18861 catalog_manager.cc:4985] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021): tablet a8a5038e9def44eb8b3b1e4aeae25652 (table default.Table_dee034aec0b24982b8dc2a3fa1b326f1 [id=66033e75b3814a8584311cf7e4a07ad3]) successfully deleted
I20260130 02:58:21.374094 19840 raft_consensus.cc:740] T 00ddbfe6218c44d788fbf65245a31dc6 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78c2c46c7520492db35d51c244fb4f74, State: Initialized, Role: FOLLOWER
I20260130 02:58:21.387710 20998 raft_consensus.cc:493] T e34d5da0ad864b2eaea0d575c982e447 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:21.388274 20998 raft_consensus.cc:515] T e34d5da0ad864b2eaea0d575c982e447 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:21.392135 19625 tablet_bootstrap.cc:492] T ab6afa40408a4f458495abf20290e55b P d870164263bb44b6b780a557daea3656: No bootstrap required, opened a new log
I20260130 02:58:21.394373 19807 tablet_bootstrap.cc:492] T dfbbe9045e7c4aa5b520218687e7e4c1 P 371a36d219164974abdf848841528152: No bootstrap required, opened a new log
I20260130 02:58:21.394865 19807 ts_tablet_manager.cc:1403] T dfbbe9045e7c4aa5b520218687e7e4c1 P 371a36d219164974abdf848841528152: Time spent bootstrapping tablet: real 0.027s user 0.002s sys 0.010s
I20260130 02:58:21.395346 19625 ts_tablet_manager.cc:1403] T ab6afa40408a4f458495abf20290e55b P d870164263bb44b6b780a557daea3656: Time spent bootstrapping tablet: real 0.231s user 0.005s sys 0.005s
I20260130 02:58:21.397681 19225 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e34d5da0ad864b2eaea0d575c982e447" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152" is_pre_election: true
I20260130 02:58:21.401504 19807 raft_consensus.cc:359] T dfbbe9045e7c4aa5b520218687e7e4c1 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:21.402164 19807 raft_consensus.cc:385] T dfbbe9045e7c4aa5b520218687e7e4c1 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:21.402416 19807 raft_consensus.cc:740] T dfbbe9045e7c4aa5b520218687e7e4c1 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 371a36d219164974abdf848841528152, State: Initialized, Role: FOLLOWER
I20260130 02:58:21.403412 19807 consensus_queue.cc:260] T dfbbe9045e7c4aa5b520218687e7e4c1 P 371a36d219164974abdf848841528152 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:21.410490 19807 ts_tablet_manager.cc:1434] T dfbbe9045e7c4aa5b520218687e7e4c1 P 371a36d219164974abdf848841528152: Time spent starting tablet: real 0.015s user 0.003s sys 0.002s
I20260130 02:58:21.411496 19807 tablet_bootstrap.cc:492] T 133e9bfe175f44aab1a53114abe67e4d P 371a36d219164974abdf848841528152: Bootstrap starting.
I20260130 02:58:21.416889 19807 tablet_bootstrap.cc:654] T 133e9bfe175f44aab1a53114abe67e4d P 371a36d219164974abdf848841528152: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:21.374760 19840 consensus_queue.cc:260] T 00ddbfe6218c44d788fbf65245a31dc6 P 78c2c46c7520492db35d51c244fb4f74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
W20260130 02:58:21.420063 19314 consensus_peers.cc:597] T 8e57ae31ab3941f789ac69627384e3b0 P 78c2c46c7520492db35d51c244fb4f74 -> Peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Couldn't send request to peer 371a36d219164974abdf848841528152. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: STOPPING. This is attempt 1: this message will repeat every 5th retry.
W20260130 02:58:21.420522 19314 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.100.65:42021, user_credentials={real_user=slave}} blocked reactor thread for 118234us
I20260130 02:58:21.424911 19372 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "12c3f3ec523040e7b7cd356e3da57e0d" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74" is_pre_election: true
W20260130 02:58:21.426911 18861 catalog_manager.cc:5058] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021): alter failed for tablet a8a5038e9def44eb8b3b1e4aeae25652 (table default.Table_dee034aec0b24982b8dc2a3fa1b326f1 [id=66033e75b3814a8584311cf7e4a07ad3]),no further retry: Not found: Tablet not found: a8a5038e9def44eb8b3b1e4aeae25652
I20260130 02:58:21.396113 20998 leader_election.cc:290] T e34d5da0ad864b2eaea0d575c982e447 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767)
W20260130 02:58:21.398957 19463 leader_election.cc:343] T e34d5da0ad864b2eaea0d575c982e447 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:21.402727 19573 heartbeater.cc:344] Connected to a master server at 127.18.100.124:34629
I20260130 02:58:21.433657 20998 raft_consensus.cc:493] T ca06f07bb98f45a7bf63aebc28348e08 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:21.434165 20998 raft_consensus.cc:515] T ca06f07bb98f45a7bf63aebc28348e08 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:21.435779 19358 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:60110 (request call id 322) took 1318 ms. Trace:
I20260130 02:58:21.436187 20998 leader_election.cc:290] T ca06f07bb98f45a7bf63aebc28348e08 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767)
I20260130 02:58:21.436371 20987 raft_consensus.cc:493] T f7607df5efd543438fb2dff3667cc8c8 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:21.436129 19358 rpcz_store.cc:276] 0130 02:58:20.117296 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:58:20.117421 (+ 125us) service_pool.cc:225] Handling call
0130 02:58:20.702684 (+585263us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0130 02:58:20.702809 (+ 125us) ts_tablet_manager.cc:682] Creating new metadata...
0130 02:58:21.090957 (+388148us) tablet_metadata.cc:641] Metadata flushed
0130 02:58:21.282458 (+191501us) tablet_replica.cc:206] Creating consensus instance
0130 02:58:21.435446 (+152988us) spinlock_profiling.cc:243] Waited 5.34 ms on lock 0x7b5800000098. stack: 00007f213f77c373 00007f213f77c270 00007f213f2f1574 0000000000508744 00000000005086d9 00007f213f5ee8b1 00007f213f770db1 00007f213f7713e2 00007f21485cb189 00007f214781e108 00007f21478e4683 00007f21478e041f 00007f21478e0bdd 00007f214788f4c9 00007f2145e51bf4 00007f2145e51b82
0130 02:58:21.435722 (+ 276us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":12288256}
I20260130 02:58:21.436803 20987 raft_consensus.cc:515] T f7607df5efd543438fb2dff3667cc8c8 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:21.438707 20987 leader_election.cc:290] T f7607df5efd543438fb2dff3667cc8c8 P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d870164263bb44b6b780a557daea3656 (127.18.100.67:39571), 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767)
I20260130 02:58:21.439996 20899 raft_consensus.cc:2243] T 8e57ae31ab3941f789ac69627384e3b0 P 78c2c46c7520492db35d51c244fb4f74 [term 1 LEADER]: Raft consensus shutting down.
I20260130 02:58:21.440562 19520 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f7607df5efd543438fb2dff3667cc8c8" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
I20260130 02:58:21.441159 19520 raft_consensus.cc:2468] T f7607df5efd543438fb2dff3667cc8c8 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 371a36d219164974abdf848841528152 in term 0.
I20260130 02:58:21.441197 19225 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ca06f07bb98f45a7bf63aebc28348e08" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152" is_pre_election: true
W20260130 02:58:21.442875 19463 leader_election.cc:343] T ca06f07bb98f45a7bf63aebc28348e08 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:21.425779 19371 raft_consensus.cc:2468] T 12c3f3ec523040e7b7cd356e3da57e0d P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 371a36d219164974abdf848841528152 in term 1.
I20260130 02:58:21.446764 19159 leader_election.cc:304] T f7607df5efd543438fb2dff3667cc8c8 P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 371a36d219164974abdf848841528152, d870164263bb44b6b780a557daea3656; no voters:
I20260130 02:58:21.447497 20987 raft_consensus.cc:2804] T f7607df5efd543438fb2dff3667cc8c8 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260130 02:58:21.447851 20987 raft_consensus.cc:493] T f7607df5efd543438fb2dff3667cc8c8 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260130 02:58:21.448148 20987 raft_consensus.cc:3060] T f7607df5efd543438fb2dff3667cc8c8 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:21.454447 19840 ts_tablet_manager.cc:1434] T 00ddbfe6218c44d788fbf65245a31dc6 P 78c2c46c7520492db35d51c244fb4f74: Time spent starting tablet: real 0.134s user 0.007s sys 0.001s
I20260130 02:58:21.457779 21027 raft_consensus.cc:493] T a3258852aed24ed582a79f77860e6f0d P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:21.458249 21027 raft_consensus.cc:515] T a3258852aed24ed582a79f77860e6f0d P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:21.460233 21027 leader_election.cc:290] T a3258852aed24ed582a79f77860e6f0d P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d870164263bb44b6b780a557daea3656 (127.18.100.67:39571), 371a36d219164974abdf848841528152 (127.18.100.65:42021)
I20260130 02:58:21.461555 21000 raft_consensus.cc:493] T 2b0edaad6877460cb7982efc62b00b66 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:21.461977 21000 raft_consensus.cc:515] T 2b0edaad6877460cb7982efc62b00b66 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:21.464833 20987 raft_consensus.cc:515] T f7607df5efd543438fb2dff3667cc8c8 P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:21.467841 19520 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f7607df5efd543438fb2dff3667cc8c8" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656"
I20260130 02:58:21.469331 19520 raft_consensus.cc:3060] T f7607df5efd543438fb2dff3667cc8c8 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:21.470958 20899 raft_consensus.cc:2272] T 8e57ae31ab3941f789ac69627384e3b0 P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260130 02:58:21.473142 20899 ts_tablet_manager.cc:1916] T 8e57ae31ab3941f789ac69627384e3b0 P 78c2c46c7520492db35d51c244fb4f74: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260130 02:58:21.481420 19225 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2b0edaad6877460cb7982efc62b00b66" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152" is_pre_election: true
W20260130 02:58:21.482757 19463 leader_election.cc:343] T 2b0edaad6877460cb7982efc62b00b66 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:21.485782 21000 leader_election.cc:290] T 2b0edaad6877460cb7982efc62b00b66 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767)
I20260130 02:58:21.497047 19840 tablet_bootstrap.cc:492] T 9896ff9364744857968e87e80e776dd2 P 78c2c46c7520492db35d51c244fb4f74: Bootstrap starting.
I20260130 02:58:21.500088 19520 raft_consensus.cc:2468] T f7607df5efd543438fb2dff3667cc8c8 P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 371a36d219164974abdf848841528152 in term 1.
I20260130 02:58:21.500900 20987 leader_election.cc:290] T f7607df5efd543438fb2dff3667cc8c8 P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 election: Requested vote from peers d870164263bb44b6b780a557daea3656 (127.18.100.67:39571), 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767)
I20260130 02:58:21.502436 19372 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f7607df5efd543438fb2dff3667cc8c8" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74" is_pre_election: true
I20260130 02:58:21.502988 19372 raft_consensus.cc:2468] T f7607df5efd543438fb2dff3667cc8c8 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 371a36d219164974abdf848841528152 in term 0.
I20260130 02:58:21.504922 19159 leader_election.cc:304] T f7607df5efd543438fb2dff3667cc8c8 P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 371a36d219164974abdf848841528152, d870164263bb44b6b780a557daea3656; no voters:
I20260130 02:58:21.505738 20987 raft_consensus.cc:2804] T f7607df5efd543438fb2dff3667cc8c8 P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Leader election won for term 1
I20260130 02:58:21.507483 21000 raft_consensus.cc:493] T a3258852aed24ed582a79f77860e6f0d P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260130 02:58:21.507699 19314 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.100.65:42021, user_credentials={real_user=slave}} blocked reactor thread for 53276.7us
I20260130 02:58:21.507978 21000 raft_consensus.cc:515] T a3258852aed24ed582a79f77860e6f0d P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:21.508328 19314 leader_election.cc:304] T 12c3f3ec523040e7b7cd356e3da57e0d P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78c2c46c7520492db35d51c244fb4f74; no voters: 371a36d219164974abdf848841528152, d870164263bb44b6b780a557daea3656
I20260130 02:58:21.509266 21026 raft_consensus.cc:2749] T 12c3f3ec523040e7b7cd356e3da57e0d P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260130 02:58:21.511476 19225 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a3258852aed24ed582a79f77860e6f0d" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152" is_pre_election: true
I20260130 02:58:21.511736 19222 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2b8034a92b2043e5b57cf02f1ee6296d" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152" is_pre_election: true
I20260130 02:58:21.512080 19225 raft_consensus.cc:2468] T a3258852aed24ed582a79f77860e6f0d P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d870164263bb44b6b780a557daea3656 in term 0.
I20260130 02:58:21.513110 19463 leader_election.cc:304] T a3258852aed24ed582a79f77860e6f0d P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 371a36d219164974abdf848841528152, d870164263bb44b6b780a557daea3656; no voters:
I20260130 02:58:21.513836 20998 raft_consensus.cc:2804] T a3258852aed24ed582a79f77860e6f0d P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260130 02:58:21.514184 20998 raft_consensus.cc:493] T a3258852aed24ed582a79f77860e6f0d P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260130 02:58:21.514487 20998 raft_consensus.cc:3060] T a3258852aed24ed582a79f77860e6f0d P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:21.514966 19371 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f7607df5efd543438fb2dff3667cc8c8" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74"
I20260130 02:58:21.515489 19371 raft_consensus.cc:3060] T f7607df5efd543438fb2dff3667cc8c8 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:21.518962 19225 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "39bcc41f10ae4241b370529364454586" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152" is_pre_election: true
I20260130 02:58:21.521591 19371 raft_consensus.cc:2468] T f7607df5efd543438fb2dff3667cc8c8 P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 371a36d219164974abdf848841528152 in term 1.
I20260130 02:58:21.522192 20998 raft_consensus.cc:515] T a3258852aed24ed582a79f77860e6f0d P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:21.524683 19222 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a3258852aed24ed582a79f77860e6f0d" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152"
I20260130 02:58:21.525169 19222 raft_consensus.cc:3060] T a3258852aed24ed582a79f77860e6f0d P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:21.526732 20998 leader_election.cc:290] T a3258852aed24ed582a79f77860e6f0d P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 election: Requested vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767)
I20260130 02:58:21.531248 20920 consensus_queue.cc:1048] T 13f0ace4d9de43abab08eca32cdb30c1 P 371a36d219164974abdf848841528152 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260130 02:58:21.532255 19222 raft_consensus.cc:2468] T a3258852aed24ed582a79f77860e6f0d P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d870164263bb44b6b780a557daea3656 in term 1.
I20260130 02:58:21.533046 19463 leader_election.cc:304] T a3258852aed24ed582a79f77860e6f0d P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 371a36d219164974abdf848841528152, d870164263bb44b6b780a557daea3656; no voters:
I20260130 02:58:21.534761 20998 raft_consensus.cc:2804] T a3258852aed24ed582a79f77860e6f0d P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Leader election won for term 1
I20260130 02:58:21.534765 19225 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a3258852aed24ed582a79f77860e6f0d" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152" is_pre_election: true
I20260130 02:58:21.431252 19625 raft_consensus.cc:359] T ab6afa40408a4f458495abf20290e55b P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:21.535162 19625 raft_consensus.cc:385] T ab6afa40408a4f458495abf20290e55b P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:21.535398 19225 raft_consensus.cc:2393] T a3258852aed24ed582a79f77860e6f0d P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 78c2c46c7520492db35d51c244fb4f74 in current term 1: Already voted for candidate d870164263bb44b6b780a557daea3656 in this term.
I20260130 02:58:21.535391 20998 raft_consensus.cc:697] T a3258852aed24ed582a79f77860e6f0d P d870164263bb44b6b780a557daea3656 [term 1 LEADER]: Becoming Leader. State: Replica: d870164263bb44b6b780a557daea3656, State: Running, Role: LEADER
I20260130 02:58:21.536499 20998 consensus_queue.cc:237] T a3258852aed24ed582a79f77860e6f0d P d870164263bb44b6b780a557daea3656 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:21.523504 20987 raft_consensus.cc:697] T f7607df5efd543438fb2dff3667cc8c8 P 371a36d219164974abdf848841528152 [term 1 LEADER]: Becoming Leader. State: Replica: 371a36d219164974abdf848841528152, State: Running, Role: LEADER
I20260130 02:58:21.535423 19625 raft_consensus.cc:740] T ab6afa40408a4f458495abf20290e55b P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d870164263bb44b6b780a557daea3656, State: Initialized, Role: FOLLOWER
I20260130 02:58:21.540256 20987 consensus_queue.cc:237] T f7607df5efd543438fb2dff3667cc8c8 P 371a36d219164974abdf848841528152 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:21.540685 19625 consensus_queue.cc:260] T ab6afa40408a4f458495abf20290e55b P d870164263bb44b6b780a557daea3656 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:21.542601 19625 ts_tablet_manager.cc:1434] T ab6afa40408a4f458495abf20290e55b P d870164263bb44b6b780a557daea3656: Time spent starting tablet: real 0.113s user 0.005s sys 0.000s
I20260130 02:58:21.514477 21000 leader_election.cc:290] T a3258852aed24ed582a79f77860e6f0d P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767)
I20260130 02:58:21.559963 19807 tablet_bootstrap.cc:492] T 133e9bfe175f44aab1a53114abe67e4d P 371a36d219164974abdf848841528152: No bootstrap required, opened a new log
I20260130 02:58:21.560379 19807 ts_tablet_manager.cc:1403] T 133e9bfe175f44aab1a53114abe67e4d P 371a36d219164974abdf848841528152: Time spent bootstrapping tablet: real 0.149s user 0.021s sys 0.038s
I20260130 02:58:21.562491 19807 raft_consensus.cc:359] T 133e9bfe175f44aab1a53114abe67e4d P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:21.563356 19807 raft_consensus.cc:385] T 133e9bfe175f44aab1a53114abe67e4d P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:21.563645 19625 tablet_bootstrap.cc:492] T 0855d88806914a36b4bf6b49036f0ad0 P d870164263bb44b6b780a557daea3656: Bootstrap starting.
I20260130 02:58:21.563674 19807 raft_consensus.cc:740] T 133e9bfe175f44aab1a53114abe67e4d P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 371a36d219164974abdf848841528152, State: Initialized, Role: FOLLOWER
I20260130 02:58:21.564414 19807 consensus_queue.cc:260] T 133e9bfe175f44aab1a53114abe67e4d P 371a36d219164974abdf848841528152 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:21.565972 19807 ts_tablet_manager.cc:1434] T 133e9bfe175f44aab1a53114abe67e4d P 371a36d219164974abdf848841528152: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
W20260130 02:58:21.530017 19313 outbound_call.cc:321] RPC callback for RPC call kudu.master.MasterService.TSHeartbeat -> {remote=127.18.100.124:34629, user_credentials={real_user=slave}} blocked reactor thread for 114776us
I20260130 02:58:21.567286 19807 tablet_bootstrap.cc:492] T ca06f07bb98f45a7bf63aebc28348e08 P 371a36d219164974abdf848841528152: Bootstrap starting.
I20260130 02:58:21.569087 19372 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "657e9b0d8c05413f904129dce745710b" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74" is_pre_election: true
I20260130 02:58:21.569738 19372 raft_consensus.cc:2468] T 657e9b0d8c05413f904129dce745710b P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d870164263bb44b6b780a557daea3656 in term 0.
W20260130 02:58:21.544912 19314 leader_election.cc:343] T 2b8034a92b2043e5b57cf02f1ee6296d P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:21.570508 19314 leader_election.cc:304] T 2b8034a92b2043e5b57cf02f1ee6296d P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78c2c46c7520492db35d51c244fb4f74; no voters: 371a36d219164974abdf848841528152, d870164263bb44b6b780a557daea3656
I20260130 02:58:21.571520 21026 raft_consensus.cc:2749] T 2b8034a92b2043e5b57cf02f1ee6296d P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260130 02:58:21.573880 19314 leader_election.cc:343] T 39bcc41f10ae4241b370529364454586 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:21.574304 19314 leader_election.cc:304] T 39bcc41f10ae4241b370529364454586 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78c2c46c7520492db35d51c244fb4f74; no voters: 371a36d219164974abdf848841528152, d870164263bb44b6b780a557daea3656
I20260130 02:58:21.574942 21026 raft_consensus.cc:2749] T 39bcc41f10ae4241b370529364454586 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260130 02:58:21.580662 19374 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "133e9bfe175f44aab1a53114abe67e4d" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74"
I20260130 02:58:21.581278 19374 raft_consensus.cc:3060] T 133e9bfe175f44aab1a53114abe67e4d P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:21.582741 19807 tablet_bootstrap.cc:654] T ca06f07bb98f45a7bf63aebc28348e08 P 371a36d219164974abdf848841528152: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:21.588997 19374 raft_consensus.cc:2468] T 133e9bfe175f44aab1a53114abe67e4d P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d870164263bb44b6b780a557daea3656 in term 1.
I20260130 02:58:21.595834 19807 tablet_bootstrap.cc:492] T ca06f07bb98f45a7bf63aebc28348e08 P 371a36d219164974abdf848841528152: No bootstrap required, opened a new log
I20260130 02:58:21.596272 19807 ts_tablet_manager.cc:1403] T ca06f07bb98f45a7bf63aebc28348e08 P 371a36d219164974abdf848841528152: Time spent bootstrapping tablet: real 0.029s user 0.004s sys 0.007s
I20260130 02:58:21.598521 19807 raft_consensus.cc:359] T ca06f07bb98f45a7bf63aebc28348e08 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:21.599157 19807 raft_consensus.cc:385] T ca06f07bb98f45a7bf63aebc28348e08 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:21.599440 19807 raft_consensus.cc:740] T ca06f07bb98f45a7bf63aebc28348e08 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 371a36d219164974abdf848841528152, State: Initialized, Role: FOLLOWER
I20260130 02:58:21.599987 19807 consensus_queue.cc:260] T ca06f07bb98f45a7bf63aebc28348e08 P 371a36d219164974abdf848841528152 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:21.600486 19625 tablet_bootstrap.cc:654] T 0855d88806914a36b4bf6b49036f0ad0 P d870164263bb44b6b780a557daea3656: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:21.608556 19375 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e34d5da0ad864b2eaea0d575c982e447" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74" is_pre_election: true
I20260130 02:58:21.609143 19375 raft_consensus.cc:2468] T e34d5da0ad864b2eaea0d575c982e447 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d870164263bb44b6b780a557daea3656 in term 0.
I20260130 02:58:21.610428 19807 ts_tablet_manager.cc:1434] T ca06f07bb98f45a7bf63aebc28348e08 P 371a36d219164974abdf848841528152: Time spent starting tablet: real 0.014s user 0.006s sys 0.000s
I20260130 02:58:21.610409 19375 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ca06f07bb98f45a7bf63aebc28348e08" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74" is_pre_election: true
I20260130 02:58:21.610957 19375 raft_consensus.cc:2468] T ca06f07bb98f45a7bf63aebc28348e08 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d870164263bb44b6b780a557daea3656 in term 0.
I20260130 02:58:21.613422 19375 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2b0edaad6877460cb7982efc62b00b66" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74" is_pre_election: true
I20260130 02:58:21.614127 19375 raft_consensus.cc:2468] T 2b0edaad6877460cb7982efc62b00b66 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d870164263bb44b6b780a557daea3656 in term 0.
I20260130 02:58:21.615721 20926 consensus_queue.cc:1048] T 320dbe98a32a43b9899bbeed80b20f55 P 371a36d219164974abdf848841528152 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260130 02:58:21.564821 19840 tablet_bootstrap.cc:654] T 9896ff9364744857968e87e80e776dd2 P 78c2c46c7520492db35d51c244fb4f74: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:21.624260 21042 tablet.cc:1722] T 9fe0448bcc4a4346b3c67fc3f626ed3c P 78c2c46c7520492db35d51c244fb4f74: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260130 02:58:21.626674 18867 catalog_manager.cc:5654] T af0f90071a334016aa17fae2e3a97fc8 P d870164263bb44b6b780a557daea3656 reported cstate change: term changed from 0 to 1, leader changed from <none> to d870164263bb44b6b780a557daea3656 (127.18.100.67). New cstate: current_term: 1 leader_uuid: "d870164263bb44b6b780a557daea3656" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } health_report { overall_health: HEALTHY } } }
I20260130 02:58:21.628785 18867 catalog_manager.cc:5746] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571) does not have the latest schema for tablet 345eaaa5625c48d099280b7fbb2e514b (table default.Table_c1886dd9a7934e7797e0e7e0b065d692 [id=7a2d5367224b492b844199088b01b149]). Expected version 1 got 0
I20260130 02:58:21.629719 21037 raft_consensus.cc:493] T d480bf18140245e3870a62582d5a8318 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:21.629918 18867 catalog_manager.cc:5746] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571) does not have the latest schema for tablet b1e8b78eff0d4a0b92059adb2de4a3ce (table default.Table_d0b4516d24384e5386c1debaf2543adb [id=f1342f9497b74e7a9ecad1de8082578d]). Expected version 1 got 0
I20260130 02:58:21.630147 21037 raft_consensus.cc:515] T d480bf18140245e3870a62582d5a8318 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } }
I20260130 02:58:21.632038 21037 leader_election.cc:290] T d480bf18140245e3870a62582d5a8318 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d870164263bb44b6b780a557daea3656 (127.18.100.67:39571), 371a36d219164974abdf848841528152 (127.18.100.65:42021)
I20260130 02:58:21.634462 19374 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a3258852aed24ed582a79f77860e6f0d" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74"
I20260130 02:58:21.635049 19374 raft_consensus.cc:3060] T a3258852aed24ed582a79f77860e6f0d P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:21.637293 19807 tablet_bootstrap.cc:492] T 657e9b0d8c05413f904129dce745710b P 371a36d219164974abdf848841528152: Bootstrap starting.
I20260130 02:58:21.637856 19371 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a3258852aed24ed582a79f77860e6f0d" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74" is_pre_election: true
I20260130 02:58:21.642483 19807 tablet_bootstrap.cc:654] T 657e9b0d8c05413f904129dce745710b P 371a36d219164974abdf848841528152: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:21.648118 21045 tablet.cc:1722] T 9fe0448bcc4a4346b3c67fc3f626ed3c P 371a36d219164974abdf848841528152: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260130 02:58:21.665585 19625 tablet_bootstrap.cc:492] T 0855d88806914a36b4bf6b49036f0ad0 P d870164263bb44b6b780a557daea3656: No bootstrap required, opened a new log
I20260130 02:58:21.665992 19625 ts_tablet_manager.cc:1403] T 0855d88806914a36b4bf6b49036f0ad0 P d870164263bb44b6b780a557daea3656: Time spent bootstrapping tablet: real 0.103s user 0.009s sys 0.000s
I20260130 02:58:21.668108 19625 raft_consensus.cc:359] T 0855d88806914a36b4bf6b49036f0ad0 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:21.670159 19625 raft_consensus.cc:385] T 0855d88806914a36b4bf6b49036f0ad0 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:21.672286 18867 catalog_manager.cc:5746] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571) does not have the latest schema for tablet 9fe0448bcc4a4346b3c67fc3f626ed3c (table default.Table_5c1c447e85524ac5a61ffd8a48509d36 [id=70aaf21ad9bd4d959f46939c6b4a4880]). Expected version 1 got 0
I20260130 02:58:21.672933 18867 catalog_manager.cc:5654] T 3aa76993beb94b12bc706a76db26a03b P d870164263bb44b6b780a557daea3656 reported cstate change: term changed from 0 to 1, leader changed from <none> to d870164263bb44b6b780a557daea3656 (127.18.100.67). New cstate: current_term: 1 leader_uuid: "d870164263bb44b6b780a557daea3656" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } health_report { overall_health: UNKNOWN } } }
I20260130 02:58:21.684026 19462 leader_election.cc:304] T 657e9b0d8c05413f904129dce745710b P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 78c2c46c7520492db35d51c244fb4f74, d870164263bb44b6b780a557daea3656; no voters: 371a36d219164974abdf848841528152
I20260130 02:58:21.686671 19522 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a3258852aed24ed582a79f77860e6f0d" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
I20260130 02:58:21.687144 20956 raft_consensus.cc:2804] T 657e9b0d8c05413f904129dce745710b P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260130 02:58:21.687508 20956 raft_consensus.cc:493] T 657e9b0d8c05413f904129dce745710b P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260130 02:58:21.687819 20956 raft_consensus.cc:3060] T 657e9b0d8c05413f904129dce745710b P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:21.691821 19522 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d480bf18140245e3870a62582d5a8318" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
I20260130 02:58:21.692380 19522 raft_consensus.cc:2468] T d480bf18140245e3870a62582d5a8318 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78c2c46c7520492db35d51c244fb4f74 in term 0.
I20260130 02:58:21.702890 20956 raft_consensus.cc:515] T 657e9b0d8c05413f904129dce745710b P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:21.708825 19374 raft_consensus.cc:2468] T a3258852aed24ed582a79f77860e6f0d P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d870164263bb44b6b780a557daea3656 in term 1.
I20260130 02:58:21.653357 19222 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d480bf18140245e3870a62582d5a8318" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152" is_pre_election: true
I20260130 02:58:21.661217 19807 tablet_bootstrap.cc:492] T 657e9b0d8c05413f904129dce745710b P 371a36d219164974abdf848841528152: No bootstrap required, opened a new log
I20260130 02:58:21.708545 19225 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "657e9b0d8c05413f904129dce745710b" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152"
I20260130 02:58:21.711714 19807 ts_tablet_manager.cc:1403] T 657e9b0d8c05413f904129dce745710b P 371a36d219164974abdf848841528152: Time spent bootstrapping tablet: real 0.075s user 0.013s sys 0.000s
W20260130 02:58:21.712495 19463 leader_election.cc:343] T 657e9b0d8c05413f904129dce745710b P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:21.714035 19807 raft_consensus.cc:359] T 657e9b0d8c05413f904129dce745710b P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
W20260130 02:58:21.714438 19314 leader_election.cc:343] T d480bf18140245e3870a62582d5a8318 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:21.714649 19807 raft_consensus.cc:385] T 657e9b0d8c05413f904129dce745710b P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:21.670423 19625 raft_consensus.cc:740] T 0855d88806914a36b4bf6b49036f0ad0 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d870164263bb44b6b780a557daea3656, State: Initialized, Role: FOLLOWER
I20260130 02:58:21.714902 19807 raft_consensus.cc:740] T 657e9b0d8c05413f904129dce745710b P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 371a36d219164974abdf848841528152, State: Initialized, Role: FOLLOWER
I20260130 02:58:21.715441 19807 consensus_queue.cc:260] T 657e9b0d8c05413f904129dce745710b P 371a36d219164974abdf848841528152 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:21.716629 19625 consensus_queue.cc:260] T 0855d88806914a36b4bf6b49036f0ad0 P d870164263bb44b6b780a557daea3656 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:21.717801 19807 ts_tablet_manager.cc:1434] T 657e9b0d8c05413f904129dce745710b P 371a36d219164974abdf848841528152: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20260130 02:58:21.727777 19625 ts_tablet_manager.cc:1434] T 0855d88806914a36b4bf6b49036f0ad0 P d870164263bb44b6b780a557daea3656: Time spent starting tablet: real 0.062s user 0.005s sys 0.000s
I20260130 02:58:21.735062 18867 catalog_manager.cc:5746] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571) does not have the latest schema for tablet 25fd48e693424537b91ba7c3fcc96878 (table default.Table_ebc8b8379c274c18b0a06a919dbe0ee0 [id=28156e0843674fce88f5914a3e4195ae]). Expected version 1 got 0
I20260130 02:58:21.739171 19840 tablet_bootstrap.cc:492] T 9896ff9364744857968e87e80e776dd2 P 78c2c46c7520492db35d51c244fb4f74: No bootstrap required, opened a new log
I20260130 02:58:21.739605 19840 ts_tablet_manager.cc:1403] T 9896ff9364744857968e87e80e776dd2 P 78c2c46c7520492db35d51c244fb4f74: Time spent bootstrapping tablet: real 0.243s user 0.010s sys 0.000s
I20260130 02:58:21.740692 18867 catalog_manager.cc:5746] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571) does not have the latest schema for tablet 274530359b174ff79f046c2d4a6afa76 (table default.Table_c4f4904b6fac4b9880979cf0314e71c1 [id=c456c6418c93499a9d08f652720854ca]). Expected version 1 got 0
I20260130 02:58:21.743062 18867 catalog_manager.cc:5746] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571) does not have the latest schema for tablet 838e49e5e8874db393bdc15a1c92f87e (table default.Table_da3d581b15a94840898777f74611994e [id=b7e241ced53e4160ac6fa1f65ef3674e]). Expected version 1 got 0
I20260130 02:58:21.715566 20956 leader_election.cc:290] T 657e9b0d8c05413f904129dce745710b P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 election: Requested vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767)
W20260130 02:58:21.730127 19462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.100.66:40767, user_credentials={real_user=slave}} blocked reactor thread for 46182.5us
I20260130 02:58:21.749007 19462 leader_election.cc:304] T 133e9bfe175f44aab1a53114abe67e4d P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 78c2c46c7520492db35d51c244fb4f74, d870164263bb44b6b780a557daea3656; no voters: 371a36d219164974abdf848841528152
I20260130 02:58:21.750342 20998 raft_consensus.cc:2804] T 133e9bfe175f44aab1a53114abe67e4d P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Leader election won for term 1
I20260130 02:58:21.750837 20998 raft_consensus.cc:697] T 133e9bfe175f44aab1a53114abe67e4d P d870164263bb44b6b780a557daea3656 [term 1 LEADER]: Becoming Leader. State: Replica: d870164263bb44b6b780a557daea3656, State: Running, Role: LEADER
I20260130 02:58:21.751657 20998 consensus_queue.cc:237] T 133e9bfe175f44aab1a53114abe67e4d P d870164263bb44b6b780a557daea3656 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:21.753064 19840 raft_consensus.cc:359] T 9896ff9364744857968e87e80e776dd2 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:21.753670 19840 raft_consensus.cc:385] T 9896ff9364744857968e87e80e776dd2 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:21.753913 19840 raft_consensus.cc:740] T 9896ff9364744857968e87e80e776dd2 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78c2c46c7520492db35d51c244fb4f74, State: Initialized, Role: FOLLOWER
I20260130 02:58:21.754448 19840 consensus_queue.cc:260] T 9896ff9364744857968e87e80e776dd2 P 78c2c46c7520492db35d51c244fb4f74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:21.757620 19313 leader_election.cc:304] T a3258852aed24ed582a79f77860e6f0d P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78c2c46c7520492db35d51c244fb4f74; no voters: 371a36d219164974abdf848841528152, d870164263bb44b6b780a557daea3656
I20260130 02:58:21.759159 21027 raft_consensus.cc:2749] T a3258852aed24ed582a79f77860e6f0d P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260130 02:58:21.761646 19840 ts_tablet_manager.cc:1434] T 9896ff9364744857968e87e80e776dd2 P 78c2c46c7520492db35d51c244fb4f74: Time spent starting tablet: real 0.022s user 0.005s sys 0.001s
I20260130 02:58:21.762552 19840 tablet_bootstrap.cc:492] T 0d14fb95ace9447f810ccbed4d5befdf P 78c2c46c7520492db35d51c244fb4f74: Bootstrap starting.
I20260130 02:58:21.759763 19462 leader_election.cc:304] T e34d5da0ad864b2eaea0d575c982e447 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 78c2c46c7520492db35d51c244fb4f74, d870164263bb44b6b780a557daea3656; no voters: 371a36d219164974abdf848841528152
I20260130 02:58:21.768802 21000 raft_consensus.cc:2804] T e34d5da0ad864b2eaea0d575c982e447 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260130 02:58:21.769187 21000 raft_consensus.cc:493] T e34d5da0ad864b2eaea0d575c982e447 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260130 02:58:21.769279 19840 tablet_bootstrap.cc:654] T 0d14fb95ace9447f810ccbed4d5befdf P 78c2c46c7520492db35d51c244fb4f74: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:21.769569 21000 raft_consensus.cc:3060] T e34d5da0ad864b2eaea0d575c982e447 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:21.775180 20956 consensus_queue.cc:1048] T 2c79a3aecab844698e70834999e1caac P d870164263bb44b6b780a557daea3656 [LEADER]: Connected to new peer: Peer: permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260130 02:58:21.776419 20899 ts_tablet_manager.cc:1929] T 8e57ae31ab3941f789ac69627384e3b0 P 78c2c46c7520492db35d51c244fb4f74: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260130 02:58:21.776978 20899 log.cc:1199] T 8e57ae31ab3941f789ac69627384e3b0 P 78c2c46c7520492db35d51c244fb4f74: Deleting WAL directory at /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-1/wal/wals/8e57ae31ab3941f789ac69627384e3b0
I20260130 02:58:21.777758 19462 leader_election.cc:304] T ca06f07bb98f45a7bf63aebc28348e08 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 78c2c46c7520492db35d51c244fb4f74, d870164263bb44b6b780a557daea3656; no voters: 371a36d219164974abdf848841528152
I20260130 02:58:21.777973 20899 ts_tablet_manager.cc:1950] T 8e57ae31ab3941f789ac69627384e3b0 P 78c2c46c7520492db35d51c244fb4f74: Deleting consensus metadata
I20260130 02:58:21.779240 20899 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60110 (request call id 318) took 2007 ms. Trace:
I20260130 02:58:21.779767 21049 raft_consensus.cc:2804] T ca06f07bb98f45a7bf63aebc28348e08 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260130 02:58:21.780128 21049 raft_consensus.cc:493] T ca06f07bb98f45a7bf63aebc28348e08 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260130 02:58:21.780433 21049 raft_consensus.cc:3060] T ca06f07bb98f45a7bf63aebc28348e08 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:21.781898 19313 leader_election.cc:304] T d480bf18140245e3870a62582d5a8318 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 78c2c46c7520492db35d51c244fb4f74, d870164263bb44b6b780a557daea3656; no voters: 371a36d219164974abdf848841528152
I20260130 02:58:21.783036 21021 raft_consensus.cc:2804] T d480bf18140245e3870a62582d5a8318 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260130 02:58:21.783357 21021 raft_consensus.cc:493] T d480bf18140245e3870a62582d5a8318 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260130 02:58:21.783648 21021 raft_consensus.cc:3060] T d480bf18140245e3870a62582d5a8318 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:21.788429 19462 leader_election.cc:304] T 2b0edaad6877460cb7982efc62b00b66 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 78c2c46c7520492db35d51c244fb4f74, d870164263bb44b6b780a557daea3656; no voters: 371a36d219164974abdf848841528152
I20260130 02:58:21.789618 20998 raft_consensus.cc:2804] T 2b0edaad6877460cb7982efc62b00b66 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260130 02:58:21.789976 20998 raft_consensus.cc:493] T 2b0edaad6877460cb7982efc62b00b66 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260130 02:58:21.790277 20998 raft_consensus.cc:3060] T 2b0edaad6877460cb7982efc62b00b66 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:21.790704 21021 raft_consensus.cc:515] T d480bf18140245e3870a62582d5a8318 P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } }
I20260130 02:58:21.793373 21021 leader_election.cc:290] T d480bf18140245e3870a62582d5a8318 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 election: Requested vote from peers d870164263bb44b6b780a557daea3656 (127.18.100.67:39571), 371a36d219164974abdf848841528152 (127.18.100.65:42021)
I20260130 02:58:21.793954 19222 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d480bf18140245e3870a62582d5a8318" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152"
W20260130 02:58:21.795137 19314 leader_election.cc:343] T d480bf18140245e3870a62582d5a8318 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:21.798825 19625 tablet_bootstrap.cc:492] T 1fc8f09d6329407c8078542b4123ef4d P d870164263bb44b6b780a557daea3656: Bootstrap starting.
I20260130 02:58:21.799757 19807 tablet_bootstrap.cc:492] T d480bf18140245e3870a62582d5a8318 P 371a36d219164974abdf848841528152: Bootstrap starting.
I20260130 02:58:21.801606 21000 raft_consensus.cc:515] T e34d5da0ad864b2eaea0d575c982e447 P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:21.805477 19807 tablet_bootstrap.cc:654] T d480bf18140245e3870a62582d5a8318 P 371a36d219164974abdf848841528152: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:21.808332 19840 tablet_bootstrap.cc:492] T 0d14fb95ace9447f810ccbed4d5befdf P 78c2c46c7520492db35d51c244fb4f74: No bootstrap required, opened a new log
I20260130 02:58:21.808743 19840 ts_tablet_manager.cc:1403] T 0d14fb95ace9447f810ccbed4d5befdf P 78c2c46c7520492db35d51c244fb4f74: Time spent bootstrapping tablet: real 0.046s user 0.003s sys 0.008s
I20260130 02:58:21.808627 20956 raft_consensus.cc:493] T dfbbe9045e7c4aa5b520218687e7e4c1 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:21.809041 20956 raft_consensus.cc:515] T dfbbe9045e7c4aa5b520218687e7e4c1 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:21.810966 19840 raft_consensus.cc:359] T 0d14fb95ace9447f810ccbed4d5befdf P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:21.811741 19840 raft_consensus.cc:385] T 0d14fb95ace9447f810ccbed4d5befdf P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:21.811988 19840 raft_consensus.cc:740] T 0d14fb95ace9447f810ccbed4d5befdf P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78c2c46c7520492db35d51c244fb4f74, State: Initialized, Role: FOLLOWER
I20260130 02:58:21.812443 19222 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dfbbe9045e7c4aa5b520218687e7e4c1" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152" is_pre_election: true
I20260130 02:58:21.812937 19222 raft_consensus.cc:2468] T dfbbe9045e7c4aa5b520218687e7e4c1 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d870164263bb44b6b780a557daea3656 in term 0.
I20260130 02:58:21.812654 19840 consensus_queue.cc:260] T 0d14fb95ace9447f810ccbed4d5befdf P 78c2c46c7520492db35d51c244fb4f74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:21.814019 19463 leader_election.cc:304] T dfbbe9045e7c4aa5b520218687e7e4c1 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 371a36d219164974abdf848841528152, d870164263bb44b6b780a557daea3656; no voters:
I20260130 02:58:21.816390 19840 ts_tablet_manager.cc:1434] T 0d14fb95ace9447f810ccbed4d5befdf P 78c2c46c7520492db35d51c244fb4f74: Time spent starting tablet: real 0.007s user 0.005s sys 0.001s
I20260130 02:58:21.817224 19840 tablet_bootstrap.cc:492] T dd377fab8c544bf69121dc9e9a060b29 P 78c2c46c7520492db35d51c244fb4f74: Bootstrap starting.
I20260130 02:58:21.819165 20956 leader_election.cc:290] T dfbbe9045e7c4aa5b520218687e7e4c1 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767)
I20260130 02:58:21.828224 20998 raft_consensus.cc:515] T 2b0edaad6877460cb7982efc62b00b66 P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:21.831965 19840 tablet_bootstrap.cc:654] T dd377fab8c544bf69121dc9e9a060b29 P 78c2c46c7520492db35d51c244fb4f74: Neither blocks nor log segments found. Creating new log.
W20260130 02:58:21.837977 19313 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.100.67:39571, user_credentials={real_user=slave}} blocked reactor thread for 56145.5us
I20260130 02:58:21.844890 19840 tablet_bootstrap.cc:492] T dd377fab8c544bf69121dc9e9a060b29 P 78c2c46c7520492db35d51c244fb4f74: No bootstrap required, opened a new log
I20260130 02:58:21.845295 19840 ts_tablet_manager.cc:1403] T dd377fab8c544bf69121dc9e9a060b29 P 78c2c46c7520492db35d51c244fb4f74: Time spent bootstrapping tablet: real 0.028s user 0.007s sys 0.001s
I20260130 02:58:21.845530 21026 raft_consensus.cc:493] T 49efcfced0b74c48962393149ac2a500 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:21.845919 21026 raft_consensus.cc:515] T 49efcfced0b74c48962393149ac2a500 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } }
I20260130 02:58:21.847476 19840 raft_consensus.cc:359] T dd377fab8c544bf69121dc9e9a060b29 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:21.847811 21026 leader_election.cc:290] T 49efcfced0b74c48962393149ac2a500 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d870164263bb44b6b780a557daea3656 (127.18.100.67:39571), 371a36d219164974abdf848841528152 (127.18.100.65:42021)
I20260130 02:58:21.848080 19840 raft_consensus.cc:385] T dd377fab8c544bf69121dc9e9a060b29 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:21.848312 19840 raft_consensus.cc:740] T dd377fab8c544bf69121dc9e9a060b29 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78c2c46c7520492db35d51c244fb4f74, State: Initialized, Role: FOLLOWER
I20260130 02:58:21.848841 19840 consensus_queue.cc:260] T dd377fab8c544bf69121dc9e9a060b29 P 78c2c46c7520492db35d51c244fb4f74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:21.860960 19840 ts_tablet_manager.cc:1434] T dd377fab8c544bf69121dc9e9a060b29 P 78c2c46c7520492db35d51c244fb4f74: Time spent starting tablet: real 0.015s user 0.005s sys 0.000s
I20260130 02:58:21.861514 19221 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "49efcfced0b74c48962393149ac2a500" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152" is_pre_election: true
I20260130 02:58:21.861855 19840 tablet_bootstrap.cc:492] T 6d1f121f49cd47c2b8247daaf6874e01 P 78c2c46c7520492db35d51c244fb4f74: Bootstrap starting.
W20260130 02:58:21.862634 19314 leader_election.cc:343] T 49efcfced0b74c48962393149ac2a500 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:21.869097 19840 tablet_bootstrap.cc:654] T 6d1f121f49cd47c2b8247daaf6874e01 P 78c2c46c7520492db35d51c244fb4f74: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:21.880133 19807 tablet_bootstrap.cc:492] T d480bf18140245e3870a62582d5a8318 P 371a36d219164974abdf848841528152: No bootstrap required, opened a new log
I20260130 02:58:21.880553 19807 ts_tablet_manager.cc:1403] T d480bf18140245e3870a62582d5a8318 P 371a36d219164974abdf848841528152: Time spent bootstrapping tablet: real 0.081s user 0.010s sys 0.002s
I20260130 02:58:21.850878 19522 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d480bf18140245e3870a62582d5a8318" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656"
I20260130 02:58:21.891047 19522 raft_consensus.cc:3060] T d480bf18140245e3870a62582d5a8318 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:21.892036 20998 leader_election.cc:290] T 2b0edaad6877460cb7982efc62b00b66 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 election: Requested vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767)
I20260130 02:58:21.876698 19521 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "49efcfced0b74c48962393149ac2a500" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
I20260130 02:58:21.894213 19521 raft_consensus.cc:2468] T 49efcfced0b74c48962393149ac2a500 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78c2c46c7520492db35d51c244fb4f74 in term 0.
I20260130 02:58:21.899246 19522 raft_consensus.cc:2468] T d480bf18140245e3870a62582d5a8318 P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78c2c46c7520492db35d51c244fb4f74 in term 1.
I20260130 02:58:21.833427 21000 leader_election.cc:290] T e34d5da0ad864b2eaea0d575c982e447 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 election: Requested vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767)
I20260130 02:58:21.882545 21060 raft_consensus.cc:2804] T dfbbe9045e7c4aa5b520218687e7e4c1 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260130 02:58:21.900622 21060 raft_consensus.cc:493] T dfbbe9045e7c4aa5b520218687e7e4c1 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260130 02:58:21.901011 21060 raft_consensus.cc:3060] T dfbbe9045e7c4aa5b520218687e7e4c1 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Advancing to term 1
W20260130 02:58:21.883371 19462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.100.66:40767, user_credentials={real_user=slave}} blocked reactor thread for 72339us
W20260130 02:58:21.886018 19463 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.100.65:42021, user_credentials={real_user=slave}} blocked reactor thread for 72059.3us
I20260130 02:58:21.886399 21061 tablet.cc:1722] T 9fe0448bcc4a4346b3c67fc3f626ed3c P d870164263bb44b6b780a557daea3656: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260130 02:58:21.907877 19807 raft_consensus.cc:359] T d480bf18140245e3870a62582d5a8318 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } }
I20260130 02:58:21.908257 19221 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2b0edaad6877460cb7982efc62b00b66" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152"
I20260130 02:58:21.908499 19807 raft_consensus.cc:385] T d480bf18140245e3870a62582d5a8318 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:21.908751 19807 raft_consensus.cc:740] T d480bf18140245e3870a62582d5a8318 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 371a36d219164974abdf848841528152, State: Initialized, Role: FOLLOWER
I20260130 02:58:21.909272 19807 consensus_queue.cc:260] T d480bf18140245e3870a62582d5a8318 P 371a36d219164974abdf848841528152 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } }
I20260130 02:58:21.909577 19225 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e34d5da0ad864b2eaea0d575c982e447" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152"
W20260130 02:58:21.917471 19463 leader_election.cc:343] T 2b0edaad6877460cb7982efc62b00b66 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Illegal state: must be running to vote when last-logged opid is not known
W20260130 02:58:21.918118 19463 leader_election.cc:343] T e34d5da0ad864b2eaea0d575c982e447 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:21.918215 21000 raft_consensus.cc:493] T eb7a755ab62b4552bb8588264c5f7b75 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:21.919334 19807 ts_tablet_manager.cc:1434] T d480bf18140245e3870a62582d5a8318 P 371a36d219164974abdf848841528152: Time spent starting tablet: real 0.039s user 0.006s sys 0.000s
I20260130 02:58:21.920220 19807 tablet_bootstrap.cc:492] T 49efcfced0b74c48962393149ac2a500 P 371a36d219164974abdf848841528152: Bootstrap starting.
I20260130 02:58:21.921538 20987 consensus_queue.cc:1048] T 320dbe98a32a43b9899bbeed80b20f55 P 371a36d219164974abdf848841528152 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260130 02:58:21.924551 20909 lock_manager.cc:585] Waited 3 seconds to obtain row lock on key '<redacted>' (key hash 16383881659362995755) tablet 00000000000000000000000000000000 cur holder 0x7b5800064e00
I20260130 02:58:21.918742 21000 raft_consensus.cc:515] T eb7a755ab62b4552bb8588264c5f7b75 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:21.936463 19313 leader_election.cc:304] T 49efcfced0b74c48962393149ac2a500 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 78c2c46c7520492db35d51c244fb4f74, d870164263bb44b6b780a557daea3656; no voters: 371a36d219164974abdf848841528152
I20260130 02:58:21.938072 21000 leader_election.cc:290] T eb7a755ab62b4552bb8588264c5f7b75 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767)
I20260130 02:58:21.918546 21060 raft_consensus.cc:515] T dfbbe9045e7c4aa5b520218687e7e4c1 P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:21.940413 21060 leader_election.cc:290] T dfbbe9045e7c4aa5b520218687e7e4c1 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 election: Requested vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767)
I20260130 02:58:21.943025 19222 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dfbbe9045e7c4aa5b520218687e7e4c1" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152"
I20260130 02:58:21.943506 19222 raft_consensus.cc:3060] T dfbbe9045e7c4aa5b520218687e7e4c1 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:21.944168 20988 raft_consensus.cc:493] T 9896ff9364744857968e87e80e776dd2 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:21.944669 20988 raft_consensus.cc:515] T 9896ff9364744857968e87e80e776dd2 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:21.943050 19221 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "eb7a755ab62b4552bb8588264c5f7b75" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152" is_pre_election: true
I20260130 02:58:21.948746 20988 leader_election.cc:290] T 9896ff9364744857968e87e80e776dd2 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d870164263bb44b6b780a557daea3656 (127.18.100.67:39571), 371a36d219164974abdf848841528152 (127.18.100.65:42021)
I20260130 02:58:21.949602 19371 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dfbbe9045e7c4aa5b520218687e7e4c1" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74" is_pre_election: true
I20260130 02:58:21.950142 19371 raft_consensus.cc:2468] T dfbbe9045e7c4aa5b520218687e7e4c1 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d870164263bb44b6b780a557daea3656 in term 0.
I20260130 02:58:21.950911 19374 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "657e9b0d8c05413f904129dce745710b" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74"
I20260130 02:58:21.951416 19374 raft_consensus.cc:3060] T 657e9b0d8c05413f904129dce745710b P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:21.779708 20899 rpcz_store.cc:276] 0130 02:58:19.771655 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:58:19.810689 (+ 39034us) service_pool.cc:225] Handling call
0130 02:58:20.721577 (+910888us) ts_tablet_manager.cc:1151] Deleting tablet 8e57ae31ab3941f789ac69627384e3b0
0130 02:58:20.721616 (+ 39us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0130 02:58:21.448840 (+727224us) spinlock_profiling.cc:243] Waited 6.8 ms on lock 0x7b58000de2d0. stack: 00007f213f77c373 00007f213f77c270 00007f213f2f1574 0000000000508744 00000000005086d9 00007f21467cd5a1 00007f21467c00da 00007f2146842177 00007f214686c005 00007f2146c8e2ec 00007f21478e531a 00007f2147915f80 00007f21478fc54d 00007f21478fc4f9 00007f21478fc489 00007f21478fc451
0130 02:58:21.595305 (+146465us) tablet_metadata.cc:641] Metadata flushed
0130 02:58:21.775764 (+180459us) tablet_metadata.cc:641] Metadata flushed
0130 02:58:21.779186 (+ 3422us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":738506,"spinlock_wait_cycles":15637888,"tablet-delete.queue_time_us":910306}
I20260130 02:58:21.955577 19222 raft_consensus.cc:2468] T dfbbe9045e7c4aa5b520218687e7e4c1 P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d870164263bb44b6b780a557daea3656 in term 1.
I20260130 02:58:21.961964 20899 tablet_replica.cc:333] T a8a5038e9def44eb8b3b1e4aeae25652 P 78c2c46c7520492db35d51c244fb4f74: stopping tablet replica
I20260130 02:58:21.962873 20899 raft_consensus.cc:2243] T a8a5038e9def44eb8b3b1e4aeae25652 P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260130 02:58:21.963403 20899 raft_consensus.cc:2272] T a8a5038e9def44eb8b3b1e4aeae25652 P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260130 02:58:21.964478 19807 tablet_bootstrap.cc:654] T 49efcfced0b74c48962393149ac2a500 P 371a36d219164974abdf848841528152: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:21.965355 20899 ts_tablet_manager.cc:1916] T a8a5038e9def44eb8b3b1e4aeae25652 P 78c2c46c7520492db35d51c244fb4f74: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260130 02:58:21.967154 18860 catalog_manager.cc:4985] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767): tablet 8e57ae31ab3941f789ac69627384e3b0 (table default.Table_1ae68ae3f0404843aca3c72f02678ac6 [id=8a9182f49e18489fafec8dca190c1b2f]) successfully deleted
I20260130 02:58:21.973687 21060 raft_consensus.cc:493] T ab6afa40408a4f458495abf20290e55b P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:21.974256 21060 raft_consensus.cc:515] T ab6afa40408a4f458495abf20290e55b P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:21.976011 19625 tablet_bootstrap.cc:654] T 1fc8f09d6329407c8078542b4123ef4d P d870164263bb44b6b780a557daea3656: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:21.976117 21060 leader_election.cc:290] T ab6afa40408a4f458495abf20290e55b P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767)
I20260130 02:58:21.977257 19222 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9896ff9364744857968e87e80e776dd2" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152" is_pre_election: true
I20260130 02:58:21.977541 21026 raft_consensus.cc:2804] T 49efcfced0b74c48962393149ac2a500 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260130 02:58:21.977887 21026 raft_consensus.cc:493] T 49efcfced0b74c48962393149ac2a500 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260130 02:58:21.978216 21026 raft_consensus.cc:3060] T 49efcfced0b74c48962393149ac2a500 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Advancing to term 1
W20260130 02:58:21.978302 19314 leader_election.cc:343] T 9896ff9364744857968e87e80e776dd2 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:21.837807 21049 raft_consensus.cc:515] T ca06f07bb98f45a7bf63aebc28348e08 P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
W20260130 02:58:21.986438 19313 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.100.67:39571, user_credentials={real_user=slave}} blocked reactor thread for 50046.6us
I20260130 02:58:21.987083 19313 leader_election.cc:304] T d480bf18140245e3870a62582d5a8318 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 78c2c46c7520492db35d51c244fb4f74, d870164263bb44b6b780a557daea3656; no voters: 371a36d219164974abdf848841528152
I20260130 02:58:22.002592 19807 tablet_bootstrap.cc:492] T 49efcfced0b74c48962393149ac2a500 P 371a36d219164974abdf848841528152: No bootstrap required, opened a new log
I20260130 02:58:22.002993 19807 ts_tablet_manager.cc:1403] T 49efcfced0b74c48962393149ac2a500 P 371a36d219164974abdf848841528152: Time spent bootstrapping tablet: real 0.083s user 0.010s sys 0.000s
I20260130 02:58:22.005283 19807 raft_consensus.cc:359] T 49efcfced0b74c48962393149ac2a500 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } }
I20260130 02:58:22.009689 19807 raft_consensus.cc:385] T 49efcfced0b74c48962393149ac2a500 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:22.009960 19807 raft_consensus.cc:740] T 49efcfced0b74c48962393149ac2a500 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 371a36d219164974abdf848841528152, State: Initialized, Role: FOLLOWER
I20260130 02:58:22.010502 19807 consensus_queue.cc:260] T 49efcfced0b74c48962393149ac2a500 P 371a36d219164974abdf848841528152 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } }
I20260130 02:58:22.013608 21021 raft_consensus.cc:493] T 00ddbfe6218c44d788fbf65245a31dc6 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:22.014045 21021 raft_consensus.cc:515] T 00ddbfe6218c44d788fbf65245a31dc6 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:22.016705 19222 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00ddbfe6218c44d788fbf65245a31dc6" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152" is_pre_election: true
W20260130 02:58:22.018031 19314 leader_election.cc:343] T 00ddbfe6218c44d788fbf65245a31dc6 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:22.018954 21021 leader_election.cc:290] T 00ddbfe6218c44d788fbf65245a31dc6 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), d870164263bb44b6b780a557daea3656 (127.18.100.67:39571)
I20260130 02:58:22.027563 19807 ts_tablet_manager.cc:1434] T 49efcfced0b74c48962393149ac2a500 P 371a36d219164974abdf848841528152: Time spent starting tablet: real 0.024s user 0.005s sys 0.000s
I20260130 02:58:22.028482 19807 tablet_bootstrap.cc:492] T aae68390dbbe4dc58ce3d3bd01725ae6 P 371a36d219164974abdf848841528152: Bootstrap starting.
I20260130 02:58:22.033401 19807 tablet_bootstrap.cc:654] T aae68390dbbe4dc58ce3d3bd01725ae6 P 371a36d219164974abdf848841528152: Neither blocks nor log segments found. Creating new log.
W20260130 02:58:22.035429 19313 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.100.67:39571, user_credentials={real_user=slave}} blocked reactor thread for 48408.2us
I20260130 02:58:22.037667 19375 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e34d5da0ad864b2eaea0d575c982e447" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74"
I20260130 02:58:22.038244 19375 raft_consensus.cc:3060] T e34d5da0ad864b2eaea0d575c982e447 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:22.051882 21026 raft_consensus.cc:515] T 49efcfced0b74c48962393149ac2a500 P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } }
I20260130 02:58:22.057611 19222 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "49efcfced0b74c48962393149ac2a500" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152"
I20260130 02:58:22.058120 19222 raft_consensus.cc:3060] T 49efcfced0b74c48962393149ac2a500 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:22.065809 21021 raft_consensus.cc:2804] T d480bf18140245e3870a62582d5a8318 P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Leader election won for term 1
I20260130 02:58:22.067016 21021 raft_consensus.cc:697] T d480bf18140245e3870a62582d5a8318 P 78c2c46c7520492db35d51c244fb4f74 [term 1 LEADER]: Becoming Leader. State: Replica: 78c2c46c7520492db35d51c244fb4f74, State: Running, Role: LEADER
I20260130 02:58:22.067834 21021 consensus_queue.cc:237] T d480bf18140245e3870a62582d5a8318 P 78c2c46c7520492db35d51c244fb4f74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } }
I20260130 02:58:22.075663 19807 tablet_bootstrap.cc:492] T aae68390dbbe4dc58ce3d3bd01725ae6 P 371a36d219164974abdf848841528152: No bootstrap required, opened a new log
I20260130 02:58:22.076076 19807 ts_tablet_manager.cc:1403] T aae68390dbbe4dc58ce3d3bd01725ae6 P 371a36d219164974abdf848841528152: Time spent bootstrapping tablet: real 0.048s user 0.007s sys 0.002s
I20260130 02:58:22.076903 19222 raft_consensus.cc:2468] T 49efcfced0b74c48962393149ac2a500 P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78c2c46c7520492db35d51c244fb4f74 in term 1.
I20260130 02:58:22.077929 19314 leader_election.cc:304] T 49efcfced0b74c48962393149ac2a500 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 371a36d219164974abdf848841528152, 78c2c46c7520492db35d51c244fb4f74; no voters:
I20260130 02:58:22.078230 19807 raft_consensus.cc:359] T aae68390dbbe4dc58ce3d3bd01725ae6 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } }
I20260130 02:58:22.078719 21037 raft_consensus.cc:2804] T 49efcfced0b74c48962393149ac2a500 P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Leader election won for term 1
I20260130 02:58:22.078852 19807 raft_consensus.cc:385] T aae68390dbbe4dc58ce3d3bd01725ae6 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:22.079149 21037 raft_consensus.cc:697] T 49efcfced0b74c48962393149ac2a500 P 78c2c46c7520492db35d51c244fb4f74 [term 1 LEADER]: Becoming Leader. State: Replica: 78c2c46c7520492db35d51c244fb4f74, State: Running, Role: LEADER
I20260130 02:58:22.079869 21037 consensus_queue.cc:237] T 49efcfced0b74c48962393149ac2a500 P 78c2c46c7520492db35d51c244fb4f74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } }
I20260130 02:58:22.083984 19374 raft_consensus.cc:2468] T 657e9b0d8c05413f904129dce745710b P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d870164263bb44b6b780a557daea3656 in term 1.
I20260130 02:58:22.079073 19807 raft_consensus.cc:740] T aae68390dbbe4dc58ce3d3bd01725ae6 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 371a36d219164974abdf848841528152, State: Initialized, Role: FOLLOWER
I20260130 02:58:22.087292 19807 consensus_queue.cc:260] T aae68390dbbe4dc58ce3d3bd01725ae6 P 371a36d219164974abdf848841528152 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } }
I20260130 02:58:22.092777 19807 ts_tablet_manager.cc:1434] T aae68390dbbe4dc58ce3d3bd01725ae6 P 371a36d219164974abdf848841528152: Time spent starting tablet: real 0.016s user 0.004s sys 0.001s
I20260130 02:58:22.093606 19807 tablet_bootstrap.cc:492] T e34d5da0ad864b2eaea0d575c982e447 P 371a36d219164974abdf848841528152: Bootstrap starting.
I20260130 02:58:22.101773 20704 raft_consensus.cc:493] T ab6afa40408a4f458495abf20290e55b P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:22.102176 20704 raft_consensus.cc:515] T ab6afa40408a4f458495abf20290e55b P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:22.104812 20704 leader_election.cc:290] T ab6afa40408a4f458495abf20290e55b P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d870164263bb44b6b780a557daea3656 (127.18.100.67:39571), 371a36d219164974abdf848841528152 (127.18.100.65:42021)
I20260130 02:58:22.111733 19372 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2b0edaad6877460cb7982efc62b00b66" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74"
I20260130 02:58:22.112372 19372 raft_consensus.cc:3060] T 2b0edaad6877460cb7982efc62b00b66 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:22.106488 19221 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ab6afa40408a4f458495abf20290e55b" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152" is_pre_election: true
I20260130 02:58:22.109174 20926 raft_consensus.cc:493] T 78e9bc3e55fe4e4a98e8eb01da59d365 P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d870164263bb44b6b780a557daea3656)
W20260130 02:58:22.131837 19314 leader_election.cc:343] T ab6afa40408a4f458495abf20290e55b P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:22.131945 20926 raft_consensus.cc:515] T 78e9bc3e55fe4e4a98e8eb01da59d365 P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:22.133687 20926 leader_election.cc:290] T 78e9bc3e55fe4e4a98e8eb01da59d365 P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767), d870164263bb44b6b780a557daea3656 (127.18.100.67:39571)
I20260130 02:58:22.098565 19807 tablet_bootstrap.cc:654] T e34d5da0ad864b2eaea0d575c982e447 P 371a36d219164974abdf848841528152: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:22.145458 18868 catalog_manager.cc:5746] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767) does not have the latest schema for tablet 345eaaa5625c48d099280b7fbb2e514b (table default.Table_c1886dd9a7934e7797e0e7e0b065d692 [id=7a2d5367224b492b844199088b01b149]). Expected version 1 got 0
I20260130 02:58:22.146363 18868 catalog_manager.cc:5746] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767) does not have the latest schema for tablet b1e8b78eff0d4a0b92059adb2de4a3ce (table default.Table_d0b4516d24384e5386c1debaf2543adb [id=f1342f9497b74e7a9ecad1de8082578d]). Expected version 1 got 0
I20260130 02:58:22.148543 20926 raft_consensus.cc:493] T 133e9bfe175f44aab1a53114abe67e4d P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:22.149004 20926 raft_consensus.cc:515] T 133e9bfe175f44aab1a53114abe67e4d P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:22.150851 20926 leader_election.cc:290] T 133e9bfe175f44aab1a53114abe67e4d P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d870164263bb44b6b780a557daea3656 (127.18.100.67:39571), 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767)
I20260130 02:58:22.169404 19625 tablet_bootstrap.cc:492] T 1fc8f09d6329407c8078542b4123ef4d P d870164263bb44b6b780a557daea3656: No bootstrap required, opened a new log
I20260130 02:58:22.173321 19371 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "133e9bfe175f44aab1a53114abe67e4d" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74" is_pre_election: true
W20260130 02:58:22.173491 19159 leader_election.cc:343] T 78e9bc3e55fe4e4a98e8eb01da59d365 P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767): Not found: Tablet not found: 78e9bc3e55fe4e4a98e8eb01da59d365
I20260130 02:58:22.174086 19371 raft_consensus.cc:2393] T 133e9bfe175f44aab1a53114abe67e4d P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 371a36d219164974abdf848841528152 in current term 1: Already voted for candidate d870164263bb44b6b780a557daea3656 in this term.
I20260130 02:58:22.174870 20988 raft_consensus.cc:493] T aae68390dbbe4dc58ce3d3bd01725ae6 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:22.176932 18868 catalog_manager.cc:5746] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767) does not have the latest schema for tablet 2b0edaad6877460cb7982efc62b00b66 (table default.Table_d95381c590c94802bdc83acef0f0df96 [id=64a8e0e079f34806aa68795f3d548c5e]). Expected version 1 got 0
I20260130 02:58:22.179453 18868 catalog_manager.cc:5746] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767) does not have the latest schema for tablet 9fe0448bcc4a4346b3c67fc3f626ed3c (table default.Table_5c1c447e85524ac5a61ffd8a48509d36 [id=70aaf21ad9bd4d959f46939c6b4a4880]). Expected version 1 got 0
I20260130 02:58:22.180757 18868 catalog_manager.cc:5746] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767) does not have the latest schema for tablet 274530359b174ff79f046c2d4a6afa76 (table default.Table_c4f4904b6fac4b9880979cf0314e71c1 [id=c456c6418c93499a9d08f652720854ca]). Expected version 1 got 0
I20260130 02:58:22.183817 19807 tablet_bootstrap.cc:492] T e34d5da0ad864b2eaea0d575c982e447 P 371a36d219164974abdf848841528152: No bootstrap required, opened a new log
W20260130 02:58:22.183789 21050 log.cc:927] Time spent T 00000000000000000000000000000000 P 3749baa287ed4f249329a74071781edd: Append to log took a long time: real 0.380s user 0.029s sys 0.000s
I20260130 02:58:22.184216 19807 ts_tablet_manager.cc:1403] T e34d5da0ad864b2eaea0d575c982e447 P 371a36d219164974abdf848841528152: Time spent bootstrapping tablet: real 0.091s user 0.010s sys 0.005s
I20260130 02:58:22.191185 18868 catalog_manager.cc:5746] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767) does not have the latest schema for tablet 657e9b0d8c05413f904129dce745710b (table default.Table_526677c8ef4144bd9de7a54a5e78e2be [id=b16908b75587459597a978c71756451c]). Expected version 1 got 0
I20260130 02:58:22.191728 18868 catalog_manager.cc:5746] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767) does not have the latest schema for tablet 49efcfced0b74c48962393149ac2a500 (table default.Table_2e81f0a0cbc449a397141b717db17025 [id=a9c919f57c0c437cb35a6921103e45a6]). Expected version 1 got 0
I20260130 02:58:22.192193 18868 catalog_manager.cc:5746] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767) does not have the latest schema for tablet 25fd48e693424537b91ba7c3fcc96878 (table default.Table_ebc8b8379c274c18b0a06a919dbe0ee0 [id=28156e0843674fce88f5914a3e4195ae]). Expected version 1 got 0
I20260130 02:58:22.193769 18868 catalog_manager.cc:5746] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767) does not have the latest schema for tablet 838e49e5e8874db393bdc15a1c92f87e (table default.Table_da3d581b15a94840898777f74611994e [id=b7e241ced53e4160ac6fa1f65ef3674e]). Expected version 1 got 0
I20260130 02:58:22.204849 18873 catalog_manager.cc:6199] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4: default.Table_8f311c5878744ef692b496ecfdfc3ede [id=4f499357303d464aafc17679ffcaaa07] alter complete (version 1)
I20260130 02:58:22.221591 18870 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:50284 (request call id 255) took 3648 ms. Trace:
I20260130 02:58:22.186422 19807 raft_consensus.cc:359] T e34d5da0ad864b2eaea0d575c982e447 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:22.223090 19807 raft_consensus.cc:385] T e34d5da0ad864b2eaea0d575c982e447 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:22.223349 19807 raft_consensus.cc:740] T e34d5da0ad864b2eaea0d575c982e447 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 371a36d219164974abdf848841528152, State: Initialized, Role: FOLLOWER
I20260130 02:58:22.223896 19807 consensus_queue.cc:260] T e34d5da0ad864b2eaea0d575c982e447 P 371a36d219164974abdf848841528152 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:22.221967 18870 rpcz_store.cc:276] 0130 02:58:18.572764 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:58:19.511767 (+939003us) service_pool.cc:225] Handling call
0130 02:58:19.512563 (+ 796us) catalog_manager.cc:2393] Looking up, locking, and authorizing table
0130 02:58:19.513467 (+ 904us) catalog_manager.cc:3708] Apply alter schema
0130 02:58:19.514021 (+ 554us) catalog_manager.cc:3740] Acquired catalog manager lock
0130 02:58:19.514210 (+ 189us) catalog_manager.cc:3898] Updating metadata on disk
0130 02:58:19.769593 (+255383us) catalog_manager.cc:3929] Committing alterations to in-memory state
0130 02:58:22.221431 (+2451838us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0130 02:58:19.724283 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0130 02:58:19.724670 (+ 387us) write_op.cc:432] Acquiring schema lock in shared mode
0130 02:58:19.724692 (+ 22us) write_op.cc:435] Acquired schema lock
0130 02:58:19.724705 (+ 13us) tablet.cc:660] Decoding operations
0130 02:58:19.724953 (+ 248us) write_op.cc:620] Acquiring the partition lock for write op
0130 02:58:19.724988 (+ 35us) write_op.cc:641] Partition lock acquired for write op
0130 02:58:19.725009 (+ 21us) tablet.cc:683] Acquiring locks for 1 operations
0130 02:58:19.725147 (+ 138us) tablet.cc:699] Row locks acquired
0130 02:58:19.725159 (+ 12us) write_op.cc:260] PREPARE: finished
0130 02:58:19.725259 (+ 100us) write_op.cc:270] Start()
0130 02:58:19.725345 (+ 86us) write_op.cc:276] Timestamp: P: 1769741899725163 usec, L: 0
0130 02:58:19.725360 (+ 15us) op_driver.cc:348] REPLICATION: starting
0130 02:58:19.725802 (+ 442us) log.cc:844] Serialized 405 byte log entry
0130 02:58:19.749288 (+ 23486us) op_driver.cc:464] REPLICATION: finished
0130 02:58:19.751249 (+ 1961us) write_op.cc:301] APPLY: starting
0130 02:58:19.751297 (+ 48us) tablet.cc:1365] starting BulkCheckPresence
0130 02:58:19.751369 (+ 72us) tablet.cc:1368] finished BulkCheckPresence
0130 02:58:19.751454 (+ 85us) tablet.cc:1370] starting ApplyRowOperation cycle
0130 02:58:19.751603 (+ 149us) tablet.cc:1381] finished ApplyRowOperation cycle
0130 02:58:19.758934 (+ 7331us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0130 02:58:19.758976 (+ 42us) write_op.cc:312] APPLY: finished
0130 02:58:19.759138 (+ 162us) log.cc:844] Serialized 23 byte log entry
0130 02:58:19.759272 (+ 134us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:58:19.759330 (+ 58us) write_op.cc:454] Released schema lock
0130 02:58:19.759362 (+ 32us) write_op.cc:341] FINISH: Updating metrics
0130 02:58:19.759583 (+ 221us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:58:19.759926 (+ 343us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":1839,"apply.run_cpu_time_us":1100,"apply.run_wall_time_us":8284,"mutex_wait_us":108265,"num_ops":1,"prepare.queue_time_us":112915,"prepare.run_cpu_time_us":1763,"prepare.run_wall_time_us":3666,"raft.queue_time_us":913,"raft.run_cpu_time_us":12,"raft.run_wall_time_us":12,"replication_time_us":23852}]]}
I20260130 02:58:22.227931 21049 leader_election.cc:290] T ca06f07bb98f45a7bf63aebc28348e08 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 election: Requested vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767)
W20260130 02:58:22.230698 18870 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.18.100.65:51021 (request call id 41) took 2108 ms (client timeout 1000 ms). Trace:
I20260130 02:58:22.225345 19807 ts_tablet_manager.cc:1434] T e34d5da0ad864b2eaea0d575c982e447 P 371a36d219164974abdf848841528152: Time spent starting tablet: real 0.041s user 0.004s sys 0.000s
W20260130 02:58:22.231073 18870 rpcz_store.cc:269] 0130 02:58:20.122506 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:58:22.230480 (+2107974us) service_pool.cc:212] Skipping call since client already timed out
0130 02:58:22.230645 (+ 165us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20260130 02:58:22.232191 18870 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.18.100.67:47213 (request call id 35) took 1237 ms (client timeout 1000 ms). Trace:
W20260130 02:58:22.233054 18870 rpcz_store.cc:269] 0130 02:58:20.995182 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:58:22.232063 (+1236881us) service_pool.cc:212] Skipping call since client already timed out
0130 02:58:22.232147 (+ 84us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20260130 02:58:22.235824 19807 tablet_bootstrap.cc:492] T 2b0edaad6877460cb7982efc62b00b66 P 371a36d219164974abdf848841528152: Bootstrap starting.
I20260130 02:58:22.236471 18875 catalog_manager.cc:6199] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4: default.Table_e4d07634d84a4678b1c94374e3adc476 [id=fcbd7e3a7fc64d89aa740494d7e1a0f7] alter complete (version 1)
I20260130 02:58:22.236567 18866 catalog_manager.cc:5941] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4: Sending DeleteTablet for 3 replicas of tablet e12d130bcdf34965888753f12c1ff848
I20260130 02:58:22.237948 18866 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:50284 (request call id 249) took 5282 ms. Trace:
I20260130 02:58:22.238517 19202 tablet_service.cc:1552] Processing DeleteTablet for tablet e12d130bcdf34965888753f12c1ff848 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:58:18 UTC) from {username='slave'} at 127.0.0.1:42870
I20260130 02:58:22.239959 18871 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:50284 (request call id 256) took 3557 ms. Trace:
I20260130 02:58:22.238204 18866 rpcz_store.cc:276] 0130 02:58:16.955746 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:58:18.937150 (+1981404us) service_pool.cc:225] Handling call
0130 02:58:18.944455 (+ 7305us) catalog_manager.cc:2393] Looking up, locking, and authorizing table
0130 02:58:18.974051 (+ 29596us) catalog_manager.cc:2815] Modifying in-memory table state
0130 02:58:18.974175 (+ 124us) catalog_manager.cc:2824] Locking tablets
0130 02:58:19.033188 (+ 59013us) catalog_manager.cc:2838] Removing table and tablets from system table
0130 02:58:19.061937 (+ 28749us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:58:19.061964 (+ 27us) write_op.cc:454] Released schema lock
0130 02:58:19.062210 (+ 246us) catalog_manager.cc:2856] Removing table from by-name map
0130 02:58:19.351337 (+289127us) catalog_manager.cc:2872] Committing in-memory state
0130 02:58:22.236366 (+2885029us) catalog_manager.cc:2876] Aborting table tasks
0130 02:58:22.237881 (+ 1515us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0130 02:58:19.035032 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0130 02:58:19.035564 (+ 532us) write_op.cc:432] Acquiring schema lock in shared mode
0130 02:58:19.035590 (+ 26us) write_op.cc:435] Acquired schema lock
0130 02:58:19.035608 (+ 18us) tablet.cc:660] Decoding operations
0130 02:58:19.036494 (+ 886us) write_op.cc:620] Acquiring the partition lock for write op
0130 02:58:19.036555 (+ 61us) write_op.cc:641] Partition lock acquired for write op
0130 02:58:19.036579 (+ 24us) tablet.cc:683] Acquiring locks for 2 operations
0130 02:58:19.036866 (+ 287us) tablet.cc:699] Row locks acquired
0130 02:58:19.036884 (+ 18us) write_op.cc:260] PREPARE: finished
0130 02:58:19.037006 (+ 122us) write_op.cc:270] Start()
0130 02:58:19.037091 (+ 85us) write_op.cc:276] Timestamp: P: 1769741899036934 usec, L: 0
0130 02:58:19.037110 (+ 19us) op_driver.cc:348] REPLICATION: starting
0130 02:58:19.037645 (+ 535us) log.cc:844] Serialized 731 byte log entry
0130 02:58:19.048285 (+ 10640us) op_driver.cc:464] REPLICATION: finished
0130 02:58:19.048922 (+ 637us) write_op.cc:301] APPLY: starting
0130 02:58:19.048976 (+ 54us) tablet.cc:1365] starting BulkCheckPresence
0130 02:58:19.049069 (+ 93us) tablet.cc:1368] finished BulkCheckPresence
0130 02:58:19.049159 (+ 90us) tablet.cc:1370] starting ApplyRowOperation cycle
0130 02:58:19.049412 (+ 253us) tablet.cc:1381] finished ApplyRowOperation cycle
0130 02:58:19.049550 (+ 138us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0130 02:58:19.049591 (+ 41us) write_op.cc:312] APPLY: finished
0130 02:58:19.049763 (+ 172us) log.cc:844] Serialized 29 byte log entry
0130 02:58:19.049905 (+ 142us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:58:19.049973 (+ 68us) write_op.cc:454] Released schema lock
0130 02:58:19.050002 (+ 29us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":474,"apply.run_cpu_time_us":1248,"apply.run_wall_time_us":1249,"num_ops":2,"prepare.queue_time_us":94,"prepare.run_cpu_time_us":2942,"prepare.run_wall_time_us":5953,"raft.queue_time_us":83,"raft.run_cpu_time_us":705,"raft.run_wall_time_us":704,"replication_time_us":11063}]]}
I20260130 02:58:22.240229 18871 rpcz_store.cc:276] 0130 02:58:18.682617 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:58:19.516056 (+833439us) service_pool.cc:225] Handling call
0130 02:58:19.526026 (+ 9970us) catalog_manager.cc:2393] Looking up, locking, and authorizing table
0130 02:58:19.527210 (+ 1184us) catalog_manager.cc:3708] Apply alter schema
0130 02:58:19.527736 (+ 526us) catalog_manager.cc:3740] Acquired catalog manager lock
0130 02:58:19.527935 (+ 199us) catalog_manager.cc:3898] Updating metadata on disk
0130 02:58:19.650874 (+122939us) catalog_manager.cc:3929] Committing alterations to in-memory state
0130 02:58:22.239784 (+2588910us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0130 02:58:19.630799 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0130 02:58:19.631198 (+ 399us) write_op.cc:432] Acquiring schema lock in shared mode
0130 02:58:19.631229 (+ 31us) write_op.cc:435] Acquired schema lock
0130 02:58:19.631241 (+ 12us) tablet.cc:660] Decoding operations
0130 02:58:19.631484 (+ 243us) write_op.cc:620] Acquiring the partition lock for write op
0130 02:58:19.631522 (+ 38us) write_op.cc:641] Partition lock acquired for write op
0130 02:58:19.631542 (+ 20us) tablet.cc:683] Acquiring locks for 1 operations
0130 02:58:19.631674 (+ 132us) tablet.cc:699] Row locks acquired
0130 02:58:19.631687 (+ 13us) write_op.cc:260] PREPARE: finished
0130 02:58:19.631792 (+ 105us) write_op.cc:270] Start()
0130 02:58:19.631867 (+ 75us) write_op.cc:276] Timestamp: P: 1769741899631705 usec, L: 0
0130 02:58:19.631883 (+ 16us) op_driver.cc:348] REPLICATION: starting
0130 02:58:19.632370 (+ 487us) log.cc:844] Serialized 405 byte log entry
0130 02:58:19.647904 (+ 15534us) op_driver.cc:464] REPLICATION: finished
0130 02:58:19.648231 (+ 327us) write_op.cc:301] APPLY: starting
0130 02:58:19.648289 (+ 58us) tablet.cc:1365] starting BulkCheckPresence
0130 02:58:19.648373 (+ 84us) tablet.cc:1368] finished BulkCheckPresence
0130 02:58:19.648479 (+ 106us) tablet.cc:1370] starting ApplyRowOperation cycle
0130 02:58:19.648653 (+ 174us) tablet.cc:1381] finished ApplyRowOperation cycle
0130 02:58:19.648778 (+ 125us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0130 02:58:19.648820 (+ 42us) write_op.cc:312] APPLY: finished
0130 02:58:19.649007 (+ 187us) log.cc:844] Serialized 23 byte log entry
0130 02:58:19.649153 (+ 146us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:58:19.649389 (+ 236us) write_op.cc:454] Released schema lock
0130 02:58:19.649437 (+ 48us) write_op.cc:341] FINISH: Updating metrics
0130 02:58:19.649685 (+ 248us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:58:19.649709 (+ 24us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":175,"apply.run_cpu_time_us":1434,"apply.run_wall_time_us":1434,"num_ops":1,"prepare.queue_time_us":101762,"prepare.run_cpu_time_us":1837,"prepare.run_wall_time_us":1838,"raft.queue_time_us":1236,"raft.run_cpu_time_us":736,"raft.run_wall_time_us":736,"replication_time_us":15936}]]}
I20260130 02:58:22.256467 18871 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50284:
table { table_name: "default.Table_d11b53ac041740d8818edf7d2861dc6d" } modify_external_catalogs: true
I20260130 02:58:22.257853 18871 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50284:
table { table_name: "default.Table_d11b53ac041740d8818edf7d2861dc6d" } modify_external_catalogs: true
I20260130 02:58:22.175462 21026 leader_election.cc:290] T 49efcfced0b74c48962393149ac2a500 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 election: Requested vote from peers d870164263bb44b6b780a557daea3656 (127.18.100.67:39571), 371a36d219164974abdf848841528152 (127.18.100.65:42021)
I20260130 02:58:22.268728 19807 tablet_bootstrap.cc:654] T 2b0edaad6877460cb7982efc62b00b66 P 371a36d219164974abdf848841528152: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:22.171849 19625 ts_tablet_manager.cc:1403] T 1fc8f09d6329407c8078542b4123ef4d P d870164263bb44b6b780a557daea3656: Time spent bootstrapping tablet: real 0.425s user 0.013s sys 0.001s
I20260130 02:58:22.273512 19375 raft_consensus.cc:2468] T e34d5da0ad864b2eaea0d575c982e447 P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d870164263bb44b6b780a557daea3656 in term 1.
I20260130 02:58:22.276938 19372 raft_consensus.cc:2468] T 2b0edaad6877460cb7982efc62b00b66 P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d870164263bb44b6b780a557daea3656 in term 1.
I20260130 02:58:22.285894 20984 raft_consensus.cc:493] T 0d14fb95ace9447f810ccbed4d5befdf P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:22.288077 20984 raft_consensus.cc:515] T 0d14fb95ace9447f810ccbed4d5befdf P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:22.289884 20984 leader_election.cc:290] T 0d14fb95ace9447f810ccbed4d5befdf P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d870164263bb44b6b780a557daea3656 (127.18.100.67:39571), 371a36d219164974abdf848841528152 (127.18.100.65:42021)
I20260130 02:58:22.262149 19358 tablet_service.cc:1552] Processing DeleteTablet for tablet e12d130bcdf34965888753f12c1ff848 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:58:18 UTC) from {username='slave'} at 127.0.0.1:60110
I20260130 02:58:22.297134 20901 raft_consensus.cc:493] T 00000000000000000000000000000000 P 3749baa287ed4f249329a74071781edd [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 658299197a424f53a37f14fadf089ae4)
I20260130 02:58:22.297865 20901 raft_consensus.cc:515] T 00000000000000000000000000000000 P 3749baa287ed4f249329a74071781edd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "658299197a424f53a37f14fadf089ae4" member_type: VOTER last_known_addr { host: "127.18.100.126" port: 38191 } } peers { permanent_uuid: "107f274cdda543038832f06a56ab9130" member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39711 } } peers { permanent_uuid: "3749baa287ed4f249329a74071781edd" member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34629 } }
I20260130 02:58:22.307510 20872 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3749baa287ed4f249329a74071781edd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 658299197a424f53a37f14fadf089ae4. Latest consensus state: current_term: 2 leader_uuid: "658299197a424f53a37f14fadf089ae4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "658299197a424f53a37f14fadf089ae4" member_type: VOTER last_known_addr { host: "127.18.100.126" port: 38191 } } peers { permanent_uuid: "107f274cdda543038832f06a56ab9130" member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39711 } } peers { permanent_uuid: "3749baa287ed4f249329a74071781edd" member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34629 } } }
I20260130 02:58:22.308250 20872 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3749baa287ed4f249329a74071781edd [sys.catalog]: This master's current role is: FOLLOWER
I20260130 02:58:22.258836 20988 raft_consensus.cc:515] T aae68390dbbe4dc58ce3d3bd01725ae6 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } }
I20260130 02:58:22.313841 21078 tablet_replica.cc:333] T e12d130bcdf34965888753f12c1ff848 P 371a36d219164974abdf848841528152: stopping tablet replica
I20260130 02:58:22.320660 19840 tablet_bootstrap.cc:492] T 6d1f121f49cd47c2b8247daaf6874e01 P 78c2c46c7520492db35d51c244fb4f74: No bootstrap required, opened a new log
I20260130 02:58:22.321097 19840 ts_tablet_manager.cc:1403] T 6d1f121f49cd47c2b8247daaf6874e01 P 78c2c46c7520492db35d51c244fb4f74: Time spent bootstrapping tablet: real 0.459s user 0.012s sys 0.008s
I20260130 02:58:22.321415 21078 raft_consensus.cc:2243] T e12d130bcdf34965888753f12c1ff848 P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260130 02:58:22.321995 21078 raft_consensus.cc:2272] T e12d130bcdf34965888753f12c1ff848 P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260130 02:58:22.323500 19221 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0d14fb95ace9447f810ccbed4d5befdf" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152" is_pre_election: true
I20260130 02:58:22.324502 19199 tablet_service.cc:1552] Processing DeleteTablet for tablet e2aaaf00f53d418b87d5c7af286e4d84 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:58:08 UTC) from {username='slave'} at 127.0.0.1:42870
I20260130 02:58:22.324985 21078 ts_tablet_manager.cc:1916] T e12d130bcdf34965888753f12c1ff848 P 371a36d219164974abdf848841528152: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260130 02:58:22.325503 19197 tablet_service.cc:1552] Processing DeleteTablet for tablet 3389db58493448aea3cf00aca411047c with delete_type TABLET_DATA_DELETED (replaced by tablet af0f90071a334016aa17fae2e3a97fc8) from {username='slave'} at 127.0.0.1:42870
W20260130 02:58:22.326141 18873 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.18.100.65:51021 (request call id 38) took 5452 ms (client timeout 1000 ms). Trace:
I20260130 02:58:22.327353 19199 tablet_service.cc:1552] Processing DeleteTablet for tablet ec28ff12927d450ead16d26cf016138d with delete_type TABLET_DATA_DELETED (replaced by tablet 6d1f121f49cd47c2b8247daaf6874e01) from {username='slave'} at 127.0.0.1:42870
I20260130 02:58:22.327095 19202 tablet_service.cc:1552] Processing DeleteTablet for tablet 78e9bc3e55fe4e4a98e8eb01da59d365 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:58:08 UTC) from {username='slave'} at 127.0.0.1:42870
I20260130 02:58:22.326445 19197 tablet_service.cc:1552] Processing DeleteTablet for tablet c7c621b99deb457998c44ab1644d3939 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:58:16 UTC) from {username='slave'} at 127.0.0.1:42870
W20260130 02:58:22.329421 19314 leader_election.cc:343] T 0d14fb95ace9447f810ccbed4d5befdf P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:22.346395 20985 consensus_queue.cc:1048] T 12c3f3ec523040e7b7cd356e3da57e0d P 371a36d219164974abdf848841528152 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260130 02:58:22.327855 18861 catalog_manager.cc:5058] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021): alter failed for tablet a8a5038e9def44eb8b3b1e4aeae25652 (table default.Table_dee034aec0b24982b8dc2a3fa1b326f1 [id=66033e75b3814a8584311cf7e4a07ad3]),no further retry: Not found: Tablet not found: a8a5038e9def44eb8b3b1e4aeae25652
W20260130 02:58:22.345927 18875 catalog_manager.cc:5846] Failed to send AlterTable RPC for tablet 657e9b0d8c05413f904129dce745710b (table default.Table_526677c8ef4144bd9de7a54a5e78e2be [id=b16908b75587459597a978c71756451c], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet 657e9b0d8c05413f904129dce745710b consensus state has no leader
W20260130 02:58:22.350850 18875 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.18.100.66:59691 (request call id 36) took 4155 ms (client timeout 1000 ms). Trace:
I20260130 02:58:22.353134 20988 leader_election.cc:290] T aae68390dbbe4dc58ce3d3bd01725ae6 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d870164263bb44b6b780a557daea3656 (127.18.100.67:39571), 371a36d219164974abdf848841528152 (127.18.100.65:42021)
W20260130 02:58:22.351147 18875 rpcz_store.cc:269] 0130 02:58:18.195437 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:58:18.722906 (+527469us) service_pool.cc:225] Handling call
0130 02:58:22.350792 (+3627886us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0130 02:58:22.211124 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0130 02:58:22.211563 (+ 439us) write_op.cc:432] Acquiring schema lock in shared mode
0130 02:58:22.211591 (+ 28us) write_op.cc:435] Acquired schema lock
0130 02:58:22.211605 (+ 14us) tablet.cc:660] Decoding operations
0130 02:58:22.211855 (+ 250us) write_op.cc:620] Acquiring the partition lock for write op
0130 02:58:22.211898 (+ 43us) write_op.cc:641] Partition lock acquired for write op
0130 02:58:22.211922 (+ 24us) tablet.cc:683] Acquiring locks for 1 operations
0130 02:58:22.212057 (+ 135us) tablet.cc:699] Row locks acquired
0130 02:58:22.212075 (+ 18us) write_op.cc:260] PREPARE: finished
0130 02:58:22.212192 (+ 117us) write_op.cc:270] Start()
0130 02:58:22.212285 (+ 93us) write_op.cc:276] Timestamp: P: 1769741902212104 usec, L: 0
0130 02:58:22.212313 (+ 28us) op_driver.cc:348] REPLICATION: starting
0130 02:58:22.212833 (+ 520us) log.cc:844] Serialized 356 byte log entry
0130 02:58:22.230135 (+ 17302us) op_driver.cc:464] REPLICATION: finished
0130 02:58:22.230360 (+ 225us) write_op.cc:301] APPLY: starting
0130 02:58:22.230426 (+ 66us) tablet.cc:1365] starting BulkCheckPresence
0130 02:58:22.230521 (+ 95us) tablet.cc:1368] finished BulkCheckPresence
0130 02:58:22.230639 (+ 118us) tablet.cc:1370] starting ApplyRowOperation cycle
0130 02:58:22.230836 (+ 197us) tablet.cc:1381] finished ApplyRowOperation cycle
0130 02:58:22.230976 (+ 140us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0130 02:58:22.231014 (+ 38us) write_op.cc:312] APPLY: finished
0130 02:58:22.231178 (+ 164us) log.cc:844] Serialized 23 byte log entry
0130 02:58:22.231306 (+ 128us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:58:22.231363 (+ 57us) write_op.cc:454] Released schema lock
0130 02:58:22.231397 (+ 34us) write_op.cc:341] FINISH: Updating metrics
0130 02:58:22.231644 (+ 247us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:58:22.231666 (+ 22us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":56,"reported_tablets":72,"child_traces":[["op",{"apply.queue_time_us":76,"apply.run_cpu_time_us":1252,"apply.run_wall_time_us":1251,"num_ops":1,"prepare.queue_time_us":109,"prepare.run_cpu_time_us":2022,"prepare.run_wall_time_us":2021,"raft.queue_time_us":89,"raft.run_cpu_time_us":682,"raft.run_wall_time_us":682,"replication_time_us":17707,"spinlock_wait_cycles":178176}]]}
I20260130 02:58:22.360580 19840 raft_consensus.cc:359] T 6d1f121f49cd47c2b8247daaf6874e01 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:22.361202 19840 raft_consensus.cc:385] T 6d1f121f49cd47c2b8247daaf6874e01 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:22.361455 19840 raft_consensus.cc:740] T 6d1f121f49cd47c2b8247daaf6874e01 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78c2c46c7520492db35d51c244fb4f74, State: Initialized, Role: FOLLOWER
I20260130 02:58:22.354420 18875 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50284:
name: "default.Table_d2a3438ed66741eab245ae6d6c5c2f37"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260130 02:58:22.364198 18875 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d2a3438ed66741eab245ae6d6c5c2f37 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260130 02:58:22.362016 19840 consensus_queue.cc:260] T 6d1f121f49cd47c2b8247daaf6874e01 P 78c2c46c7520492db35d51c244fb4f74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:22.366161 19840 ts_tablet_manager.cc:1434] T 6d1f121f49cd47c2b8247daaf6874e01 P 78c2c46c7520492db35d51c244fb4f74: Time spent starting tablet: real 0.045s user 0.003s sys 0.001s
W20260130 02:58:22.326483 18873 rpcz_store.cc:269] 0130 02:58:16.873933 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:58:16.928575 (+ 54642us) service_pool.cc:225] Handling call
0130 02:58:22.326063 (+5397488us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0130 02:58:18.657376 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0130 02:58:18.657828 (+ 452us) write_op.cc:432] Acquiring schema lock in shared mode
0130 02:58:18.657854 (+ 26us) write_op.cc:435] Acquired schema lock
0130 02:58:18.657870 (+ 16us) tablet.cc:660] Decoding operations
0130 02:58:18.658382 (+ 512us) write_op.cc:620] Acquiring the partition lock for write op
0130 02:58:18.658430 (+ 48us) write_op.cc:641] Partition lock acquired for write op
0130 02:58:18.658462 (+ 32us) tablet.cc:683] Acquiring locks for 4 operations
0130 02:58:18.658834 (+ 372us) tablet.cc:699] Row locks acquired
0130 02:58:18.658851 (+ 17us) write_op.cc:260] PREPARE: finished
0130 02:58:18.658966 (+ 115us) write_op.cc:270] Start()
0130 02:58:18.659055 (+ 89us) write_op.cc:276] Timestamp: P: 1769741898658886 usec, L: 0
0130 02:58:18.659079 (+ 24us) op_driver.cc:348] REPLICATION: starting
0130 02:58:18.659656 (+ 577us) log.cc:844] Serialized 1674 byte log entry
0130 02:58:18.671629 (+ 11973us) op_driver.cc:464] REPLICATION: finished
0130 02:58:18.671932 (+ 303us) write_op.cc:301] APPLY: starting
0130 02:58:18.671996 (+ 64us) tablet.cc:1365] starting BulkCheckPresence
0130 02:58:18.672128 (+ 132us) tablet.cc:1368] finished BulkCheckPresence
0130 02:58:18.672229 (+ 101us) tablet.cc:1370] starting ApplyRowOperation cycle
0130 02:58:18.672636 (+ 407us) tablet.cc:1381] finished ApplyRowOperation cycle
0130 02:58:18.672783 (+ 147us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=4
0130 02:58:18.672826 (+ 43us) write_op.cc:312] APPLY: finished
0130 02:58:18.673031 (+ 205us) log.cc:844] Serialized 41 byte log entry
0130 02:58:18.673165 (+ 134us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:58:18.673264 (+ 99us) write_op.cc:454] Released schema lock
0130 02:58:18.673298 (+ 34us) write_op.cc:341] FINISH: Updating metrics
0130 02:58:18.673506 (+ 208us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:58:18.673534 (+ 28us) write_op.cc:454] Released schema lock
Related trace 'op':
0130 02:58:18.891457 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0130 02:58:18.891940 (+ 483us) write_op.cc:432] Acquiring schema lock in shared mode
0130 02:58:18.891971 (+ 31us) write_op.cc:435] Acquired schema lock
0130 02:58:18.891988 (+ 17us) tablet.cc:660] Decoding operations
0130 02:58:18.892300 (+ 312us) write_op.cc:620] Acquiring the partition lock for write op
0130 02:58:18.892347 (+ 47us) write_op.cc:641] Partition lock acquired for write op
0130 02:58:18.892373 (+ 26us) tablet.cc:683] Acquiring locks for 1 operations
0130 02:58:18.892539 (+ 166us) tablet.cc:699] Row locks acquired
0130 02:58:18.892556 (+ 17us) write_op.cc:260] PREPARE: finished
0130 02:58:18.892687 (+ 131us) write_op.cc:270] Start()
0130 02:58:18.892777 (+ 90us) write_op.cc:276] Timestamp: P: 1769741898892627 usec, L: 0
0130 02:58:18.892796 (+ 19us) op_driver.cc:348] REPLICATION: starting
0130 02:58:18.893535 (+ 739us) log.cc:844] Serialized 356 byte log entry
0130 02:58:18.901885 (+ 8350us) op_driver.cc:464] REPLICATION: finished
0130 02:58:18.902184 (+ 299us) write_op.cc:301] APPLY: starting
0130 02:58:18.902238 (+ 54us) tablet.cc:1365] starting BulkCheckPresence
0130 02:58:18.902326 (+ 88us) tablet.cc:1368] finished BulkCheckPresence
0130 02:58:18.902411 (+ 85us) tablet.cc:1370] starting ApplyRowOperation cycle
0130 02:58:18.902595 (+ 184us) tablet.cc:1381] finished ApplyRowOperation cycle
0130 02:58:18.902742 (+ 147us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0130 02:58:18.902783 (+ 41us) write_op.cc:312] APPLY: finished
0130 02:58:18.902946 (+ 163us) log.cc:844] Serialized 23 byte log entry
0130 02:58:18.903082 (+ 136us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:58:18.903146 (+ 64us) write_op.cc:454] Released schema lock
0130 02:58:18.903177 (+ 31us) write_op.cc:341] FINISH: Updating metrics
0130 02:58:18.903385 (+ 208us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:58:18.903409 (+ 24us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":143699,"reported_tablets":53,"child_traces":[["op",{"apply.queue_time_us":167,"apply.run_cpu_time_us":1533,"apply.run_wall_time_us":1533,"num_ops":4,"prepare.queue_time_us":779,"prepare.run_cpu_time_us":2576,"prepare.run_wall_time_us":2577,"raft.queue_time_us":1315,"raft.run_cpu_time_us":622,"raft.run_wall_time_us":622,"replication_time_us":12459}],["op",{"apply.queue_time_us":166,"apply.run_cpu_time_us":1166,"apply.run_wall_time_us":1166,"mutex_wait_us":90246,"num_ops":1,"prepare.queue_time_us":90371,"prepare.run_cpu_time_us":2433,"prepare.run_wall_time_us":2434,"raft.queue_time_us":1110,"raft.run_cpu_time_us":613,"raft.run_wall_time_us":613,"replication_time_us":8991}]]}
I20260130 02:58:22.370998 20899 ts_tablet_manager.cc:1929] T a8a5038e9def44eb8b3b1e4aeae25652 P 78c2c46c7520492db35d51c244fb4f74: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260130 02:58:22.371439 20899 log.cc:1199] T a8a5038e9def44eb8b3b1e4aeae25652 P 78c2c46c7520492db35d51c244fb4f74: Deleting WAL directory at /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-1/wal/wals/a8a5038e9def44eb8b3b1e4aeae25652
I20260130 02:58:22.372220 20899 ts_tablet_manager.cc:1950] T a8a5038e9def44eb8b3b1e4aeae25652 P 78c2c46c7520492db35d51c244fb4f74: Deleting consensus metadata
I20260130 02:58:22.372175 21078 ts_tablet_manager.cc:1929] T e12d130bcdf34965888753f12c1ff848 P 371a36d219164974abdf848841528152: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260130 02:58:22.373980 18873 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50284:
name: "default.Table_ba19837ced4c45d79be506e93308449b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260130 02:58:22.382977 18873 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ba19837ced4c45d79be506e93308449b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260130 02:58:22.399852 21078 log.cc:1199] T e12d130bcdf34965888753f12c1ff848 P 371a36d219164974abdf848841528152: Deleting WAL directory at /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-0/wal/wals/e12d130bcdf34965888753f12c1ff848
I20260130 02:58:22.401295 21078 ts_tablet_manager.cc:1950] T e12d130bcdf34965888753f12c1ff848 P 371a36d219164974abdf848841528152: Deleting consensus metadata
I20260130 02:58:22.404520 19222 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "aae68390dbbe4dc58ce3d3bd01725ae6" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152" is_pre_election: true
I20260130 02:58:22.405053 19222 raft_consensus.cc:2468] T aae68390dbbe4dc58ce3d3bd01725ae6 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78c2c46c7520492db35d51c244fb4f74 in term 0.
W20260130 02:58:22.407618 18859 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet 657e9b0d8c05413f904129dce745710b (table default.Table_526677c8ef4144bd9de7a54a5e78e2be [id=b16908b75587459597a978c71756451c], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 657e9b0d8c05413f904129dce745710b consensus state has no leader
I20260130 02:58:22.408751 18861 catalog_manager.cc:4985] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021): tablet e12d130bcdf34965888753f12c1ff848 (table default.Table_a675d7371dcd46b293f893ebe4939e7a [id=c7084428dde74c3ebc46943c44211d7b]) successfully deleted
I20260130 02:58:22.404724 21078 tablet_replica.cc:333] T e2aaaf00f53d418b87d5c7af286e4d84 P 371a36d219164974abdf848841528152: stopping tablet replica
I20260130 02:58:22.408998 19314 leader_election.cc:304] T aae68390dbbe4dc58ce3d3bd01725ae6 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 371a36d219164974abdf848841528152, 78c2c46c7520492db35d51c244fb4f74; no voters:
I20260130 02:58:22.409739 20704 raft_consensus.cc:2804] T aae68390dbbe4dc58ce3d3bd01725ae6 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260130 02:58:22.409772 21078 raft_consensus.cc:2243] T e2aaaf00f53d418b87d5c7af286e4d84 P 371a36d219164974abdf848841528152 [term 1 LEADER]: Raft consensus shutting down.
I20260130 02:58:22.410059 20704 raft_consensus.cc:493] T aae68390dbbe4dc58ce3d3bd01725ae6 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260130 02:58:22.410379 20704 raft_consensus.cc:3060] T aae68390dbbe4dc58ce3d3bd01725ae6 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:22.408834 18875 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:50284 (request call id 264) took 2377 ms. Trace:
I20260130 02:58:22.415370 18875 rpcz_store.cc:276] 0130 02:58:20.031377 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:58:22.353969 (+2322592us) service_pool.cc:225] Handling call
0130 02:58:22.364601 (+ 10632us) catalog_manager.cc:2101] Acquired catalog manager lock
0130 02:58:22.366078 (+ 1477us) catalog_manager.cc:2147] Created new table and tablet info
0130 02:58:22.397966 (+ 31888us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:58:22.397992 (+ 26us) write_op.cc:454] Released schema lock
0130 02:58:22.399457 (+ 1465us) catalog_manager.cc:2207] Wrote table and tablets to system table
0130 02:58:22.408511 (+ 9054us) catalog_manager.cc:2241] Inserted table and tablets into CatalogManager maps
0130 02:58:22.408780 (+ 269us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0130 02:58:22.372710 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0130 02:58:22.373198 (+ 488us) write_op.cc:432] Acquiring schema lock in shared mode
0130 02:58:22.373223 (+ 25us) write_op.cc:435] Acquired schema lock
0130 02:58:22.373237 (+ 14us) tablet.cc:660] Decoding operations
0130 02:58:22.373570 (+ 333us) write_op.cc:620] Acquiring the partition lock for write op
0130 02:58:22.373618 (+ 48us) write_op.cc:641] Partition lock acquired for write op
0130 02:58:22.373653 (+ 35us) tablet.cc:683] Acquiring locks for 2 operations
0130 02:58:22.373859 (+ 206us) tablet.cc:699] Row locks acquired
0130 02:58:22.373875 (+ 16us) write_op.cc:260] PREPARE: finished
0130 02:58:22.373994 (+ 119us) write_op.cc:270] Start()
0130 02:58:22.374086 (+ 92us) write_op.cc:276] Timestamp: P: 1769741902373950 usec, L: 0
0130 02:58:22.374105 (+ 19us) op_driver.cc:348] REPLICATION: starting
0130 02:58:22.375300 (+ 1195us) log.cc:844] Serialized 433 byte log entry
0130 02:58:22.390703 (+ 15403us) op_driver.cc:464] REPLICATION: finished
0130 02:58:22.391034 (+ 331us) write_op.cc:301] APPLY: starting
0130 02:58:22.391085 (+ 51us) tablet.cc:1365] starting BulkCheckPresence
0130 02:58:22.391186 (+ 101us) tablet.cc:1368] finished BulkCheckPresence
0130 02:58:22.391277 (+ 91us) tablet.cc:1370] starting ApplyRowOperation cycle
0130 02:58:22.391731 (+ 454us) tablet.cc:1381] finished ApplyRowOperation cycle
0130 02:58:22.391874 (+ 143us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0130 02:58:22.391920 (+ 46us) write_op.cc:312] APPLY: finished
0130 02:58:22.392114 (+ 194us) log.cc:844] Serialized 29 byte log entry
0130 02:58:22.392251 (+ 137us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:58:22.392329 (+ 78us) write_op.cc:454] Released schema lock
0130 02:58:22.392364 (+ 35us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":198,"apply.run_cpu_time_us":1498,"apply.run_wall_time_us":1499,"num_ops":2,"prepare.queue_time_us":1773,"prepare.run_cpu_time_us":2304,"prepare.run_wall_time_us":2925,"raft.queue_time_us":867,"raft.run_cpu_time_us":697,"raft.run_wall_time_us":697,"replication_time_us":16489}]]}
I20260130 02:58:22.419212 21078 raft_consensus.cc:2272] T e2aaaf00f53d418b87d5c7af286e4d84 P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260130 02:58:22.421290 21078 ts_tablet_manager.cc:1916] T e2aaaf00f53d418b87d5c7af286e4d84 P 371a36d219164974abdf848841528152: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260130 02:58:22.432626 19463 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.100.65:42021, user_credentials={real_user=slave}} blocked reactor thread for 452841us
W20260130 02:58:22.433643 19463 leader_election.cc:343] T eb7a755ab62b4552bb8588264c5f7b75 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:22.451076 20901 leader_election.cc:290] T 00000000000000000000000000000000 P 3749baa287ed4f249329a74071781edd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 658299197a424f53a37f14fadf089ae4 (127.18.100.126:38191), 107f274cdda543038832f06a56ab9130 (127.18.100.125:39711)
W20260130 02:58:22.461023 19314 consensus_peers.cc:597] T e12d130bcdf34965888753f12c1ff848 P 78c2c46c7520492db35d51c244fb4f74 -> Peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Couldn't send request to peer 371a36d219164974abdf848841528152. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e12d130bcdf34965888753f12c1ff848. This is attempt 1: this message will repeat every 5th retry.
I20260130 02:58:22.470690 18875 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50284:
name: "default.Table_bc168ff8b5f34cacbd398117647ba25a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260130 02:58:22.472218 18875 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bc168ff8b5f34cacbd398117647ba25a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260130 02:58:22.463826 20920 consensus_queue.cc:1048] T f7607df5efd543438fb2dff3667cc8c8 P 371a36d219164974abdf848841528152 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260130 02:58:22.473770 20985 raft_consensus.cc:493] T e34d5da0ad864b2eaea0d575c982e447 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:22.491216 20985 raft_consensus.cc:515] T e34d5da0ad864b2eaea0d575c982e447 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:22.493093 20985 leader_election.cc:290] T e34d5da0ad864b2eaea0d575c982e447 P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d870164263bb44b6b780a557daea3656 (127.18.100.67:39571), 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767)
I20260130 02:58:22.502863 19840 tablet_bootstrap.cc:492] T 1129f72158ec4bae89da3c94cdb70839 P 78c2c46c7520492db35d51c244fb4f74: Bootstrap starting.
I20260130 02:58:22.434544 19463 leader_election.cc:304] T dfbbe9045e7c4aa5b520218687e7e4c1 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 371a36d219164974abdf848841528152, d870164263bb44b6b780a557daea3656; no voters:
I20260130 02:58:22.508670 20899 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60110 (request call id 319) took 2473 ms. Trace:
I20260130 02:58:22.509013 20899 rpcz_store.cc:276] 0130 02:58:20.035059 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:58:20.035178 (+ 119us) service_pool.cc:225] Handling call
0130 02:58:21.961772 (+1926594us) ts_tablet_manager.cc:1151] Deleting tablet a8a5038e9def44eb8b3b1e4aeae25652
0130 02:58:21.961822 (+ 50us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0130 02:58:22.355264 (+393442us) tablet_metadata.cc:641] Metadata flushed
0130 02:58:22.370765 (+ 15501us) tablet_metadata.cc:641] Metadata flushed
0130 02:58:22.508600 (+137835us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1926057}
I20260130 02:58:22.510145 20899 tablet_replica.cc:333] T e12d130bcdf34965888753f12c1ff848 P 78c2c46c7520492db35d51c244fb4f74: stopping tablet replica
I20260130 02:58:22.448649 19625 raft_consensus.cc:359] T 1fc8f09d6329407c8078542b4123ef4d P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:22.512943 19625 raft_consensus.cc:385] T 1fc8f09d6329407c8078542b4123ef4d P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:22.513242 19625 raft_consensus.cc:740] T 1fc8f09d6329407c8078542b4123ef4d P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d870164263bb44b6b780a557daea3656, State: Initialized, Role: FOLLOWER
I20260130 02:58:22.513629 19840 tablet_bootstrap.cc:654] T 1129f72158ec4bae89da3c94cdb70839 P 78c2c46c7520492db35d51c244fb4f74: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:22.513890 19625 consensus_queue.cc:260] T 1fc8f09d6329407c8078542b4123ef4d P d870164263bb44b6b780a557daea3656 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:22.441576 20998 raft_consensus.cc:493] T 0855d88806914a36b4bf6b49036f0ad0 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:22.517246 20998 raft_consensus.cc:515] T 0855d88806914a36b4bf6b49036f0ad0 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:22.519544 20998 leader_election.cc:290] T 0855d88806914a36b4bf6b49036f0ad0 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767)
I20260130 02:58:22.521658 20899 raft_consensus.cc:2243] T e12d130bcdf34965888753f12c1ff848 P 78c2c46c7520492db35d51c244fb4f74 [term 1 LEADER]: Raft consensus shutting down.
I20260130 02:58:22.522713 20899 raft_consensus.cc:2272] T e12d130bcdf34965888753f12c1ff848 P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260130 02:58:22.524582 20899 ts_tablet_manager.cc:1916] T e12d130bcdf34965888753f12c1ff848 P 78c2c46c7520492db35d51c244fb4f74: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260130 02:58:22.526877 20985 raft_consensus.cc:493] T 657e9b0d8c05413f904129dce745710b P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:22.527335 20985 raft_consensus.cc:515] T 657e9b0d8c05413f904129dce745710b P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:22.527529 19522 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9896ff9364744857968e87e80e776dd2" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
I20260130 02:58:22.531544 20985 leader_election.cc:290] T 657e9b0d8c05413f904129dce745710b P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767), d870164263bb44b6b780a557daea3656 (127.18.100.67:39571)
I20260130 02:58:22.533151 19528 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "49efcfced0b74c48962393149ac2a500" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656"
I20260130 02:58:22.533252 19524 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00ddbfe6218c44d788fbf65245a31dc6" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
I20260130 02:58:22.533885 19528 raft_consensus.cc:3060] T 49efcfced0b74c48962393149ac2a500 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:22.534917 19520 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0d14fb95ace9447f810ccbed4d5befdf" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
I20260130 02:58:22.539120 19525 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ab6afa40408a4f458495abf20290e55b" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
I20260130 02:58:22.539695 19525 raft_consensus.cc:2468] T ab6afa40408a4f458495abf20290e55b P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78c2c46c7520492db35d51c244fb4f74 in term 0.
W20260130 02:58:22.539889 19463 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.100.65:42021, user_credentials={real_user=slave}} blocked reactor thread for 105497us
I20260130 02:58:22.540889 21056 raft_consensus.cc:493] T aae68390dbbe4dc58ce3d3bd01725ae6 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:22.543341 19222 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ab6afa40408a4f458495abf20290e55b" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152" is_pre_election: true
I20260130 02:58:22.545496 19371 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "657e9b0d8c05413f904129dce745710b" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74" is_pre_election: true
I20260130 02:58:22.546232 19371 raft_consensus.cc:2393] T 657e9b0d8c05413f904129dce745710b P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 371a36d219164974abdf848841528152 in current term 1: Already voted for candidate d870164263bb44b6b780a557daea3656 in this term.
I20260130 02:58:22.546546 19222 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ca06f07bb98f45a7bf63aebc28348e08" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152"
I20260130 02:58:22.548118 19365 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ab6afa40408a4f458495abf20290e55b" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74" is_pre_election: true
I20260130 02:58:22.548635 19365 raft_consensus.cc:2468] T ab6afa40408a4f458495abf20290e55b P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d870164263bb44b6b780a557daea3656 in term 0.
I20260130 02:58:22.548913 19370 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dfbbe9045e7c4aa5b520218687e7e4c1" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74"
I20260130 02:58:22.549386 19370 raft_consensus.cc:3060] T dfbbe9045e7c4aa5b520218687e7e4c1 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:22.549458 19369 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "eb7a755ab62b4552bb8588264c5f7b75" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74" is_pre_election: true
I20260130 02:58:22.549975 19369 raft_consensus.cc:2468] T eb7a755ab62b4552bb8588264c5f7b75 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d870164263bb44b6b780a557daea3656 in term 0.
I20260130 02:58:22.550405 19364 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ca06f07bb98f45a7bf63aebc28348e08" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74"
I20260130 02:58:22.550905 19364 raft_consensus.cc:3060] T ca06f07bb98f45a7bf63aebc28348e08 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:22.541358 21056 raft_consensus.cc:515] T aae68390dbbe4dc58ce3d3bd01725ae6 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } }
I20260130 02:58:22.552528 19375 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e34d5da0ad864b2eaea0d575c982e447" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74" is_pre_election: true
I20260130 02:58:22.553187 19375 raft_consensus.cc:2393] T e34d5da0ad864b2eaea0d575c982e447 P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 371a36d219164974abdf848841528152 in current term 1: Already voted for candidate d870164263bb44b6b780a557daea3656 in this term.
I20260130 02:58:22.553124 21056 leader_election.cc:290] T aae68390dbbe4dc58ce3d3bd01725ae6 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767), 371a36d219164974abdf848841528152 (127.18.100.65:42021)
I20260130 02:58:22.557370 19378 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0855d88806914a36b4bf6b49036f0ad0" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74" is_pre_election: true
I20260130 02:58:22.557854 19378 raft_consensus.cc:2468] T 0855d88806914a36b4bf6b49036f0ad0 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d870164263bb44b6b780a557daea3656 in term 0.
I20260130 02:58:22.563724 19625 ts_tablet_manager.cc:1434] T 1fc8f09d6329407c8078542b4123ef4d P d870164263bb44b6b780a557daea3656: Time spent starting tablet: real 0.290s user 0.008s sys 0.002s
I20260130 02:58:22.565210 19625 tablet_bootstrap.cc:492] T 306452e208ca4e719609f67345114a13 P d870164263bb44b6b780a557daea3656: Bootstrap starting.
I20260130 02:58:22.567605 21049 raft_consensus.cc:2804] T dfbbe9045e7c4aa5b520218687e7e4c1 P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Leader election won for term 1
I20260130 02:58:22.568053 21049 raft_consensus.cc:697] T dfbbe9045e7c4aa5b520218687e7e4c1 P d870164263bb44b6b780a557daea3656 [term 1 LEADER]: Becoming Leader. State: Replica: d870164263bb44b6b780a557daea3656, State: Running, Role: LEADER
I20260130 02:58:22.568840 21049 consensus_queue.cc:237] T dfbbe9045e7c4aa5b520218687e7e4c1 P d870164263bb44b6b780a557daea3656 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:22.571492 19520 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "aae68390dbbe4dc58ce3d3bd01725ae6" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
I20260130 02:58:22.572055 19520 raft_consensus.cc:2468] T aae68390dbbe4dc58ce3d3bd01725ae6 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78c2c46c7520492db35d51c244fb4f74 in term 0.
I20260130 02:58:22.575021 19222 raft_consensus.cc:3060] T ca06f07bb98f45a7bf63aebc28348e08 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:22.586467 20704 raft_consensus.cc:515] T aae68390dbbe4dc58ce3d3bd01725ae6 P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } }
I20260130 02:58:22.587533 19462 leader_election.cc:304] T 657e9b0d8c05413f904129dce745710b P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 78c2c46c7520492db35d51c244fb4f74, d870164263bb44b6b780a557daea3656; no voters: 371a36d219164974abdf848841528152
I20260130 02:58:22.588702 20704 leader_election.cc:290] T aae68390dbbe4dc58ce3d3bd01725ae6 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 election: Requested vote from peers d870164263bb44b6b780a557daea3656 (127.18.100.67:39571), 371a36d219164974abdf848841528152 (127.18.100.65:42021)
I20260130 02:58:22.589115 19462 leader_election.cc:304] T e34d5da0ad864b2eaea0d575c982e447 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 78c2c46c7520492db35d51c244fb4f74, d870164263bb44b6b780a557daea3656; no voters: 371a36d219164974abdf848841528152
I20260130 02:58:22.589354 21060 raft_consensus.cc:2804] T 657e9b0d8c05413f904129dce745710b P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Leader election won for term 1
I20260130 02:58:22.589766 21060 raft_consensus.cc:697] T 657e9b0d8c05413f904129dce745710b P d870164263bb44b6b780a557daea3656 [term 1 LEADER]: Becoming Leader. State: Replica: d870164263bb44b6b780a557daea3656, State: Running, Role: LEADER
I20260130 02:58:22.590075 19462 leader_election.cc:304] T 2b0edaad6877460cb7982efc62b00b66 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 78c2c46c7520492db35d51c244fb4f74, d870164263bb44b6b780a557daea3656; no voters: 371a36d219164974abdf848841528152
I20260130 02:58:22.595405 20956 raft_consensus.cc:2804] T e34d5da0ad864b2eaea0d575c982e447 P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Leader election won for term 1
I20260130 02:58:22.596454 19370 raft_consensus.cc:2468] T dfbbe9045e7c4aa5b520218687e7e4c1 P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d870164263bb44b6b780a557daea3656 in term 1.
I20260130 02:58:22.596589 21057 raft_consensus.cc:2804] T 2b0edaad6877460cb7982efc62b00b66 P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Leader election won for term 1
I20260130 02:58:22.590490 21060 consensus_queue.cc:237] T 657e9b0d8c05413f904129dce745710b P d870164263bb44b6b780a557daea3656 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:22.599480 21057 raft_consensus.cc:697] T 2b0edaad6877460cb7982efc62b00b66 P d870164263bb44b6b780a557daea3656 [term 1 LEADER]: Becoming Leader. State: Replica: d870164263bb44b6b780a557daea3656, State: Running, Role: LEADER
I20260130 02:58:22.600335 21057 consensus_queue.cc:237] T 2b0edaad6877460cb7982efc62b00b66 P d870164263bb44b6b780a557daea3656 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:22.602970 19364 raft_consensus.cc:2468] T ca06f07bb98f45a7bf63aebc28348e08 P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d870164263bb44b6b780a557daea3656 in term 1.
I20260130 02:58:22.604276 21080 catalog_manager.cc:805] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260130 02:58:22.594466 21000 consensus_queue.cc:1048] T 2c79a3aecab844698e70834999e1caac P d870164263bb44b6b780a557daea3656 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.012s
I20260130 02:58:22.636245 20704 raft_consensus.cc:493] T dd377fab8c544bf69121dc9e9a060b29 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:22.636731 20704 raft_consensus.cc:515] T dd377fab8c544bf69121dc9e9a060b29 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
W20260130 02:58:22.648059 19159 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.100.66:40767, user_credentials={real_user=slave}} blocked reactor thread for 104648us
I20260130 02:58:22.653043 21078 ts_tablet_manager.cc:1929] T e2aaaf00f53d418b87d5c7af286e4d84 P 371a36d219164974abdf848841528152: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260130 02:58:22.653482 21078 log.cc:1199] T e2aaaf00f53d418b87d5c7af286e4d84 P 371a36d219164974abdf848841528152: Deleting WAL directory at /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-0/wal/wals/e2aaaf00f53d418b87d5c7af286e4d84
I20260130 02:58:22.653777 19225 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0855d88806914a36b4bf6b49036f0ad0" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152" is_pre_election: true
I20260130 02:58:22.658847 19222 raft_consensus.cc:2468] T ca06f07bb98f45a7bf63aebc28348e08 P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d870164263bb44b6b780a557daea3656 in term 1.
I20260130 02:58:22.661394 21078 ts_tablet_manager.cc:1950] T e2aaaf00f53d418b87d5c7af286e4d84 P 371a36d219164974abdf848841528152: Deleting consensus metadata
I20260130 02:58:22.662140 19358 tablet_service.cc:1552] Processing DeleteTablet for tablet 78e9bc3e55fe4e4a98e8eb01da59d365 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:58:08 UTC) from {username='slave'} at 127.0.0.1:60110
I20260130 02:58:22.663316 21078 tablet_replica.cc:333] T 78e9bc3e55fe4e4a98e8eb01da59d365 P 371a36d219164974abdf848841528152: stopping tablet replica
I20260130 02:58:22.663385 19358 tablet_service.cc:1552] Processing DeleteTablet for tablet 04f4ac27800c4ac3a79ae12d35479757 with delete_type TABLET_DATA_DELETED (replaced by tablet 3738385130c044e0a5a5803da74c567d) from {username='slave'} at 127.0.0.1:60110
I20260130 02:58:22.664882 19358 tablet_service.cc:1552] Processing DeleteTablet for tablet 8e57ae31ab3941f789ac69627384e3b0 with delete_type TABLET_DATA_DELETED (replaced by tablet 8b47653c7c3945a68e9bf4680bfc0496) from {username='slave'} at 127.0.0.1:60110
I20260130 02:58:22.665869 19358 tablet_service.cc:1552] Processing DeleteTablet for tablet e2aaaf00f53d418b87d5c7af286e4d84 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:58:08 UTC) from {username='slave'} at 127.0.0.1:60110
I20260130 02:58:22.666771 19358 tablet_service.cc:1552] Processing DeleteTablet for tablet 2c79a3aecab844698e70834999e1caac with delete_type TABLET_DATA_DELETED (replaced by tablet 94355fc1338f479e8798070aff04fcba) from {username='slave'} at 127.0.0.1:60110
I20260130 02:58:22.666805 20985 raft_consensus.cc:493] T ca06f07bb98f45a7bf63aebc28348e08 P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:22.667241 20985 raft_consensus.cc:515] T ca06f07bb98f45a7bf63aebc28348e08 P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:22.670496 20985 leader_election.cc:290] T ca06f07bb98f45a7bf63aebc28348e08 P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d870164263bb44b6b780a557daea3656 (127.18.100.67:39571), 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767)
I20260130 02:58:22.674000 18861 catalog_manager.cc:4985] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021): tablet e2aaaf00f53d418b87d5c7af286e4d84 (table default.Table_e84c6277dfc944059f62cbe0d6b937ee [id=747af828147149efb7b7b8d33c49286d]) successfully deleted
W20260130 02:58:22.674652 18861 catalog_manager.cc:4952] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021): delete failed for tablet 3389db58493448aea3cf00aca411047c because the tablet was not found. No further retry: Not found: Tablet not found: 3389db58493448aea3cf00aca411047c
W20260130 02:58:22.675153 18861 catalog_manager.cc:4952] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021): delete failed for tablet ec28ff12927d450ead16d26cf016138d because the tablet was not found. No further retry: Not found: Tablet not found: ec28ff12927d450ead16d26cf016138d
I20260130 02:58:22.680909 18860 catalog_manager.cc:4985] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767): tablet a8a5038e9def44eb8b3b1e4aeae25652 (table default.Table_dee034aec0b24982b8dc2a3fa1b326f1 [id=66033e75b3814a8584311cf7e4a07ad3]) successfully deleted
I20260130 02:58:22.687135 19840 tablet_bootstrap.cc:492] T 1129f72158ec4bae89da3c94cdb70839 P 78c2c46c7520492db35d51c244fb4f74: No bootstrap required, opened a new log
I20260130 02:58:22.687589 19840 ts_tablet_manager.cc:1403] T 1129f72158ec4bae89da3c94cdb70839 P 78c2c46c7520492db35d51c244fb4f74: Time spent bootstrapping tablet: real 0.185s user 0.012s sys 0.000s
I20260130 02:58:22.690225 19840 raft_consensus.cc:359] T 1129f72158ec4bae89da3c94cdb70839 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:22.695772 20704 leader_election.cc:290] T dd377fab8c544bf69121dc9e9a060b29 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d870164263bb44b6b780a557daea3656 (127.18.100.67:39571), 371a36d219164974abdf848841528152 (127.18.100.65:42021)
I20260130 02:58:22.706440 19807 tablet_bootstrap.cc:492] T 2b0edaad6877460cb7982efc62b00b66 P 371a36d219164974abdf848841528152: No bootstrap required, opened a new log
I20260130 02:58:22.706872 19807 ts_tablet_manager.cc:1403] T 2b0edaad6877460cb7982efc62b00b66 P 371a36d219164974abdf848841528152: Time spent bootstrapping tablet: real 0.471s user 0.034s sys 0.008s
I20260130 02:58:22.709076 19807 raft_consensus.cc:359] T 2b0edaad6877460cb7982efc62b00b66 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:22.709702 19807 raft_consensus.cc:385] T 2b0edaad6877460cb7982efc62b00b66 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:22.710057 19807 raft_consensus.cc:740] T 2b0edaad6877460cb7982efc62b00b66 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 371a36d219164974abdf848841528152, State: Initialized, Role: FOLLOWER
I20260130 02:58:22.710886 19225 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "aae68390dbbe4dc58ce3d3bd01725ae6" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152"
I20260130 02:58:22.710599 19807 consensus_queue.cc:260] T 2b0edaad6877460cb7982efc62b00b66 P 371a36d219164974abdf848841528152 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:22.711364 19225 raft_consensus.cc:3060] T aae68390dbbe4dc58ce3d3bd01725ae6 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:22.712236 19807 ts_tablet_manager.cc:1434] T 2b0edaad6877460cb7982efc62b00b66 P 371a36d219164974abdf848841528152: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260130 02:58:22.715270 18960 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3749baa287ed4f249329a74071781edd" candidate_term: 3 candidate_status { last_received { term: 2 index: 333 } } ignore_live_leader: false dest_uuid: "107f274cdda543038832f06a56ab9130" is_pre_election: true
I20260130 02:58:22.717355 19225 raft_consensus.cc:2468] T aae68390dbbe4dc58ce3d3bd01725ae6 P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78c2c46c7520492db35d51c244fb4f74 in term 1.
I20260130 02:58:22.721434 19221 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dd377fab8c544bf69121dc9e9a060b29" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152" is_pre_election: true
I20260130 02:58:22.723548 19314 leader_election.cc:304] T aae68390dbbe4dc58ce3d3bd01725ae6 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 371a36d219164974abdf848841528152, 78c2c46c7520492db35d51c244fb4f74; no voters:
I20260130 02:58:22.724350 20704 raft_consensus.cc:2804] T aae68390dbbe4dc58ce3d3bd01725ae6 P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Leader election won for term 1
I20260130 02:58:22.724800 20704 raft_consensus.cc:697] T aae68390dbbe4dc58ce3d3bd01725ae6 P 78c2c46c7520492db35d51c244fb4f74 [term 1 LEADER]: Becoming Leader. State: Replica: 78c2c46c7520492db35d51c244fb4f74, State: Running, Role: LEADER
I20260130 02:58:22.725518 20704 consensus_queue.cc:237] T aae68390dbbe4dc58ce3d3bd01725ae6 P 78c2c46c7520492db35d51c244fb4f74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } }
I20260130 02:58:22.610899 19528 raft_consensus.cc:2468] T 49efcfced0b74c48962393149ac2a500 P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78c2c46c7520492db35d51c244fb4f74 in term 1.
I20260130 02:58:22.664198 19505 tablet_service.cc:1552] Processing DeleteTablet for tablet e12d130bcdf34965888753f12c1ff848 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:58:18 UTC) from {username='slave'} at 127.0.0.1:59068
I20260130 02:58:22.737138 21078 raft_consensus.cc:2243] T 78e9bc3e55fe4e4a98e8eb01da59d365 P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260130 02:58:22.737699 21078 raft_consensus.cc:2272] T 78e9bc3e55fe4e4a98e8eb01da59d365 P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260130 02:58:22.739266 19364 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ca06f07bb98f45a7bf63aebc28348e08" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74" is_pre_election: true
I20260130 02:58:22.739812 21078 ts_tablet_manager.cc:1916] T 78e9bc3e55fe4e4a98e8eb01da59d365 P 371a36d219164974abdf848841528152: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260130 02:58:22.739886 19364 raft_consensus.cc:2468] T ca06f07bb98f45a7bf63aebc28348e08 P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 371a36d219164974abdf848841528152 in term 1.
W20260130 02:58:22.748947 19314 leader_election.cc:343] T dd377fab8c544bf69121dc9e9a060b29 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:22.751731 21102 tablet.cc:1722] T ba91a508b1cb4430a4d8ca26946aecc8 P 371a36d219164974abdf848841528152: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260130 02:58:22.752756 20920 raft_consensus.cc:493] T d480bf18140245e3870a62582d5a8318 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:22.753201 20920 raft_consensus.cc:515] T d480bf18140245e3870a62582d5a8318 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } }
I20260130 02:58:22.767453 19807 tablet_bootstrap.cc:492] T ab6afa40408a4f458495abf20290e55b P 371a36d219164974abdf848841528152: Bootstrap starting.
I20260130 02:58:22.770676 19358 tablet_service.cc:1552] Processing DeleteTablet for tablet c7c621b99deb457998c44ab1644d3939 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:58:16 UTC) from {username='slave'} at 127.0.0.1:60110
I20260130 02:58:22.770542 20956 raft_consensus.cc:697] T e34d5da0ad864b2eaea0d575c982e447 P d870164263bb44b6b780a557daea3656 [term 1 LEADER]: Becoming Leader. State: Replica: d870164263bb44b6b780a557daea3656, State: Running, Role: LEADER
I20260130 02:58:22.786816 18866 catalog_manager.cc:5746] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767) does not have the latest schema for tablet 345eaaa5625c48d099280b7fbb2e514b (table default.Table_c1886dd9a7934e7797e0e7e0b065d692 [id=7a2d5367224b492b844199088b01b149]). Expected version 1 got 0
I20260130 02:58:22.800019 20956 consensus_queue.cc:237] T e34d5da0ad864b2eaea0d575c982e447 P d870164263bb44b6b780a557daea3656 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
W20260130 02:58:22.801956 19462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.100.66:40767, user_credentials={real_user=slave}} blocked reactor thread for 211932us
I20260130 02:58:22.802742 19462 leader_election.cc:304] T ab6afa40408a4f458495abf20290e55b P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78c2c46c7520492db35d51c244fb4f74, d870164263bb44b6b780a557daea3656; no voters:
I20260130 02:58:22.812230 20782 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "371a36d219164974abdf848841528152" instance_seqno: 1769741868776403) as {username='slave'} at 127.18.100.65:59271; Asking this server to re-register.
I20260130 02:58:22.817070 19625 tablet_bootstrap.cc:654] T 306452e208ca4e719609f67345114a13 P d870164263bb44b6b780a557daea3656: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:22.824331 18873 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:50284 (request call id 265) took 2757 ms. Trace:
W20260130 02:58:22.826731 18860 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.100.125:39711, user_credentials={real_user=slave}} blocked reactor thread for 145160us
I20260130 02:58:22.824698 18873 rpcz_store.cc:276] 0130 02:58:20.066977 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:58:22.373604 (+2306627us) service_pool.cc:225] Handling call
0130 02:58:22.408094 (+ 34490us) catalog_manager.cc:2101] Acquired catalog manager lock
0130 02:58:22.411600 (+ 3506us) catalog_manager.cc:2147] Created new table and tablet info
0130 02:58:22.823713 (+412113us) catalog_manager.cc:2207] Wrote table and tablets to system table
0130 02:58:22.824005 (+ 292us) catalog_manager.cc:2241] Inserted table and tablets into CatalogManager maps
0130 02:58:22.824259 (+ 254us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0130 02:58:22.416277 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0130 02:58:22.416750 (+ 473us) write_op.cc:432] Acquiring schema lock in shared mode
0130 02:58:22.416779 (+ 29us) write_op.cc:435] Acquired schema lock
0130 02:58:22.416795 (+ 16us) tablet.cc:660] Decoding operations
0130 02:58:22.417100 (+ 305us) write_op.cc:620] Acquiring the partition lock for write op
0130 02:58:22.417148 (+ 48us) write_op.cc:641] Partition lock acquired for write op
0130 02:58:22.417176 (+ 28us) tablet.cc:683] Acquiring locks for 2 operations
0130 02:58:22.417438 (+ 262us) tablet.cc:699] Row locks acquired
0130 02:58:22.417455 (+ 17us) write_op.cc:260] PREPARE: finished
0130 02:58:22.417579 (+ 124us) write_op.cc:270] Start()
0130 02:58:22.417669 (+ 90us) write_op.cc:276] Timestamp: P: 1769741902417531 usec, L: 0
0130 02:58:22.417688 (+ 19us) op_driver.cc:348] REPLICATION: starting
0130 02:58:22.418242 (+ 554us) log.cc:844] Serialized 433 byte log entry
0130 02:58:22.821536 (+403294us) op_driver.cc:464] REPLICATION: finished
0130 02:58:22.821850 (+ 314us) write_op.cc:301] APPLY: starting
0130 02:58:22.821914 (+ 64us) tablet.cc:1365] starting BulkCheckPresence
0130 02:58:22.822027 (+ 113us) tablet.cc:1368] finished BulkCheckPresence
0130 02:58:22.822128 (+ 101us) tablet.cc:1370] starting ApplyRowOperation cycle
0130 02:58:22.822590 (+ 462us) tablet.cc:1381] finished ApplyRowOperation cycle
0130 02:58:22.822740 (+ 150us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0130 02:58:22.822779 (+ 39us) write_op.cc:312] APPLY: finished
0130 02:58:22.822981 (+ 202us) log.cc:844] Serialized 29 byte log entry
0130 02:58:22.823135 (+ 154us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:58:22.823195 (+ 60us) write_op.cc:454] Released schema lock
0130 02:58:22.823226 (+ 31us) write_op.cc:341] FINISH: Updating metrics
0130 02:58:22.823433 (+ 207us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:58:22.823455 (+ 22us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":164,"apply.run_cpu_time_us":1551,"apply.run_wall_time_us":1549,"num_ops":2,"prepare.queue_time_us":91,"prepare.run_cpu_time_us":2291,"prepare.run_wall_time_us":2289,"raft.queue_time_us":882,"raft.run_cpu_time_us":612,"raft.run_wall_time_us":612,"replication_time_us":403740}]]}
I20260130 02:58:22.828440 18873 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50284:
name: "default.Table_203b3a960d3e4b5cb978b6168435f3a4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260130 02:58:22.829877 18873 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_203b3a960d3e4b5cb978b6168435f3a4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260130 02:58:22.690881 19840 raft_consensus.cc:385] T 1129f72158ec4bae89da3c94cdb70839 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:22.837502 19462 leader_election.cc:304] T eb7a755ab62b4552bb8588264c5f7b75 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 78c2c46c7520492db35d51c244fb4f74, d870164263bb44b6b780a557daea3656; no voters: 371a36d219164974abdf848841528152
I20260130 02:58:22.837642 19840 raft_consensus.cc:740] T 1129f72158ec4bae89da3c94cdb70839 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78c2c46c7520492db35d51c244fb4f74, State: Initialized, Role: FOLLOWER
I20260130 02:58:22.841768 19575 heartbeater.cc:344] Connected to a master server at 127.18.100.126:38191
I20260130 02:58:22.838647 21057 raft_consensus.cc:2804] T ab6afa40408a4f458495abf20290e55b P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260130 02:58:22.849148 21057 raft_consensus.cc:493] T ab6afa40408a4f458495abf20290e55b P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260130 02:58:22.849473 21057 raft_consensus.cc:3060] T ab6afa40408a4f458495abf20290e55b P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:22.840348 21105 tablet.cc:1722] T ba91a508b1cb4430a4d8ca26946aecc8 P 78c2c46c7520492db35d51c244fb4f74: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260130 02:58:22.851955 20920 leader_election.cc:290] T d480bf18140245e3870a62582d5a8318 P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767), d870164263bb44b6b780a557daea3656 (127.18.100.67:39571)
I20260130 02:58:22.855437 18875 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:50284 (request call id 266) took 2756 ms. Trace:
I20260130 02:58:22.855755 18875 rpcz_store.cc:276] 0130 02:58:20.098988 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:58:22.418544 (+2319556us) service_pool.cc:225] Handling call
0130 02:58:22.472605 (+ 54061us) catalog_manager.cc:2101] Acquired catalog manager lock
0130 02:58:22.504207 (+ 31602us) spinlock_profiling.cc:243] Waited 26.9 ms on lock 0x7b0c000346a8. stack: 00007fb0bd3f7373 00007fb0bd3f7270 00007fb0bcf6c574 0000000000508744 00000000005086d9 00007fb0bd2698b1 00007fb0bd376b0f 00007fb0bd376949 00007fb0c61c33af 00007fb0c61c1061 00007fb0c61be737 00007fb0c61c223e 00007fb0c62e268f 00007fb0c3f05524 00007fb0c3f054b2 00007fb0c3f053e1
0130 02:58:22.504948 (+ 741us) catalog_manager.cc:2147] Created new table and tablet info
0130 02:58:22.854858 (+349910us) catalog_manager.cc:2207] Wrote table and tablets to system table
0130 02:58:22.855137 (+ 279us) catalog_manager.cc:2241] Inserted table and tablets into CatalogManager maps
0130 02:58:22.855383 (+ 246us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0130 02:58:22.508221 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0130 02:58:22.508663 (+ 442us) write_op.cc:432] Acquiring schema lock in shared mode
0130 02:58:22.508690 (+ 27us) write_op.cc:435] Acquired schema lock
0130 02:58:22.508707 (+ 17us) tablet.cc:660] Decoding operations
0130 02:58:22.509003 (+ 296us) write_op.cc:620] Acquiring the partition lock for write op
0130 02:58:22.509039 (+ 36us) write_op.cc:641] Partition lock acquired for write op
0130 02:58:22.509063 (+ 24us) tablet.cc:683] Acquiring locks for 2 operations
0130 02:58:22.509272 (+ 209us) tablet.cc:699] Row locks acquired
0130 02:58:22.509287 (+ 15us) write_op.cc:260] PREPARE: finished
0130 02:58:22.509406 (+ 119us) write_op.cc:270] Start()
0130 02:58:22.509488 (+ 82us) write_op.cc:276] Timestamp: P: 1769741902509349 usec, L: 0
0130 02:58:22.509505 (+ 17us) op_driver.cc:348] REPLICATION: starting
0130 02:58:22.510013 (+ 508us) log.cc:844] Serialized 433 byte log entry
0130 02:58:22.852069 (+342056us) op_driver.cc:464] REPLICATION: finished
0130 02:58:22.853082 (+ 1013us) write_op.cc:301] APPLY: starting
0130 02:58:22.853144 (+ 62us) tablet.cc:1365] starting BulkCheckPresence
0130 02:58:22.853254 (+ 110us) tablet.cc:1368] finished BulkCheckPresence
0130 02:58:22.853354 (+ 100us) tablet.cc:1370] starting ApplyRowOperation cycle
0130 02:58:22.853742 (+ 388us) tablet.cc:1381] finished ApplyRowOperation cycle
0130 02:58:22.853872 (+ 130us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0130 02:58:22.853910 (+ 38us) write_op.cc:312] APPLY: finished
0130 02:58:22.854093 (+ 183us) log.cc:844] Serialized 29 byte log entry
0130 02:58:22.854232 (+ 139us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:58:22.854298 (+ 66us) write_op.cc:454] Released schema lock
0130 02:58:22.854331 (+ 33us) write_op.cc:341] FINISH: Updating metrics
0130 02:58:22.854528 (+ 197us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:58:22.854549 (+ 21us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":61758720,"child_traces":[["op",{"apply.queue_time_us":847,"apply.run_cpu_time_us":1431,"apply.run_wall_time_us":1428,"num_ops":2,"prepare.queue_time_us":2097,"prepare.run_cpu_time_us":2016,"prepare.run_wall_time_us":2017,"replication_time_us":342467}]]}
I20260130 02:58:22.858217 19521 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "133e9bfe175f44aab1a53114abe67e4d" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
I20260130 02:58:22.860436 18875 master_service.cc:954] ReplaceTablet: received request to replace tablet 43cec19c51ec4f05b63800c07f9534ef from {username='slave'} at 127.0.0.1:50284
I20260130 02:58:22.863690 21095 raft_consensus.cc:493] T 2b0edaad6877460cb7982efc62b00b66 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:22.864573 21049 raft_consensus.cc:2804] T eb7a755ab62b4552bb8588264c5f7b75 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260130 02:58:22.864943 21049 raft_consensus.cc:493] T eb7a755ab62b4552bb8588264c5f7b75 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260130 02:58:22.865270 21049 raft_consensus.cc:3060] T eb7a755ab62b4552bb8588264c5f7b75 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Advancing to term 1
W20260130 02:58:22.866339 19159 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.100.66:40767, user_credentials={real_user=slave}} blocked reactor thread for 63109.9us
I20260130 02:58:22.867069 19159 leader_election.cc:304] T ca06f07bb98f45a7bf63aebc28348e08 P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 371a36d219164974abdf848841528152, 78c2c46c7520492db35d51c244fb4f74; no voters:
I20260130 02:58:22.867542 18866 catalog_manager.cc:5746] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767) does not have the latest schema for tablet b1e8b78eff0d4a0b92059adb2de4a3ce (table default.Table_d0b4516d24384e5386c1debaf2543adb [id=f1342f9497b74e7a9ecad1de8082578d]). Expected version 1 got 0
W20260130 02:58:22.870525 19313 leader_election.cc:343] T 9896ff9364744857968e87e80e776dd2 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:22.870997 19313 leader_election.cc:304] T 9896ff9364744857968e87e80e776dd2 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78c2c46c7520492db35d51c244fb4f74; no voters: 371a36d219164974abdf848841528152, d870164263bb44b6b780a557daea3656
I20260130 02:58:22.872165 21081 raft_consensus.cc:2749] T 9896ff9364744857968e87e80e776dd2 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260130 02:58:22.878432 21078 ts_tablet_manager.cc:1929] T 78e9bc3e55fe4e4a98e8eb01da59d365 P 371a36d219164974abdf848841528152: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260130 02:58:22.878943 21078 log.cc:1199] T 78e9bc3e55fe4e4a98e8eb01da59d365 P 371a36d219164974abdf848841528152: Deleting WAL directory at /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-0/wal/wals/78e9bc3e55fe4e4a98e8eb01da59d365
I20260130 02:58:22.879752 21078 ts_tablet_manager.cc:1950] T 78e9bc3e55fe4e4a98e8eb01da59d365 P 371a36d219164974abdf848841528152: Deleting consensus metadata
W20260130 02:58:22.879688 19313 leader_election.cc:343] T 00ddbfe6218c44d788fbf65245a31dc6 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:22.880077 19313 leader_election.cc:304] T 00ddbfe6218c44d788fbf65245a31dc6 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78c2c46c7520492db35d51c244fb4f74; no voters: 371a36d219164974abdf848841528152, d870164263bb44b6b780a557daea3656
I20260130 02:58:22.880764 20704 raft_consensus.cc:2749] T 00ddbfe6218c44d788fbf65245a31dc6 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260130 02:58:22.881088 18861 catalog_manager.cc:4985] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021): tablet 78e9bc3e55fe4e4a98e8eb01da59d365 (table default.Table_5d29239ce1984d2fad02607a2bdd4dd0 [id=7e271b81bb064a949685b2411b485e3e]) successfully deleted
W20260130 02:58:22.881791 19313 leader_election.cc:343] T 0d14fb95ace9447f810ccbed4d5befdf P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:22.864813 21095 raft_consensus.cc:515] T 2b0edaad6877460cb7982efc62b00b66 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:22.882143 19313 leader_election.cc:304] T 0d14fb95ace9447f810ccbed4d5befdf P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78c2c46c7520492db35d51c244fb4f74; no voters: 371a36d219164974abdf848841528152, d870164263bb44b6b780a557daea3656
I20260130 02:58:22.882833 20704 raft_consensus.cc:2749] T 0d14fb95ace9447f810ccbed4d5befdf P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260130 02:58:22.883721 21095 leader_election.cc:290] T 2b0edaad6877460cb7982efc62b00b66 P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d870164263bb44b6b780a557daea3656 (127.18.100.67:39571), 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767)
W20260130 02:58:22.890738 18861 catalog_manager.cc:4952] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021): delete failed for tablet c7c621b99deb457998c44ab1644d3939 because the tablet was not found. No further retry: Not found: Tablet not found: c7c621b99deb457998c44ab1644d3939
I20260130 02:58:22.890901 19313 leader_election.cc:304] T ab6afa40408a4f458495abf20290e55b P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 78c2c46c7520492db35d51c244fb4f74, d870164263bb44b6b780a557daea3656; no voters: 371a36d219164974abdf848841528152
I20260130 02:58:22.891952 21081 raft_consensus.cc:2804] T ab6afa40408a4f458495abf20290e55b P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260130 02:58:22.892282 21081 raft_consensus.cc:493] T ab6afa40408a4f458495abf20290e55b P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260130 02:58:22.892596 21081 raft_consensus.cc:3060] T ab6afa40408a4f458495abf20290e55b P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:22.894695 19807 tablet_bootstrap.cc:654] T ab6afa40408a4f458495abf20290e55b P 371a36d219164974abdf848841528152: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:22.899312 21081 raft_consensus.cc:515] T ab6afa40408a4f458495abf20290e55b P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
W20260130 02:58:22.900092 19462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.100.66:40767, user_credentials={real_user=slave}} blocked reactor thread for 82034.5us
I20260130 02:58:22.901383 21081 leader_election.cc:290] T ab6afa40408a4f458495abf20290e55b P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 election: Requested vote from peers d870164263bb44b6b780a557daea3656 (127.18.100.67:39571), 371a36d219164974abdf848841528152 (127.18.100.65:42021)
I20260130 02:58:22.902446 19222 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ab6afa40408a4f458495abf20290e55b" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152"
W20260130 02:58:22.911865 19314 leader_election.cc:343] T ab6afa40408a4f458495abf20290e55b P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:22.911633 19840 consensus_queue.cc:260] T 1129f72158ec4bae89da3c94cdb70839 P 78c2c46c7520492db35d51c244fb4f74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:22.907423 21095 raft_consensus.cc:2804] T ca06f07bb98f45a7bf63aebc28348e08 P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260130 02:58:22.912554 21095 raft_consensus.cc:493] T ca06f07bb98f45a7bf63aebc28348e08 P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260130 02:58:22.912956 21095 raft_consensus.cc:3060] T ca06f07bb98f45a7bf63aebc28348e08 P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Advancing to term 2
I20260130 02:58:22.913424 19840 ts_tablet_manager.cc:1434] T 1129f72158ec4bae89da3c94cdb70839 P 78c2c46c7520492db35d51c244fb4f74: Time spent starting tablet: real 0.226s user 0.006s sys 0.000s
I20260130 02:58:22.917264 20899 ts_tablet_manager.cc:1929] T e12d130bcdf34965888753f12c1ff848 P 78c2c46c7520492db35d51c244fb4f74: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260130 02:58:22.917630 20899 log.cc:1199] T e12d130bcdf34965888753f12c1ff848 P 78c2c46c7520492db35d51c244fb4f74: Deleting WAL directory at /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-1/wal/wals/e12d130bcdf34965888753f12c1ff848
I20260130 02:58:22.918262 20899 ts_tablet_manager.cc:1950] T e12d130bcdf34965888753f12c1ff848 P 78c2c46c7520492db35d51c244fb4f74: Deleting consensus metadata
I20260130 02:58:22.931273 18873 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:50284 (request call id 267) took 2793 ms. Trace:
I20260130 02:58:22.931803 18873 rpcz_store.cc:276] 0130 02:58:20.137519 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:58:22.828064 (+2690545us) service_pool.cc:225] Handling call
0130 02:58:22.830228 (+ 2164us) catalog_manager.cc:2101] Acquired catalog manager lock
0130 02:58:22.833461 (+ 3233us) catalog_manager.cc:2147] Created new table and tablet info
0130 02:58:22.930702 (+ 97241us) catalog_manager.cc:2207] Wrote table and tablets to system table
0130 02:58:22.930969 (+ 267us) catalog_manager.cc:2241] Inserted table and tablets into CatalogManager maps
0130 02:58:22.931217 (+ 248us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0130 02:58:22.859663 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0130 02:58:22.860110 (+ 447us) write_op.cc:432] Acquiring schema lock in shared mode
0130 02:58:22.860136 (+ 26us) write_op.cc:435] Acquired schema lock
0130 02:58:22.860147 (+ 11us) tablet.cc:660] Decoding operations
0130 02:58:22.860466 (+ 319us) write_op.cc:620] Acquiring the partition lock for write op
0130 02:58:22.860514 (+ 48us) write_op.cc:641] Partition lock acquired for write op
0130 02:58:22.860538 (+ 24us) tablet.cc:683] Acquiring locks for 2 operations
0130 02:58:22.860754 (+ 216us) tablet.cc:699] Row locks acquired
0130 02:58:22.860770 (+ 16us) write_op.cc:260] PREPARE: finished
0130 02:58:22.860880 (+ 110us) write_op.cc:270] Start()
0130 02:58:22.860962 (+ 82us) write_op.cc:276] Timestamp: P: 1769741902860788 usec, L: 0
0130 02:58:22.860980 (+ 18us) op_driver.cc:348] REPLICATION: starting
0130 02:58:22.861474 (+ 494us) log.cc:844] Serialized 433 byte log entry
0130 02:58:22.925675 (+ 64201us) op_driver.cc:464] REPLICATION: finished
0130 02:58:22.925961 (+ 286us) write_op.cc:301] APPLY: starting
0130 02:58:22.926015 (+ 54us) tablet.cc:1365] starting BulkCheckPresence
0130 02:58:22.926117 (+ 102us) tablet.cc:1368] finished BulkCheckPresence
0130 02:58:22.926209 (+ 92us) tablet.cc:1370] starting ApplyRowOperation cycle
0130 02:58:22.926669 (+ 460us) tablet.cc:1381] finished ApplyRowOperation cycle
0130 02:58:22.926802 (+ 133us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0130 02:58:22.926844 (+ 42us) write_op.cc:312] APPLY: finished
0130 02:58:22.927023 (+ 179us) log.cc:844] Serialized 29 byte log entry
0130 02:58:22.927183 (+ 160us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:58:22.927246 (+ 63us) write_op.cc:454] Released schema lock
0130 02:58:22.927276 (+ 30us) write_op.cc:341] FINISH: Updating metrics
0130 02:58:22.927476 (+ 200us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:58:22.927499 (+ 23us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":157,"apply.run_cpu_time_us":1481,"apply.run_wall_time_us":1480,"num_ops":2,"prepare.queue_time_us":25081,"prepare.run_cpu_time_us":2019,"prepare.run_wall_time_us":2019,"replication_time_us":64597}]]}
I20260130 02:58:22.934662 19522 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dd377fab8c544bf69121dc9e9a060b29" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
I20260130 02:58:22.940800 19222 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "aae68390dbbe4dc58ce3d3bd01725ae6" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152" is_pre_election: true
I20260130 02:58:22.941442 19222 raft_consensus.cc:2393] T aae68390dbbe4dc58ce3d3bd01725ae6 P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d870164263bb44b6b780a557daea3656 in current term 1: Already voted for candidate 78c2c46c7520492db35d51c244fb4f74 in this term.
I20260130 02:58:22.943181 19462 leader_election.cc:304] T 0855d88806914a36b4bf6b49036f0ad0 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78c2c46c7520492db35d51c244fb4f74, d870164263bb44b6b780a557daea3656; no voters:
I20260130 02:58:22.945719 19364 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d480bf18140245e3870a62582d5a8318" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74" is_pre_election: true
I20260130 02:58:22.953765 21056 raft_consensus.cc:2804] T 0855d88806914a36b4bf6b49036f0ad0 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260130 02:58:22.936182 20777 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" instance_seqno: 1769741870705948) as {username='slave'} at 127.18.100.66:39475; Asking this server to re-register.
I20260130 02:58:22.954133 21056 raft_consensus.cc:493] T 0855d88806914a36b4bf6b49036f0ad0 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260130 02:58:22.954466 21056 raft_consensus.cc:3060] T 0855d88806914a36b4bf6b49036f0ad0 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:22.956382 19528 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ca06f07bb98f45a7bf63aebc28348e08" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
I20260130 02:58:22.962845 18873 master_service.cc:954] ReplaceTablet: received request to replace tablet ba91a508b1cb4430a4d8ca26946aecc8 from {username='slave'} at 127.0.0.1:50284
I20260130 02:58:22.948141 18866 catalog_manager.cc:5746] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767) does not have the latest schema for tablet 2b0edaad6877460cb7982efc62b00b66 (table default.Table_d95381c590c94802bdc83acef0f0df96 [id=64a8e0e079f34806aa68795f3d548c5e]). Expected version 1 got 0
I20260130 02:58:22.964051 19425 heartbeater.cc:461] Registering TS with master...
I20260130 02:58:22.965250 18866 catalog_manager.cc:5746] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767) does not have the latest schema for tablet 9fe0448bcc4a4346b3c67fc3f626ed3c (table default.Table_5c1c447e85524ac5a61ffd8a48509d36 [id=70aaf21ad9bd4d959f46939c6b4a4880]). Expected version 1 got 0
I20260130 02:58:22.965963 21103 tablet_replica.cc:333] T e12d130bcdf34965888753f12c1ff848 P d870164263bb44b6b780a557daea3656: stopping tablet replica
I20260130 02:58:22.967077 21103 raft_consensus.cc:2243] T e12d130bcdf34965888753f12c1ff848 P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260130 02:58:22.967634 19625 tablet_bootstrap.cc:492] T 306452e208ca4e719609f67345114a13 P d870164263bb44b6b780a557daea3656: No bootstrap required, opened a new log
I20260130 02:58:22.967785 19528 raft_consensus.cc:2468] T ca06f07bb98f45a7bf63aebc28348e08 P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 371a36d219164974abdf848841528152 in term 1.
I20260130 02:58:22.968030 19625 ts_tablet_manager.cc:1403] T 306452e208ca4e719609f67345114a13 P d870164263bb44b6b780a557daea3656: Time spent bootstrapping tablet: real 0.403s user 0.015s sys 0.011s
I20260130 02:58:22.975044 21095 raft_consensus.cc:515] T ca06f07bb98f45a7bf63aebc28348e08 P 371a36d219164974abdf848841528152 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:22.976562 21081 raft_consensus.cc:493] T cfd12a72726846e587cff9d6a4d61bd6 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:22.977031 21081 raft_consensus.cc:515] T cfd12a72726846e587cff9d6a4d61bd6 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:22.979507 19222 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cfd12a72726846e587cff9d6a4d61bd6" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152" is_pre_election: true
W20260130 02:58:22.980911 19314 leader_election.cc:343] T cfd12a72726846e587cff9d6a4d61bd6 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:22.982084 19222 raft_consensus.cc:3060] T d480bf18140245e3870a62582d5a8318 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:22.989883 21021 consensus_queue.cc:1048] T d480bf18140245e3870a62582d5a8318 P 78c2c46c7520492db35d51c244fb4f74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260130 02:58:22.990993 21081 leader_election.cc:290] T cfd12a72726846e587cff9d6a4d61bd6 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), d870164263bb44b6b780a557daea3656 (127.18.100.67:39571)
I20260130 02:58:22.993985 19374 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2b0edaad6877460cb7982efc62b00b66" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74" is_pre_election: true
I20260130 02:58:22.994750 19374 raft_consensus.cc:2393] T 2b0edaad6877460cb7982efc62b00b66 P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 371a36d219164974abdf848841528152 in current term 1: Already voted for candidate d870164263bb44b6b780a557daea3656 in this term.
I20260130 02:58:23.000658 19462 leader_election.cc:304] T ca06f07bb98f45a7bf63aebc28348e08 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78c2c46c7520492db35d51c244fb4f74, d870164263bb44b6b780a557daea3656; no voters:
I20260130 02:58:23.004204 19807 tablet_bootstrap.cc:492] T ab6afa40408a4f458495abf20290e55b P 371a36d219164974abdf848841528152: No bootstrap required, opened a new log
I20260130 02:58:23.004626 19807 ts_tablet_manager.cc:1403] T ab6afa40408a4f458495abf20290e55b P 371a36d219164974abdf848841528152: Time spent bootstrapping tablet: real 0.237s user 0.008s sys 0.001s
I20260130 02:58:22.968798 19524 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "aae68390dbbe4dc58ce3d3bd01725ae6" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656"
I20260130 02:58:23.007854 19625 raft_consensus.cc:359] T 306452e208ca4e719609f67345114a13 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:23.008484 19524 raft_consensus.cc:3060] T aae68390dbbe4dc58ce3d3bd01725ae6 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:22.955951 21049 raft_consensus.cc:515] T eb7a755ab62b4552bb8588264c5f7b75 P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:23.010249 21049 leader_election.cc:290] T eb7a755ab62b4552bb8588264c5f7b75 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 election: Requested vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767)
I20260130 02:58:22.967657 21103 raft_consensus.cc:2272] T e12d130bcdf34965888753f12c1ff848 P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260130 02:58:23.005998 21057 raft_consensus.cc:515] T ab6afa40408a4f458495abf20290e55b P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:23.008531 19625 raft_consensus.cc:385] T 306452e208ca4e719609f67345114a13 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:23.011430 19625 raft_consensus.cc:740] T 306452e208ca4e719609f67345114a13 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d870164263bb44b6b780a557daea3656, State: Initialized, Role: FOLLOWER
I20260130 02:58:22.972606 18885 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3749baa287ed4f249329a74071781edd" candidate_term: 3 candidate_status { last_received { term: 2 index: 333 } } ignore_live_leader: false dest_uuid: "658299197a424f53a37f14fadf089ae4" is_pre_election: true
I20260130 02:58:23.014283 18866 catalog_manager.cc:5746] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767) does not have the latest schema for tablet 274530359b174ff79f046c2d4a6afa76 (table default.Table_c4f4904b6fac4b9880979cf0314e71c1 [id=c456c6418c93499a9d08f652720854ca]). Expected version 1 got 0
I20260130 02:58:23.015686 21095 leader_election.cc:290] T ca06f07bb98f45a7bf63aebc28348e08 P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 2 election: Requested vote from peers d870164263bb44b6b780a557daea3656 (127.18.100.67:39571), 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767)
I20260130 02:58:23.016924 18866 catalog_manager.cc:5746] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767) does not have the latest schema for tablet ba91a508b1cb4430a4d8ca26946aecc8 (table default.Table_aa496bf2019748f29823a9a253b1f3ca [id=780188e2335f43fbb11de8212eb38d56]). Expected version 1 got 0
I20260130 02:58:23.018041 18866 catalog_manager.cc:5746] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767) does not have the latest schema for tablet f2d01bfdc9c4494386bd02bed655fe63 (table default.Table_aaf139c1a55d4a1783298ca12ae0d5c0 [id=d128767e303145a885eec3c5f714a27f]). Expected version 4 got 3
I20260130 02:58:23.019032 20613 leader_election.cc:304] T 00000000000000000000000000000000 P 3749baa287ed4f249329a74071781edd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3749baa287ed4f249329a74071781edd; no voters: 107f274cdda543038832f06a56ab9130, 658299197a424f53a37f14fadf089ae4
I20260130 02:58:23.022128 19625 consensus_queue.cc:260] T 306452e208ca4e719609f67345114a13 P d870164263bb44b6b780a557daea3656 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:23.022732 21000 raft_consensus.cc:2804] T ca06f07bb98f45a7bf63aebc28348e08 P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Leader election won for term 1
I20260130 02:58:23.028618 21000 raft_consensus.cc:697] T ca06f07bb98f45a7bf63aebc28348e08 P d870164263bb44b6b780a557daea3656 [term 1 LEADER]: Becoming Leader. State: Replica: d870164263bb44b6b780a557daea3656, State: Running, Role: LEADER
I20260130 02:58:23.027616 21057 leader_election.cc:290] T ab6afa40408a4f458495abf20290e55b P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 election: Requested vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767)
I20260130 02:58:23.029884 19625 ts_tablet_manager.cc:1434] T 306452e208ca4e719609f67345114a13 P d870164263bb44b6b780a557daea3656: Time spent starting tablet: real 0.062s user 0.006s sys 0.000s
I20260130 02:58:23.029564 21000 consensus_queue.cc:237] T ca06f07bb98f45a7bf63aebc28348e08 P d870164263bb44b6b780a557daea3656 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:23.030697 21103 ts_tablet_manager.cc:1916] T e12d130bcdf34965888753f12c1ff848 P d870164263bb44b6b780a557daea3656: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260130 02:58:23.036991 19518 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cfd12a72726846e587cff9d6a4d61bd6" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
I20260130 02:58:23.040481 19517 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e34d5da0ad864b2eaea0d575c982e447" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
W20260130 02:58:23.042944 19463 leader_election.cc:343] T ab6afa40408a4f458495abf20290e55b P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:23.044284 21021 consensus_queue.cc:1048] T 49efcfced0b74c48962393149ac2a500 P 78c2c46c7520492db35d51c244fb4f74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260130 02:58:23.048238 18866 catalog_manager.cc:5746] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767) does not have the latest schema for tablet 657e9b0d8c05413f904129dce745710b (table default.Table_526677c8ef4144bd9de7a54a5e78e2be [id=b16908b75587459597a978c71756451c]). Expected version 1 got 0
I20260130 02:58:23.051225 19221 raft_consensus.cc:3060] T 133e9bfe175f44aab1a53114abe67e4d P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:23.055970 18866 catalog_manager.cc:5746] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767) does not have the latest schema for tablet 49efcfced0b74c48962393149ac2a500 (table default.Table_2e81f0a0cbc449a397141b717db17025 [id=a9c919f57c0c437cb35a6921103e45a6]). Expected version 1 got 0
I20260130 02:58:23.059166 19270 heartbeater.cc:461] Registering TS with master...
I20260130 02:58:23.060678 19374 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ca06f07bb98f45a7bf63aebc28348e08" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74"
I20260130 02:58:23.062134 19374 raft_consensus.cc:3060] T ca06f07bb98f45a7bf63aebc28348e08 P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Advancing to term 2
I20260130 02:58:23.065308 19364 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "aae68390dbbe4dc58ce3d3bd01725ae6" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74" is_pre_election: true
I20260130 02:58:23.071496 20704 raft_consensus.cc:493] T 6d1f121f49cd47c2b8247daaf6874e01 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:23.071938 20704 raft_consensus.cc:515] T 6d1f121f49cd47c2b8247daaf6874e01 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:23.073511 20704 leader_election.cc:290] T 6d1f121f49cd47c2b8247daaf6874e01 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), d870164263bb44b6b780a557daea3656 (127.18.100.67:39571)
I20260130 02:58:23.079254 19520 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "657e9b0d8c05413f904129dce745710b" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
W20260130 02:58:23.086602 19463 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.100.65:42021, user_credentials={real_user=slave}} blocked reactor thread for 43666.7us
I20260130 02:58:23.087392 21113 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 3749baa287ed4f249329a74071781edd [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260130 02:58:23.087903 19524 raft_consensus.cc:2468] T aae68390dbbe4dc58ce3d3bd01725ae6 P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78c2c46c7520492db35d51c244fb4f74 in term 1.
I20260130 02:58:23.082053 21056 raft_consensus.cc:515] T 0855d88806914a36b4bf6b49036f0ad0 P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:23.090667 19222 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6d1f121f49cd47c2b8247daaf6874e01" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152" is_pre_election: true
I20260130 02:58:23.085781 18866 catalog_manager.cc:5654] T 25fd48e693424537b91ba7c3fcc96878 P 78c2c46c7520492db35d51c244fb4f74 reported cstate change: term changed from 0 to 1, leader changed from <none> to 371a36d219164974abdf848841528152 (127.18.100.65). New cstate: current_term: 1 leader_uuid: "371a36d219164974abdf848841528152" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } }
I20260130 02:58:23.093966 18866 catalog_manager.cc:5746] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767) does not have the latest schema for tablet 25fd48e693424537b91ba7c3fcc96878 (table default.Table_ebc8b8379c274c18b0a06a919dbe0ee0 [id=28156e0843674fce88f5914a3e4195ae]). Expected version 1 got 0
I20260130 02:58:23.096812 18866 catalog_manager.cc:5746] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767) does not have the latest schema for tablet 838e49e5e8874db393bdc15a1c92f87e (table default.Table_da3d581b15a94840898777f74611994e [id=b7e241ced53e4160ac6fa1f65ef3674e]). Expected version 1 got 0
I20260130 02:58:23.099032 20777 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "d870164263bb44b6b780a557daea3656" instance_seqno: 1769741872714340) as {username='slave'} at 127.18.100.67:33827; Asking this server to re-register.
I20260130 02:58:23.100346 19374 raft_consensus.cc:2468] T ca06f07bb98f45a7bf63aebc28348e08 P 78c2c46c7520492db35d51c244fb4f74 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 371a36d219164974abdf848841528152 in term 2.
I20260130 02:58:23.092152 19807 raft_consensus.cc:359] T ab6afa40408a4f458495abf20290e55b P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:23.104321 19807 raft_consensus.cc:385] T ab6afa40408a4f458495abf20290e55b P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:23.100884 19527 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ab6afa40408a4f458495abf20290e55b" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656"
I20260130 02:58:23.106107 19527 raft_consensus.cc:2393] T ab6afa40408a4f458495abf20290e55b P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 78c2c46c7520492db35d51c244fb4f74 in current term 1: Already voted for candidate d870164263bb44b6b780a557daea3656 in this term.
I20260130 02:58:23.106411 19625 tablet_bootstrap.cc:492] T cfd12a72726846e587cff9d6a4d61bd6 P d870164263bb44b6b780a557daea3656: Bootstrap starting.
I20260130 02:58:23.110598 19807 raft_consensus.cc:740] T ab6afa40408a4f458495abf20290e55b P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 371a36d219164974abdf848841528152, State: Initialized, Role: FOLLOWER
I20260130 02:58:23.111287 19807 consensus_queue.cc:260] T ab6afa40408a4f458495abf20290e55b P 371a36d219164974abdf848841528152 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:23.113318 19807 ts_tablet_manager.cc:1434] T ab6afa40408a4f458495abf20290e55b P 371a36d219164974abdf848841528152: Time spent starting tablet: real 0.108s user 0.005s sys 0.001s
I20260130 02:58:23.113603 19625 tablet_bootstrap.cc:654] T cfd12a72726846e587cff9d6a4d61bd6 P d870164263bb44b6b780a557daea3656: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:23.114213 19807 tablet_bootstrap.cc:492] T eb7a755ab62b4552bb8588264c5f7b75 P 371a36d219164974abdf848841528152: Bootstrap starting.
W20260130 02:58:23.117723 19463 leader_election.cc:343] T 0855d88806914a36b4bf6b49036f0ad0 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:23.123696 21056 leader_election.cc:290] T 0855d88806914a36b4bf6b49036f0ad0 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 election: Requested vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767)
I20260130 02:58:23.121196 20810 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33758 (request call id 0) took 5161 ms. Trace:
I20260130 02:58:23.130146 20987 consensus_queue.cc:1048] T 12c3f3ec523040e7b7cd356e3da57e0d P 371a36d219164974abdf848841528152 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260130 02:58:23.133924 21103 ts_tablet_manager.cc:1929] T e12d130bcdf34965888753f12c1ff848 P d870164263bb44b6b780a557daea3656: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260130 02:58:23.134369 21103 log.cc:1199] T e12d130bcdf34965888753f12c1ff848 P d870164263bb44b6b780a557daea3656: Deleting WAL directory at /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-2/wal/wals/e12d130bcdf34965888753f12c1ff848
W20260130 02:58:23.114183 19314 leader_election.cc:343] T 6d1f121f49cd47c2b8247daaf6874e01 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:23.139258 19573 heartbeater.cc:461] Registering TS with master...
I20260130 02:58:23.153416 21103 ts_tablet_manager.cc:1950] T e12d130bcdf34965888753f12c1ff848 P d870164263bb44b6b780a557daea3656: Deleting consensus metadata
I20260130 02:58:23.155010 18860 catalog_manager.cc:4985] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): tablet e12d130bcdf34965888753f12c1ff848 (table default.Table_a675d7371dcd46b293f893ebe4939e7a [id=c7084428dde74c3ebc46943c44211d7b]) successfully deleted
I20260130 02:58:23.151820 19625 tablet_bootstrap.cc:492] T cfd12a72726846e587cff9d6a4d61bd6 P d870164263bb44b6b780a557daea3656: No bootstrap required, opened a new log
I20260130 02:58:23.162315 19625 ts_tablet_manager.cc:1403] T cfd12a72726846e587cff9d6a4d61bd6 P d870164263bb44b6b780a557daea3656: Time spent bootstrapping tablet: real 0.056s user 0.008s sys 0.003s
I20260130 02:58:23.164666 19625 raft_consensus.cc:359] T cfd12a72726846e587cff9d6a4d61bd6 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:23.165259 19625 raft_consensus.cc:385] T cfd12a72726846e587cff9d6a4d61bd6 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:23.165524 19625 raft_consensus.cc:740] T cfd12a72726846e587cff9d6a4d61bd6 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d870164263bb44b6b780a557daea3656, State: Initialized, Role: FOLLOWER
I20260130 02:58:23.170612 21000 consensus_queue.cc:1048] T 133e9bfe175f44aab1a53114abe67e4d P d870164263bb44b6b780a557daea3656 [LEADER]: Connected to new peer: Peer: permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260130 02:58:23.177647 19625 consensus_queue.cc:260] T cfd12a72726846e587cff9d6a4d61bd6 P d870164263bb44b6b780a557daea3656 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:23.179328 19625 ts_tablet_manager.cc:1434] T cfd12a72726846e587cff9d6a4d61bd6 P d870164263bb44b6b780a557daea3656: Time spent starting tablet: real 0.017s user 0.004s sys 0.000s
I20260130 02:58:23.180943 19524 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d480bf18140245e3870a62582d5a8318" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
I20260130 02:58:23.181622 19524 raft_consensus.cc:2393] T d480bf18140245e3870a62582d5a8318 P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 371a36d219164974abdf848841528152 in current term 1: Already voted for candidate 78c2c46c7520492db35d51c244fb4f74 in this term.
I20260130 02:58:23.183326 19221 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ab6afa40408a4f458495abf20290e55b" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152"
I20260130 02:58:23.183900 19221 raft_consensus.cc:3060] T ab6afa40408a4f458495abf20290e55b P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:23.198935 19225 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "eb7a755ab62b4552bb8588264c5f7b75" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152"
I20260130 02:58:23.200945 19225 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0855d88806914a36b4bf6b49036f0ad0" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152"
I20260130 02:58:23.203195 18875 catalog_manager.cc:5941] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4: Sending DeleteTablet for 3 replicas of tablet 43cec19c51ec4f05b63800c07f9534ef
I20260130 02:58:23.204430 18875 catalog_manager.cc:6724] ReplaceTablet: tablet 43cec19c51ec4f05b63800c07f9534ef deleted and replaced by tablet 3379641837df4b6ea360e8e27dfe4f8b
I20260130 02:58:23.204840 18875 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:50284 (request call id 268) took 2681 ms. Trace:
I20260130 02:58:23.206413 19505 tablet_service.cc:1552] Processing DeleteTablet for tablet 43cec19c51ec4f05b63800c07f9534ef with delete_type TABLET_DATA_DELETED (replaced by tablet 3379641837df4b6ea360e8e27dfe4f8b) from {username='slave'} at 127.0.0.1:59068
I20260130 02:58:23.205194 18875 rpcz_store.cc:276] 0130 02:58:20.523593 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:58:22.858510 (+2334917us) service_pool.cc:225] Handling call
0130 02:58:23.204786 (+346276us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0130 02:58:23.188943 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0130 02:58:23.189387 (+ 444us) write_op.cc:432] Acquiring schema lock in shared mode
0130 02:58:23.189413 (+ 26us) write_op.cc:435] Acquired schema lock
0130 02:58:23.189430 (+ 17us) tablet.cc:660] Decoding operations
0130 02:58:23.189747 (+ 317us) write_op.cc:620] Acquiring the partition lock for write op
0130 02:58:23.189793 (+ 46us) write_op.cc:641] Partition lock acquired for write op
0130 02:58:23.189814 (+ 21us) tablet.cc:683] Acquiring locks for 2 operations
0130 02:58:23.190020 (+ 206us) tablet.cc:699] Row locks acquired
0130 02:58:23.190038 (+ 18us) write_op.cc:260] PREPARE: finished
0130 02:58:23.190162 (+ 124us) write_op.cc:270] Start()
0130 02:58:23.190249 (+ 87us) write_op.cc:276] Timestamp: P: 1769741903190123 usec, L: 0
0130 02:58:23.190264 (+ 15us) op_driver.cc:348] REPLICATION: starting
0130 02:58:23.190898 (+ 634us) log.cc:844] Serialized 662 byte log entry
0130 02:58:23.201093 (+ 10195us) op_driver.cc:464] REPLICATION: finished
0130 02:58:23.201404 (+ 311us) write_op.cc:301] APPLY: starting
0130 02:58:23.201454 (+ 50us) tablet.cc:1365] starting BulkCheckPresence
0130 02:58:23.201552 (+ 98us) tablet.cc:1368] finished BulkCheckPresence
0130 02:58:23.201659 (+ 107us) tablet.cc:1370] starting ApplyRowOperation cycle
0130 02:58:23.202001 (+ 342us) tablet.cc:1381] finished ApplyRowOperation cycle
0130 02:58:23.202200 (+ 199us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0130 02:58:23.202240 (+ 40us) write_op.cc:312] APPLY: finished
0130 02:58:23.202406 (+ 166us) log.cc:844] Serialized 29 byte log entry
0130 02:58:23.202544 (+ 138us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:58:23.202636 (+ 92us) write_op.cc:454] Released schema lock
0130 02:58:23.202682 (+ 46us) write_op.cc:341] FINISH: Updating metrics
0130 02:58:23.202958 (+ 276us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:58:23.202981 (+ 23us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":162,"apply.run_cpu_time_us":1458,"apply.run_wall_time_us":1458,"num_ops":2,"prepare.queue_time_us":105,"prepare.run_cpu_time_us":2254,"prepare.run_wall_time_us":2254,"raft.queue_time_us":969,"raft.run_cpu_time_us":707,"raft.run_wall_time_us":707,"replication_time_us":10730}]]}
I20260130 02:58:23.208927 20777 ts_manager.cc:194] Registered new tserver with Master: d870164263bb44b6b780a557daea3656 (127.18.100.67:39571)
I20260130 02:58:23.210856 21103 tablet_replica.cc:333] T 43cec19c51ec4f05b63800c07f9534ef P d870164263bb44b6b780a557daea3656: stopping tablet replica
W20260130 02:58:23.211675 19159 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.100.67:39571, user_credentials={real_user=slave}} blocked reactor thread for 77279.4us
I20260130 02:58:23.212116 19625 tablet_bootstrap.cc:492] T cd30d7adcc4545848e3501a205a10293 P d870164263bb44b6b780a557daea3656: Bootstrap starting.
I20260130 02:58:23.127952 19364 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "eb7a755ab62b4552bb8588264c5f7b75" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74"
I20260130 02:58:23.215415 19364 raft_consensus.cc:3060] T eb7a755ab62b4552bb8588264c5f7b75 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:23.216811 21103 raft_consensus.cc:2243] T 43cec19c51ec4f05b63800c07f9534ef P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260130 02:58:23.217386 21103 raft_consensus.cc:2272] T 43cec19c51ec4f05b63800c07f9534ef P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260130 02:58:23.218983 19197 tablet_service.cc:1552] Processing DeleteTablet for tablet 43cec19c51ec4f05b63800c07f9534ef with delete_type TABLET_DATA_DELETED (replaced by tablet 3379641837df4b6ea360e8e27dfe4f8b) from {username='slave'} at 127.0.0.1:42870
I20260130 02:58:23.219599 21078 tablet_replica.cc:333] T 43cec19c51ec4f05b63800c07f9534ef P 371a36d219164974abdf848841528152: stopping tablet replica
I20260130 02:58:23.219805 21103 ts_tablet_manager.cc:1916] T 43cec19c51ec4f05b63800c07f9534ef P d870164263bb44b6b780a557daea3656: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260130 02:58:23.130287 19375 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ab6afa40408a4f458495abf20290e55b" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74"
I20260130 02:58:23.141835 19840 tablet_bootstrap.cc:492] T 920f52b975ab480fbf42656171222f09 P 78c2c46c7520492db35d51c244fb4f74: Bootstrap starting.
I20260130 02:58:23.220378 21078 raft_consensus.cc:2243] T 43cec19c51ec4f05b63800c07f9534ef P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260130 02:58:23.220850 21078 raft_consensus.cc:2272] T 43cec19c51ec4f05b63800c07f9534ef P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260130 02:58:23.223523 21078 ts_tablet_manager.cc:1916] T 43cec19c51ec4f05b63800c07f9534ef P 371a36d219164974abdf848841528152: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260130 02:58:23.230710 19221 raft_consensus.cc:2468] T ab6afa40408a4f458495abf20290e55b P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d870164263bb44b6b780a557daea3656 in term 1.
I20260130 02:58:23.231393 19527 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2b0edaad6877460cb7982efc62b00b66" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
I20260130 02:58:23.245096 19527 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ca06f07bb98f45a7bf63aebc28348e08" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656"
I20260130 02:58:23.254948 19358 tablet_service.cc:1552] Processing DeleteTablet for tablet 43cec19c51ec4f05b63800c07f9534ef with delete_type TABLET_DATA_DELETED (replaced by tablet 3379641837df4b6ea360e8e27dfe4f8b) from {username='slave'} at 127.0.0.1:60110
I20260130 02:58:23.256767 21000 consensus_queue.cc:1048] T a3258852aed24ed582a79f77860e6f0d P d870164263bb44b6b780a557daea3656 [LEADER]: Connected to new peer: Peer: permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260130 02:58:23.258323 19463 leader_election.cc:343] T eb7a755ab62b4552bb8588264c5f7b75 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Illegal state: must be running to vote when last-logged opid is not known
W20260130 02:58:23.258888 19463 leader_election.cc:343] T 0855d88806914a36b4bf6b49036f0ad0 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:23.263589 19625 tablet_bootstrap.cc:654] T cd30d7adcc4545848e3501a205a10293 P d870164263bb44b6b780a557daea3656: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:23.271552 21103 ts_tablet_manager.cc:1929] T 43cec19c51ec4f05b63800c07f9534ef P d870164263bb44b6b780a557daea3656: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260130 02:58:23.278780 21103 log.cc:1199] T 43cec19c51ec4f05b63800c07f9534ef P d870164263bb44b6b780a557daea3656: Deleting WAL directory at /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-2/wal/wals/43cec19c51ec4f05b63800c07f9534ef
I20260130 02:58:23.279791 21103 ts_tablet_manager.cc:1950] T 43cec19c51ec4f05b63800c07f9534ef P d870164263bb44b6b780a557daea3656: Deleting consensus metadata
I20260130 02:58:23.277498 19463 leader_election.cc:304] T ab6afa40408a4f458495abf20290e55b P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 371a36d219164974abdf848841528152, d870164263bb44b6b780a557daea3656; no voters:
I20260130 02:58:23.281108 21000 raft_consensus.cc:2804] T ab6afa40408a4f458495abf20290e55b P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Leader election won for term 1
I20260130 02:58:23.281560 21000 raft_consensus.cc:697] T ab6afa40408a4f458495abf20290e55b P d870164263bb44b6b780a557daea3656 [term 1 LEADER]: Becoming Leader. State: Replica: d870164263bb44b6b780a557daea3656, State: Running, Role: LEADER
I20260130 02:58:23.282322 21000 consensus_queue.cc:237] T ab6afa40408a4f458495abf20290e55b P d870164263bb44b6b780a557daea3656 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:23.283298 19625 tablet_bootstrap.cc:492] T cd30d7adcc4545848e3501a205a10293 P d870164263bb44b6b780a557daea3656: No bootstrap required, opened a new log
I20260130 02:58:23.283723 19625 ts_tablet_manager.cc:1403] T cd30d7adcc4545848e3501a205a10293 P d870164263bb44b6b780a557daea3656: Time spent bootstrapping tablet: real 0.072s user 0.013s sys 0.001s
I20260130 02:58:23.288231 19625 raft_consensus.cc:359] T cd30d7adcc4545848e3501a205a10293 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:23.289631 19625 raft_consensus.cc:385] T cd30d7adcc4545848e3501a205a10293 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:23.289898 19625 raft_consensus.cc:740] T cd30d7adcc4545848e3501a205a10293 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d870164263bb44b6b780a557daea3656, State: Initialized, Role: FOLLOWER
I20260130 02:58:23.290747 19375 raft_consensus.cc:2393] T ab6afa40408a4f458495abf20290e55b P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate d870164263bb44b6b780a557daea3656 in current term 1: Already voted for candidate 78c2c46c7520492db35d51c244fb4f74 in this term.
I20260130 02:58:23.290966 18860 catalog_manager.cc:4985] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): tablet 43cec19c51ec4f05b63800c07f9534ef (table default.Table_8f632ff84d624b72a10cb90fb9e78636 [id=3cf79022ea954f6397f0a0497e6e6a6c]) successfully deleted
I20260130 02:58:23.290832 21122 tablet.cc:1722] T f2d01bfdc9c4494386bd02bed655fe63 P d870164263bb44b6b780a557daea3656: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20260130 02:58:23.298655 19625 consensus_queue.cc:260] T cd30d7adcc4545848e3501a205a10293 P d870164263bb44b6b780a557daea3656 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:23.301352 21102 tablet.cc:1722] T f2d01bfdc9c4494386bd02bed655fe63 P 371a36d219164974abdf848841528152: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20260130 02:58:23.302722 19625 ts_tablet_manager.cc:1434] T cd30d7adcc4545848e3501a205a10293 P d870164263bb44b6b780a557daea3656: Time spent starting tablet: real 0.019s user 0.006s sys 0.000s
I20260130 02:58:23.303685 19625 tablet_bootstrap.cc:492] T c76f0dc7762c49c389fa6fef41f02141 P d870164263bb44b6b780a557daea3656: Bootstrap starting.
I20260130 02:58:23.318724 19625 tablet_bootstrap.cc:654] T c76f0dc7762c49c389fa6fef41f02141 P d870164263bb44b6b780a557daea3656: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:23.319193 20899 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60110 (request call id 332) took 1057 ms. Trace:
I20260130 02:58:23.319587 20899 rpcz_store.cc:276] 0130 02:58:22.261823 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:58:22.262015 (+ 192us) service_pool.cc:225] Handling call
0130 02:58:22.509974 (+247959us) ts_tablet_manager.cc:1151] Deleting tablet e12d130bcdf34965888753f12c1ff848
0130 02:58:22.510017 (+ 43us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0130 02:58:22.728656 (+218639us) tablet_metadata.cc:641] Metadata flushed
0130 02:58:22.917048 (+188392us) tablet_metadata.cc:641] Metadata flushed
0130 02:58:23.319125 (+402077us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":16270,"tablet-delete.queue_time_us":218796}
I20260130 02:58:23.322191 20899 tablet_replica.cc:333] T 2c79a3aecab844698e70834999e1caac P 78c2c46c7520492db35d51c244fb4f74: stopping tablet replica
I20260130 02:58:23.326720 19840 tablet_bootstrap.cc:654] T 920f52b975ab480fbf42656171222f09 P 78c2c46c7520492db35d51c244fb4f74: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:23.332082 19364 raft_consensus.cc:2468] T eb7a755ab62b4552bb8588264c5f7b75 P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d870164263bb44b6b780a557daea3656 in term 1.
I20260130 02:58:23.335084 20899 raft_consensus.cc:2243] T 2c79a3aecab844698e70834999e1caac P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260130 02:58:23.335783 20899 raft_consensus.cc:2272] T 2c79a3aecab844698e70834999e1caac P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260130 02:58:23.344902 19625 tablet_bootstrap.cc:492] T c76f0dc7762c49c389fa6fef41f02141 P d870164263bb44b6b780a557daea3656: No bootstrap required, opened a new log
I20260130 02:58:23.345340 19625 ts_tablet_manager.cc:1403] T c76f0dc7762c49c389fa6fef41f02141 P d870164263bb44b6b780a557daea3656: Time spent bootstrapping tablet: real 0.042s user 0.006s sys 0.005s
I20260130 02:58:23.347875 19625 raft_consensus.cc:359] T c76f0dc7762c49c389fa6fef41f02141 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:23.348544 19625 raft_consensus.cc:385] T c76f0dc7762c49c389fa6fef41f02141 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:23.348803 19625 raft_consensus.cc:740] T c76f0dc7762c49c389fa6fef41f02141 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d870164263bb44b6b780a557daea3656, State: Initialized, Role: FOLLOWER
W20260130 02:58:23.350229 21115 consensus_queue.cc:1175] T 2c79a3aecab844698e70834999e1caac P 78c2c46c7520492db35d51c244fb4f74 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "78c2c46c7520492db35d51c244fb4f74" status { last_received { term: 1 index: 1 } last_committed_idx: 1 last_received_current_leader { term: 1 index: 1 } }
I20260130 02:58:23.354641 20899 ts_tablet_manager.cc:1916] T 2c79a3aecab844698e70834999e1caac P 78c2c46c7520492db35d51c244fb4f74: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260130 02:58:23.360929 19430 tablet.cc:2378] T 1129f72158ec4bae89da3c94cdb70839 P 78c2c46c7520492db35d51c244fb4f74: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260130 02:58:23.362691 19625 consensus_queue.cc:260] T c76f0dc7762c49c389fa6fef41f02141 P d870164263bb44b6b780a557daea3656 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:23.365128 19625 ts_tablet_manager.cc:1434] T c76f0dc7762c49c389fa6fef41f02141 P d870164263bb44b6b780a557daea3656: Time spent starting tablet: real 0.020s user 0.005s sys 0.000s
I20260130 02:58:23.366012 19625 tablet_bootstrap.cc:492] T 2b8034a92b2043e5b57cf02f1ee6296d P d870164263bb44b6b780a557daea3656: Bootstrap starting.
I20260130 02:58:23.370020 19372 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0855d88806914a36b4bf6b49036f0ad0" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74"
I20260130 02:58:23.370766 19372 raft_consensus.cc:3060] T 0855d88806914a36b4bf6b49036f0ad0 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:23.371884 19625 tablet_bootstrap.cc:654] T 2b8034a92b2043e5b57cf02f1ee6296d P d870164263bb44b6b780a557daea3656: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:23.382462 18873 catalog_manager.cc:5941] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4: Sending DeleteTablet for 3 replicas of tablet ba91a508b1cb4430a4d8ca26946aecc8
W20260130 02:58:23.385106 19569 debug-util.cc:398] Leaking SignalData structure 0x7b0800123340 after lost signal to thread 19438
I20260130 02:58:23.385767 19505 tablet_service.cc:1552] Processing DeleteTablet for tablet ba91a508b1cb4430a4d8ca26946aecc8 with delete_type TABLET_DATA_DELETED (replaced by tablet c2c927e461f04fb48bfa9e3a685b077d) from {username='slave'} at 127.0.0.1:59068
W20260130 02:58:23.386363 19569 debug-util.cc:398] Leaking SignalData structure 0x7b0800083e00 after lost signal to thread 19572
I20260130 02:58:23.386466 18873 catalog_manager.cc:6724] ReplaceTablet: tablet ba91a508b1cb4430a4d8ca26946aecc8 deleted and replaced by tablet c2c927e461f04fb48bfa9e3a685b077d
I20260130 02:58:23.386591 21103 tablet_replica.cc:333] T ba91a508b1cb4430a4d8ca26946aecc8 P d870164263bb44b6b780a557daea3656: stopping tablet replica
W20260130 02:58:23.386698 19569 debug-util.cc:398] Leaking SignalData structure 0x7b080014fb80 after lost signal to thread 21019
W20260130 02:58:23.386983 19569 debug-util.cc:398] Leaking SignalData structure 0x7b0800124e80 after lost signal to thread 21046
I20260130 02:58:23.250240 19159 leader_election.cc:304] T 133e9bfe175f44aab1a53114abe67e4d P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 371a36d219164974abdf848841528152; no voters: 78c2c46c7520492db35d51c244fb4f74, d870164263bb44b6b780a557daea3656
I20260130 02:58:23.387755 21103 raft_consensus.cc:2243] T ba91a508b1cb4430a4d8ca26946aecc8 P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260130 02:58:23.388252 21103 pending_rounds.cc:70] T ba91a508b1cb4430a4d8ca26946aecc8 P d870164263bb44b6b780a557daea3656: Trying to abort 1 pending ops.
W20260130 02:58:23.388224 19159 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.100.67:39571, user_credentials={real_user=slave}} blocked reactor thread for 140546us
W20260130 02:58:23.388830 19159 leader_election.cc:343] T 78e9bc3e55fe4e4a98e8eb01da59d365 P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): Not found: Tablet not found: 78e9bc3e55fe4e4a98e8eb01da59d365
I20260130 02:58:23.388487 21103 pending_rounds.cc:77] T ba91a508b1cb4430a4d8ca26946aecc8 P d870164263bb44b6b780a557daea3656: Aborting op as it isn't in flight: id { term: 1 index: 2 } timestamp: 7248862831521841152 op_type: ALTER_SCHEMA_OP alter_schema_request { tablet_id: "ba91a508b1cb4430a4d8ca26946aecc8" schema { columns { id: 10 name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } schema_version: 1 new_table_name: "default.Table_aa496bf2019748f29823a9a253b1f3ca" dest_uuid: "371a36d219164974abdf848841528152" new_extra_config { } }
I20260130 02:58:23.389154 19159 leader_election.cc:304] T 78e9bc3e55fe4e4a98e8eb01da59d365 P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 371a36d219164974abdf848841528152; no voters: 78c2c46c7520492db35d51c244fb4f74, d870164263bb44b6b780a557daea3656
W20260130 02:58:23.389707 19159 raft_consensus.cc:2686] T 78e9bc3e55fe4e4a98e8eb01da59d365 P 371a36d219164974abdf848841528152: unable to run election callback: Service unavailable: Thread pool token was shut down
I20260130 02:58:23.390035 21103 raft_consensus.cc:2272] T ba91a508b1cb4430a4d8ca26946aecc8 P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260130 02:58:23.391033 19197 tablet_service.cc:1552] Processing DeleteTablet for tablet ba91a508b1cb4430a4d8ca26946aecc8 with delete_type TABLET_DATA_DELETED (replaced by tablet c2c927e461f04fb48bfa9e3a685b077d) from {username='slave'} at 127.0.0.1:42870
I20260130 02:58:23.393178 21103 ts_tablet_manager.cc:1916] T ba91a508b1cb4430a4d8ca26946aecc8 P d870164263bb44b6b780a557daea3656: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260130 02:58:23.401901 19625 tablet_bootstrap.cc:492] T 2b8034a92b2043e5b57cf02f1ee6296d P d870164263bb44b6b780a557daea3656: No bootstrap required, opened a new log
I20260130 02:58:23.402349 19625 ts_tablet_manager.cc:1403] T 2b8034a92b2043e5b57cf02f1ee6296d P d870164263bb44b6b780a557daea3656: Time spent bootstrapping tablet: real 0.037s user 0.010s sys 0.000s
I20260130 02:58:23.405364 19625 raft_consensus.cc:359] T 2b8034a92b2043e5b57cf02f1ee6296d P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:23.405961 19625 raft_consensus.cc:385] T 2b8034a92b2043e5b57cf02f1ee6296d P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:23.406208 19625 raft_consensus.cc:740] T 2b8034a92b2043e5b57cf02f1ee6296d P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d870164263bb44b6b780a557daea3656, State: Initialized, Role: FOLLOWER
I20260130 02:58:23.409735 19625 consensus_queue.cc:260] T 2b8034a92b2043e5b57cf02f1ee6296d P d870164263bb44b6b780a557daea3656 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:23.420692 21103 ts_tablet_manager.cc:1929] T ba91a508b1cb4430a4d8ca26946aecc8 P d870164263bb44b6b780a557daea3656: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
W20260130 02:58:23.423838 19313 leader_election.cc:343] T dd377fab8c544bf69121dc9e9a060b29 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:23.424275 19313 leader_election.cc:304] T dd377fab8c544bf69121dc9e9a060b29 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78c2c46c7520492db35d51c244fb4f74; no voters: 371a36d219164974abdf848841528152, d870164263bb44b6b780a557daea3656
I20260130 02:58:23.424849 19625 ts_tablet_manager.cc:1434] T 2b8034a92b2043e5b57cf02f1ee6296d P d870164263bb44b6b780a557daea3656: Time spent starting tablet: real 0.022s user 0.006s sys 0.000s
I20260130 02:58:23.434933 21103 log.cc:1199] T ba91a508b1cb4430a4d8ca26946aecc8 P d870164263bb44b6b780a557daea3656: Deleting WAL directory at /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-2/wal/wals/ba91a508b1cb4430a4d8ca26946aecc8
I20260130 02:58:23.435119 20987 raft_consensus.cc:2749] T 133e9bfe175f44aab1a53114abe67e4d P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260130 02:58:23.435518 19625 tablet_bootstrap.cc:492] T 94355fc1338f479e8798070aff04fcba P d870164263bb44b6b780a557daea3656: Bootstrap starting.
I20260130 02:58:23.436058 21103 ts_tablet_manager.cc:1950] T ba91a508b1cb4430a4d8ca26946aecc8 P d870164263bb44b6b780a557daea3656: Deleting consensus metadata
I20260130 02:58:23.452823 19625 tablet_bootstrap.cc:654] T 94355fc1338f479e8798070aff04fcba P d870164263bb44b6b780a557daea3656: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:23.453365 21133 raft_consensus.cc:2749] T dd377fab8c544bf69121dc9e9a060b29 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260130 02:58:23.456221 18873 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:50284 (request call id 269) took 2871 ms. Trace:
I20260130 02:58:23.440029 18860 catalog_manager.cc:4985] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): tablet ba91a508b1cb4430a4d8ca26946aecc8 (table default.Table_aa496bf2019748f29823a9a253b1f3ca [id=780188e2335f43fbb11de8212eb38d56]) successfully deleted
I20260130 02:58:23.456588 18873 rpcz_store.cc:276] 0130 02:58:20.584628 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:58:22.962658 (+2378030us) service_pool.cc:225] Handling call
0130 02:58:23.456159 (+493501us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0130 02:58:23.191302 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0130 02:58:23.191744 (+ 442us) write_op.cc:432] Acquiring schema lock in shared mode
0130 02:58:23.191768 (+ 24us) write_op.cc:435] Acquired schema lock
0130 02:58:23.191783 (+ 15us) tablet.cc:660] Decoding operations
0130 02:58:23.192081 (+ 298us) write_op.cc:620] Acquiring the partition lock for write op
0130 02:58:23.192121 (+ 40us) write_op.cc:641] Partition lock acquired for write op
0130 02:58:23.192148 (+ 27us) tablet.cc:683] Acquiring locks for 2 operations
0130 02:58:23.192353 (+ 205us) tablet.cc:699] Row locks acquired
0130 02:58:23.192366 (+ 13us) write_op.cc:260] PREPARE: finished
0130 02:58:23.192486 (+ 120us) write_op.cc:270] Start()
0130 02:58:23.192566 (+ 80us) write_op.cc:276] Timestamp: P: 1769741903192448 usec, L: 0
0130 02:58:23.192582 (+ 16us) op_driver.cc:348] REPLICATION: starting
0130 02:58:23.193036 (+ 454us) log.cc:844] Serialized 662 byte log entry
0130 02:58:23.241655 (+ 48619us) op_driver.cc:464] REPLICATION: finished
0130 02:58:23.242821 (+ 1166us) write_op.cc:301] APPLY: starting
0130 02:58:23.242875 (+ 54us) tablet.cc:1365] starting BulkCheckPresence
0130 02:58:23.242973 (+ 98us) tablet.cc:1368] finished BulkCheckPresence
0130 02:58:23.243061 (+ 88us) tablet.cc:1370] starting ApplyRowOperation cycle
0130 02:58:23.243400 (+ 339us) tablet.cc:1381] finished ApplyRowOperation cycle
0130 02:58:23.243597 (+ 197us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0130 02:58:23.243649 (+ 52us) write_op.cc:312] APPLY: finished
0130 02:58:23.243834 (+ 185us) log.cc:844] Serialized 29 byte log entry
0130 02:58:23.243974 (+ 140us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:58:23.244040 (+ 66us) write_op.cc:454] Released schema lock
0130 02:58:23.244074 (+ 34us) write_op.cc:341] FINISH: Updating metrics
0130 02:58:23.244499 (+ 425us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:58:23.244523 (+ 24us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":1023,"apply.run_cpu_time_us":1478,"apply.run_wall_time_us":1649,"num_ops":2,"prepare.queue_time_us":1555,"prepare.run_cpu_time_us":1930,"prepare.run_wall_time_us":1929,"replication_time_us":48961}]]}
W20260130 02:58:23.470952 19280 tablet.cc:2378] T ab6afa40408a4f458495abf20290e55b P 371a36d219164974abdf848841528152: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260130 02:58:23.471978 19314 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.100.65:42021, user_credentials={real_user=slave}} blocked reactor thread for 217628us
I20260130 02:58:23.474598 19358 tablet_service.cc:1552] Processing DeleteTablet for tablet ba91a508b1cb4430a4d8ca26946aecc8 with delete_type TABLET_DATA_DELETED (replaced by tablet c2c927e461f04fb48bfa9e3a685b077d) from {username='slave'} at 127.0.0.1:60110
I20260130 02:58:23.474876 18860 catalog_manager.cc:4985] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767): tablet e12d130bcdf34965888753f12c1ff848 (table default.Table_a675d7371dcd46b293f893ebe4939e7a [id=c7084428dde74c3ebc46943c44211d7b]) successfully deleted
W20260130 02:58:23.475512 18860 catalog_manager.cc:4952] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767): delete failed for tablet 78e9bc3e55fe4e4a98e8eb01da59d365 because the tablet was not found. No further retry: Not found: Tablet not found: 78e9bc3e55fe4e4a98e8eb01da59d365
W20260130 02:58:23.476071 18860 catalog_manager.cc:4952] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767): delete failed for tablet 04f4ac27800c4ac3a79ae12d35479757 because the tablet was not found. No further retry: Not found: Tablet not found: 04f4ac27800c4ac3a79ae12d35479757
I20260130 02:58:23.476156 19625 tablet_bootstrap.cc:492] T 94355fc1338f479e8798070aff04fcba P d870164263bb44b6b780a557daea3656: No bootstrap required, opened a new log
I20260130 02:58:23.476627 19625 ts_tablet_manager.cc:1403] T 94355fc1338f479e8798070aff04fcba P d870164263bb44b6b780a557daea3656: Time spent bootstrapping tablet: real 0.041s user 0.005s sys 0.006s
W20260130 02:58:23.476575 18860 catalog_manager.cc:4952] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767): delete failed for tablet 8e57ae31ab3941f789ac69627384e3b0 because the tablet was not found. No further retry: Not found: Tablet not found: 8e57ae31ab3941f789ac69627384e3b0
W20260130 02:58:23.477063 18860 catalog_manager.cc:4952] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767): delete failed for tablet e2aaaf00f53d418b87d5c7af286e4d84 because the tablet was not found. No further retry: Not found: Tablet not found: e2aaaf00f53d418b87d5c7af286e4d84
W20260130 02:58:23.479795 19580 tablet.cc:2378] T 2b8034a92b2043e5b57cf02f1ee6296d P d870164263bb44b6b780a557daea3656: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260130 02:58:23.488935 19159 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.100.67:39571, user_credentials={real_user=slave}} blocked reactor thread for 100131us
I20260130 02:58:23.488080 19625 raft_consensus.cc:359] T 94355fc1338f479e8798070aff04fcba P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:23.491493 19625 raft_consensus.cc:385] T 94355fc1338f479e8798070aff04fcba P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:23.492640 19625 raft_consensus.cc:740] T 94355fc1338f479e8798070aff04fcba P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d870164263bb44b6b780a557daea3656, State: Initialized, Role: FOLLOWER
I20260130 02:58:23.493247 19625 consensus_queue.cc:260] T 94355fc1338f479e8798070aff04fcba P d870164263bb44b6b780a557daea3656 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:23.496167 19807 tablet_bootstrap.cc:654] T eb7a755ab62b4552bb8588264c5f7b75 P 371a36d219164974abdf848841528152: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:23.501945 19625 ts_tablet_manager.cc:1434] T 94355fc1338f479e8798070aff04fcba P d870164263bb44b6b780a557daea3656: Time spent starting tablet: real 0.019s user 0.004s sys 0.000s
I20260130 02:58:23.509845 19840 tablet_bootstrap.cc:492] T 920f52b975ab480fbf42656171222f09 P 78c2c46c7520492db35d51c244fb4f74: No bootstrap required, opened a new log
I20260130 02:58:23.510265 19840 ts_tablet_manager.cc:1403] T 920f52b975ab480fbf42656171222f09 P 78c2c46c7520492db35d51c244fb4f74: Time spent bootstrapping tablet: real 0.369s user 0.011s sys 0.003s
I20260130 02:58:23.514984 19625 tablet_bootstrap.cc:492] T 39bcc41f10ae4241b370529364454586 P d870164263bb44b6b780a557daea3656: Bootstrap starting.
I20260130 02:58:23.520360 19372 raft_consensus.cc:2468] T 0855d88806914a36b4bf6b49036f0ad0 P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d870164263bb44b6b780a557daea3656 in term 1.
W20260130 02:58:23.523106 21050 log.cc:927] Time spent T 00000000000000000000000000000000 P 3749baa287ed4f249329a74071781edd: Append to log took a long time: real 0.397s user 0.003s sys 0.000s
I20260130 02:58:23.542934 20987 consensus_queue.cc:1048] T f7607df5efd543438fb2dff3667cc8c8 P 371a36d219164974abdf848841528152 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260130 02:58:23.545497 19625 tablet_bootstrap.cc:654] T 39bcc41f10ae4241b370529364454586 P d870164263bb44b6b780a557daea3656: Neither blocks nor log segments found. Creating new log.
W20260130 02:58:23.547356 18859 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet 657e9b0d8c05413f904129dce745710b (table default.Table_526677c8ef4144bd9de7a54a5e78e2be [id=b16908b75587459597a978c71756451c], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 657e9b0d8c05413f904129dce745710b consensus state has no leader
W20260130 02:58:23.546820 19159 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.100.67:39571, user_credentials={real_user=slave}} blocked reactor thread for 53410.7us
I20260130 02:58:23.549919 19159 leader_election.cc:304] T e34d5da0ad864b2eaea0d575c982e447 P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 371a36d219164974abdf848841528152; no voters: 78c2c46c7520492db35d51c244fb4f74, d870164263bb44b6b780a557daea3656
I20260130 02:58:23.550606 20987 raft_consensus.cc:3060] T e34d5da0ad864b2eaea0d575c982e447 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:23.558748 20987 raft_consensus.cc:2749] T e34d5da0ad864b2eaea0d575c982e447 P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260130 02:58:23.579258 19159 leader_election.cc:304] T 657e9b0d8c05413f904129dce745710b P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 371a36d219164974abdf848841528152; no voters: 78c2c46c7520492db35d51c244fb4f74, d870164263bb44b6b780a557daea3656
I20260130 02:58:23.580703 21095 raft_consensus.cc:3060] T 657e9b0d8c05413f904129dce745710b P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:23.584373 19625 tablet_bootstrap.cc:492] T 39bcc41f10ae4241b370529364454586 P d870164263bb44b6b780a557daea3656: No bootstrap required, opened a new log
I20260130 02:58:23.584847 19625 ts_tablet_manager.cc:1403] T 39bcc41f10ae4241b370529364454586 P d870164263bb44b6b780a557daea3656: Time spent bootstrapping tablet: real 0.070s user 0.006s sys 0.004s
I20260130 02:58:23.587213 19625 raft_consensus.cc:359] T 39bcc41f10ae4241b370529364454586 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:23.588279 21095 raft_consensus.cc:2749] T 657e9b0d8c05413f904129dce745710b P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260130 02:58:23.588613 19625 raft_consensus.cc:385] T 39bcc41f10ae4241b370529364454586 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:23.588846 19625 raft_consensus.cc:740] T 39bcc41f10ae4241b370529364454586 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d870164263bb44b6b780a557daea3656, State: Initialized, Role: FOLLOWER
W20260130 02:58:23.589803 19314 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.100.65:42021, user_credentials={real_user=slave}} blocked reactor thread for 77197.4us
I20260130 02:58:23.598644 21094 raft_consensus.cc:493] T 94355fc1338f479e8798070aff04fcba P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:23.599093 21094 raft_consensus.cc:515] T 94355fc1338f479e8798070aff04fcba P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:23.601513 21094 leader_election.cc:290] T 94355fc1338f479e8798070aff04fcba P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767)
I20260130 02:58:23.603526 19220 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "94355fc1338f479e8798070aff04fcba" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152" is_pre_election: true
I20260130 02:58:23.603920 21134 raft_consensus.cc:493] T c76f0dc7762c49c389fa6fef41f02141 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:23.604305 21134 raft_consensus.cc:515] T c76f0dc7762c49c389fa6fef41f02141 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
W20260130 02:58:23.604631 19463 leader_election.cc:343] T 94355fc1338f479e8798070aff04fcba P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:23.606137 21134 leader_election.cc:290] T c76f0dc7762c49c389fa6fef41f02141 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), d870164263bb44b6b780a557daea3656 (127.18.100.67:39571)
I20260130 02:58:23.589411 19625 consensus_queue.cc:260] T 39bcc41f10ae4241b370529364454586 P d870164263bb44b6b780a557daea3656 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:23.614916 19625 ts_tablet_manager.cc:1434] T 39bcc41f10ae4241b370529364454586 P d870164263bb44b6b780a557daea3656: Time spent starting tablet: real 0.030s user 0.006s sys 0.000s
I20260130 02:58:23.619761 19840 raft_consensus.cc:359] T 920f52b975ab480fbf42656171222f09 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:23.623062 19840 raft_consensus.cc:385] T 920f52b975ab480fbf42656171222f09 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:23.633108 19220 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c76f0dc7762c49c389fa6fef41f02141" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152" is_pre_election: true
I20260130 02:58:23.641072 19625 tablet_bootstrap.cc:492] T 00ddbfe6218c44d788fbf65245a31dc6 P d870164263bb44b6b780a557daea3656: Bootstrap starting.
I20260130 02:58:23.655311 19840 raft_consensus.cc:740] T 920f52b975ab480fbf42656171222f09 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78c2c46c7520492db35d51c244fb4f74, State: Initialized, Role: FOLLOWER
I20260130 02:58:23.656438 19840 consensus_queue.cc:260] T 920f52b975ab480fbf42656171222f09 P 78c2c46c7520492db35d51c244fb4f74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:23.658946 19625 tablet_bootstrap.cc:654] T 00ddbfe6218c44d788fbf65245a31dc6 P d870164263bb44b6b780a557daea3656: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:23.659533 21143 raft_consensus.cc:493] T 00000000000000000000000000000000 P 3749baa287ed4f249329a74071781edd [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 658299197a424f53a37f14fadf089ae4)
I20260130 02:58:23.660118 21143 raft_consensus.cc:515] T 00000000000000000000000000000000 P 3749baa287ed4f249329a74071781edd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "658299197a424f53a37f14fadf089ae4" member_type: VOTER last_known_addr { host: "127.18.100.126" port: 38191 } } peers { permanent_uuid: "107f274cdda543038832f06a56ab9130" member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39711 } } peers { permanent_uuid: "3749baa287ed4f249329a74071781edd" member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34629 } }
I20260130 02:58:23.669051 18885 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3749baa287ed4f249329a74071781edd" candidate_term: 3 candidate_status { last_received { term: 2 index: 333 } } ignore_live_leader: false dest_uuid: "658299197a424f53a37f14fadf089ae4" is_pre_election: true
I20260130 02:58:23.675748 20998 raft_consensus.cc:493] T 1fc8f09d6329407c8078542b4123ef4d P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:23.676249 20998 raft_consensus.cc:515] T 1fc8f09d6329407c8078542b4123ef4d P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:23.678148 20998 leader_election.cc:290] T 1fc8f09d6329407c8078542b4123ef4d P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767)
I20260130 02:58:23.680501 18960 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3749baa287ed4f249329a74071781edd" candidate_term: 3 candidate_status { last_received { term: 2 index: 333 } } ignore_live_leader: false dest_uuid: "107f274cdda543038832f06a56ab9130" is_pre_election: true
I20260130 02:58:23.680918 19220 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1fc8f09d6329407c8078542b4123ef4d" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152" is_pre_election: true
W20260130 02:58:23.681921 19463 leader_election.cc:343] T 1fc8f09d6329407c8078542b4123ef4d P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:23.682547 20627 leader_election.cc:304] T 00000000000000000000000000000000 P 3749baa287ed4f249329a74071781edd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3749baa287ed4f249329a74071781edd; no voters: 107f274cdda543038832f06a56ab9130, 658299197a424f53a37f14fadf089ae4
W20260130 02:58:23.646746 19314 leader_election.cc:343] T c76f0dc7762c49c389fa6fef41f02141 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:23.649525 21121 raft_consensus.cc:493] T cd30d7adcc4545848e3501a205a10293 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:23.687692 21121 raft_consensus.cc:515] T cd30d7adcc4545848e3501a205a10293 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:23.687414 21143 leader_election.cc:290] T 00000000000000000000000000000000 P 3749baa287ed4f249329a74071781edd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 658299197a424f53a37f14fadf089ae4 (127.18.100.126:38191), 107f274cdda543038832f06a56ab9130 (127.18.100.125:39711)
I20260130 02:58:23.689445 19625 tablet_bootstrap.cc:492] T 00ddbfe6218c44d788fbf65245a31dc6 P d870164263bb44b6b780a557daea3656: No bootstrap required, opened a new log
I20260130 02:58:23.690790 21121 leader_election.cc:290] T cd30d7adcc4545848e3501a205a10293 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d870164263bb44b6b780a557daea3656 (127.18.100.67:39571), 371a36d219164974abdf848841528152 (127.18.100.65:42021)
W20260130 02:58:23.692667 19314 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.100.65:42021, user_credentials={real_user=slave}} blocked reactor thread for 45957.4us
I20260130 02:58:23.694152 19220 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cd30d7adcc4545848e3501a205a10293" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152" is_pre_election: true
W20260130 02:58:23.695312 18906 debug-util.cc:398] Leaking SignalData structure 0x7b08000c78a0 after lost signal to thread 18843
W20260130 02:58:23.696007 18906 debug-util.cc:398] Leaking SignalData structure 0x7b08000ca6a0 after lost signal to thread 18909
I20260130 02:58:23.688824 21143 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 3749baa287ed4f249329a74071781edd [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260130 02:58:23.705865 19314 leader_election.cc:343] T cd30d7adcc4545848e3501a205a10293 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:23.706846 21121 consensus_queue.cc:1048] T aae68390dbbe4dc58ce3d3bd01725ae6 P 78c2c46c7520492db35d51c244fb4f74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260130 02:58:23.710345 19220 raft_consensus.cc:3060] T 2b0edaad6877460cb7982efc62b00b66 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:23.715124 19840 ts_tablet_manager.cc:1434] T 920f52b975ab480fbf42656171222f09 P 78c2c46c7520492db35d51c244fb4f74: Time spent starting tablet: real 0.205s user 0.006s sys 0.000s
I20260130 02:58:23.715986 19807 tablet_bootstrap.cc:492] T eb7a755ab62b4552bb8588264c5f7b75 P 371a36d219164974abdf848841528152: No bootstrap required, opened a new log
I20260130 02:58:23.716045 19840 tablet_bootstrap.cc:492] T 8b47653c7c3945a68e9bf4680bfc0496 P 78c2c46c7520492db35d51c244fb4f74: Bootstrap starting.
I20260130 02:58:23.716506 19807 ts_tablet_manager.cc:1403] T eb7a755ab62b4552bb8588264c5f7b75 P 371a36d219164974abdf848841528152: Time spent bootstrapping tablet: real 0.602s user 0.004s sys 0.006s
W20260130 02:58:23.717790 19159 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.100.67:39571, user_credentials={real_user=slave}} blocked reactor thread for 137278us
I20260130 02:58:23.718860 19159 leader_election.cc:304] T d480bf18140245e3870a62582d5a8318 P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 371a36d219164974abdf848841528152; no voters: 78c2c46c7520492db35d51c244fb4f74, d870164263bb44b6b780a557daea3656
I20260130 02:58:23.723742 21095 raft_consensus.cc:2749] T d480bf18140245e3870a62582d5a8318 P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260130 02:58:23.725060 20704 raft_consensus.cc:493] T 1fc8f09d6329407c8078542b4123ef4d P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:23.725571 20704 raft_consensus.cc:515] T 1fc8f09d6329407c8078542b4123ef4d P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:23.727298 20704 leader_election.cc:290] T 1fc8f09d6329407c8078542b4123ef4d P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), d870164263bb44b6b780a557daea3656 (127.18.100.67:39571)
I20260130 02:58:23.701553 20998 consensus_queue.cc:1048] T dfbbe9045e7c4aa5b520218687e7e4c1 P d870164263bb44b6b780a557daea3656 [LEADER]: Connected to new peer: Peer: permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260130 02:58:23.720314 21085 raft_consensus.cc:493] T cfd12a72726846e587cff9d6a4d61bd6 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:23.733297 21085 raft_consensus.cc:515] T cfd12a72726846e587cff9d6a4d61bd6 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:23.735169 21085 leader_election.cc:290] T cfd12a72726846e587cff9d6a4d61bd6 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767)
I20260130 02:58:23.724413 21094 consensus_queue.cc:1048] T 2b0edaad6877460cb7982efc62b00b66 P d870164263bb44b6b780a557daea3656 [LEADER]: Connected to new peer: Peer: permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260130 02:58:23.738093 19222 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cfd12a72726846e587cff9d6a4d61bd6" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152" is_pre_election: true
I20260130 02:58:23.738819 19625 ts_tablet_manager.cc:1403] T 00ddbfe6218c44d788fbf65245a31dc6 P d870164263bb44b6b780a557daea3656: Time spent bootstrapping tablet: real 0.098s user 0.010s sys 0.000s
I20260130 02:58:23.740764 19807 raft_consensus.cc:359] T eb7a755ab62b4552bb8588264c5f7b75 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:23.741437 19807 raft_consensus.cc:385] T eb7a755ab62b4552bb8588264c5f7b75 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:23.741698 19807 raft_consensus.cc:740] T eb7a755ab62b4552bb8588264c5f7b75 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 371a36d219164974abdf848841528152, State: Initialized, Role: FOLLOWER
W20260130 02:58:23.747557 19463 leader_election.cc:343] T cfd12a72726846e587cff9d6a4d61bd6 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Illegal state: must be running to vote when last-logged opid is not known
W20260130 02:58:23.748242 19313 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.100.67:39571, user_credentials={real_user=slave}} blocked reactor thread for 322972us
I20260130 02:58:23.748015 19625 raft_consensus.cc:359] T 00ddbfe6218c44d788fbf65245a31dc6 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:23.748646 19625 raft_consensus.cc:385] T 00ddbfe6218c44d788fbf65245a31dc6 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:23.748885 19625 raft_consensus.cc:740] T 00ddbfe6218c44d788fbf65245a31dc6 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d870164263bb44b6b780a557daea3656, State: Initialized, Role: FOLLOWER
I20260130 02:58:23.751398 19807 consensus_queue.cc:260] T eb7a755ab62b4552bb8588264c5f7b75 P 371a36d219164974abdf848841528152 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:23.753013 19807 ts_tablet_manager.cc:1434] T eb7a755ab62b4552bb8588264c5f7b75 P 371a36d219164974abdf848841528152: Time spent starting tablet: real 0.036s user 0.002s sys 0.001s
I20260130 02:58:23.753878 19807 tablet_bootstrap.cc:492] T 0855d88806914a36b4bf6b49036f0ad0 P 371a36d219164974abdf848841528152: Bootstrap starting.
I20260130 02:58:23.761006 19625 consensus_queue.cc:260] T 00ddbfe6218c44d788fbf65245a31dc6 P d870164263bb44b6b780a557daea3656 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:23.762629 21094 raft_consensus.cc:493] T 2b8034a92b2043e5b57cf02f1ee6296d P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:23.763067 21094 raft_consensus.cc:515] T 2b8034a92b2043e5b57cf02f1ee6296d P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
W20260130 02:58:23.764276 19313 leader_election.cc:343] T cfd12a72726846e587cff9d6a4d61bd6 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:23.764671 19313 leader_election.cc:304] T cfd12a72726846e587cff9d6a4d61bd6 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78c2c46c7520492db35d51c244fb4f74; no voters: 371a36d219164974abdf848841528152, d870164263bb44b6b780a557daea3656
I20260130 02:58:23.764897 21094 leader_election.cc:290] T 2b8034a92b2043e5b57cf02f1ee6296d P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767)
I20260130 02:58:23.767179 19224 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2b8034a92b2043e5b57cf02f1ee6296d" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152" is_pre_election: true
W20260130 02:58:23.769269 19463 leader_election.cc:343] T 2b8034a92b2043e5b57cf02f1ee6296d P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:23.770015 19220 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1fc8f09d6329407c8078542b4123ef4d" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152" is_pre_election: true
I20260130 02:58:23.770129 21150 raft_consensus.cc:493] T 00000000000000000000000000000000 P 107f274cdda543038832f06a56ab9130 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 658299197a424f53a37f14fadf089ae4)
I20260130 02:58:23.771648 20899 ts_tablet_manager.cc:1929] T 2c79a3aecab844698e70834999e1caac P 78c2c46c7520492db35d51c244fb4f74: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260130 02:58:23.777586 19625 ts_tablet_manager.cc:1434] T 00ddbfe6218c44d788fbf65245a31dc6 P d870164263bb44b6b780a557daea3656: Time spent starting tablet: real 0.038s user 0.006s sys 0.000s
I20260130 02:58:23.778604 19625 tablet_bootstrap.cc:492] T dd377fab8c544bf69121dc9e9a060b29 P d870164263bb44b6b780a557daea3656: Bootstrap starting.
W20260130 02:58:23.778680 19159 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.100.67:39571, user_credentials={real_user=slave}} blocked reactor thread for 58929.1us
I20260130 02:58:23.779390 21150 raft_consensus.cc:515] T 00000000000000000000000000000000 P 107f274cdda543038832f06a56ab9130 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "658299197a424f53a37f14fadf089ae4" member_type: VOTER last_known_addr { host: "127.18.100.126" port: 38191 } } peers { permanent_uuid: "107f274cdda543038832f06a56ab9130" member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39711 } } peers { permanent_uuid: "3749baa287ed4f249329a74071781edd" member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34629 } }
I20260130 02:58:23.780004 19159 leader_election.cc:304] T 2b0edaad6877460cb7982efc62b00b66 P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 371a36d219164974abdf848841528152; no voters: 78c2c46c7520492db35d51c244fb4f74, d870164263bb44b6b780a557daea3656
I20260130 02:58:23.782126 21144 raft_consensus.cc:493] T eb7a755ab62b4552bb8588264c5f7b75 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:23.783423 21144 raft_consensus.cc:515] T eb7a755ab62b4552bb8588264c5f7b75 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:23.786787 21144 leader_election.cc:290] T eb7a755ab62b4552bb8588264c5f7b75 P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d870164263bb44b6b780a557daea3656 (127.18.100.67:39571), 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767)
I20260130 02:58:23.790974 21094 consensus_queue.cc:1048] T 657e9b0d8c05413f904129dce745710b P d870164263bb44b6b780a557daea3656 [LEADER]: Connected to new peer: Peer: permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260130 02:58:23.792783 18885 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "107f274cdda543038832f06a56ab9130" candidate_term: 3 candidate_status { last_received { term: 2 index: 371 } } ignore_live_leader: false dest_uuid: "658299197a424f53a37f14fadf089ae4" is_pre_election: true
I20260130 02:58:23.804153 18870 catalog_manager.cc:5746] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021) does not have the latest schema for tablet 345eaaa5625c48d099280b7fbb2e514b (table default.Table_c1886dd9a7934e7797e0e7e0b065d692 [id=7a2d5367224b492b844199088b01b149]). Expected version 1 got 0
I20260130 02:58:23.805757 18870 catalog_manager.cc:5746] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021) does not have the latest schema for tablet b1e8b78eff0d4a0b92059adb2de4a3ce (table default.Table_d0b4516d24384e5386c1debaf2543adb [id=f1342f9497b74e7a9ecad1de8082578d]). Expected version 1 got 0
I20260130 02:58:23.809614 21150 leader_election.cc:290] T 00000000000000000000000000000000 P 107f274cdda543038832f06a56ab9130 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 658299197a424f53a37f14fadf089ae4 (127.18.100.126:38191), 3749baa287ed4f249329a74071781edd (127.18.100.124:34629)
I20260130 02:58:23.810101 18870 catalog_manager.cc:5746] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021) does not have the latest schema for tablet f2d01bfdc9c4494386bd02bed655fe63 (table default.Table_aaf139c1a55d4a1783298ca12ae0d5c0 [id=d128767e303145a885eec3c5f714a27f]). Expected version 4 got 3
I20260130 02:58:23.814002 21094 raft_consensus.cc:493] T 306452e208ca4e719609f67345114a13 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:23.814452 21094 raft_consensus.cc:515] T 306452e208ca4e719609f67345114a13 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:23.819202 21094 leader_election.cc:290] T 306452e208ca4e719609f67345114a13 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767)
I20260130 02:58:23.864142 20807 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "107f274cdda543038832f06a56ab9130" candidate_term: 3 candidate_status { last_received { term: 2 index: 371 } } ignore_live_leader: false dest_uuid: "3749baa287ed4f249329a74071781edd" is_pre_election: true
I20260130 02:58:23.865005 20807 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 3749baa287ed4f249329a74071781edd [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 107f274cdda543038832f06a56ab9130 in term 2.
I20260130 02:58:23.866555 18935 leader_election.cc:304] T 00000000000000000000000000000000 P 107f274cdda543038832f06a56ab9130 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 107f274cdda543038832f06a56ab9130, 3749baa287ed4f249329a74071781edd; no voters: 658299197a424f53a37f14fadf089ae4
I20260130 02:58:23.867470 21150 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 107f274cdda543038832f06a56ab9130 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260130 02:58:23.867781 21150 raft_consensus.cc:493] T 00000000000000000000000000000000 P 107f274cdda543038832f06a56ab9130 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 658299197a424f53a37f14fadf089ae4)
I20260130 02:58:23.868098 21150 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 107f274cdda543038832f06a56ab9130 [term 2 FOLLOWER]: Advancing to term 3
I20260130 02:58:23.877265 21150 raft_consensus.cc:515] T 00000000000000000000000000000000 P 107f274cdda543038832f06a56ab9130 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "658299197a424f53a37f14fadf089ae4" member_type: VOTER last_known_addr { host: "127.18.100.126" port: 38191 } } peers { permanent_uuid: "107f274cdda543038832f06a56ab9130" member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39711 } } peers { permanent_uuid: "3749baa287ed4f249329a74071781edd" member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34629 } }
I20260130 02:58:23.815420 21121 raft_consensus.cc:2749] T cfd12a72726846e587cff9d6a4d61bd6 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260130 02:58:23.819972 19220 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "306452e208ca4e719609f67345114a13" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152" is_pre_election: true
I20260130 02:58:23.815778 20704 raft_consensus.cc:493] T 1129f72158ec4bae89da3c94cdb70839 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260130 02:58:23.879480 19463 leader_election.cc:343] T 306452e208ca4e719609f67345114a13 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:23.879264 20704 raft_consensus.cc:515] T 1129f72158ec4bae89da3c94cdb70839 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:23.881134 20704 leader_election.cc:290] T 1129f72158ec4bae89da3c94cdb70839 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d870164263bb44b6b780a557daea3656 (127.18.100.67:39571), 371a36d219164974abdf848841528152 (127.18.100.65:42021)
W20260130 02:58:23.818994 19314 leader_election.cc:343] T 1fc8f09d6329407c8078542b4123ef4d P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Illegal state: must be running to vote when last-logged opid is not known
W20260130 02:58:23.881865 19314 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.100.65:42021, user_credentials={real_user=slave}} blocked reactor thread for 62894.7us
I20260130 02:58:23.893270 20807 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "107f274cdda543038832f06a56ab9130" candidate_term: 3 candidate_status { last_received { term: 2 index: 371 } } ignore_live_leader: false dest_uuid: "3749baa287ed4f249329a74071781edd"
I20260130 02:58:23.893792 20807 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 3749baa287ed4f249329a74071781edd [term 2 FOLLOWER]: Advancing to term 3
I20260130 02:58:23.839560 19313 leader_election.cc:304] T ab6afa40408a4f458495abf20290e55b P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78c2c46c7520492db35d51c244fb4f74; no voters: 371a36d219164974abdf848841528152, d870164263bb44b6b780a557daea3656
W20260130 02:58:23.902547 19313 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.100.67:39571, user_credentials={real_user=slave}} blocked reactor thread for 63055.4us
I20260130 02:58:23.904006 19462 leader_election.cc:304] T aae68390dbbe4dc58ce3d3bd01725ae6 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d870164263bb44b6b780a557daea3656; no voters: 371a36d219164974abdf848841528152, 78c2c46c7520492db35d51c244fb4f74
I20260130 02:58:23.906448 19223 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1129f72158ec4bae89da3c94cdb70839" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152" is_pre_election: true
I20260130 02:58:23.907322 21150 leader_election.cc:290] T 00000000000000000000000000000000 P 107f274cdda543038832f06a56ab9130 [CANDIDATE]: Term 3 election: Requested vote from peers 658299197a424f53a37f14fadf089ae4 (127.18.100.126:38191), 3749baa287ed4f249329a74071781edd (127.18.100.124:34629)
I20260130 02:58:23.908802 21000 raft_consensus.cc:2749] T aae68390dbbe4dc58ce3d3bd01725ae6 P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260130 02:58:23.908951 20807 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 3749baa287ed4f249329a74071781edd [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 107f274cdda543038832f06a56ab9130 in term 3.
I20260130 02:58:23.909452 20704 raft_consensus.cc:2749] T ab6afa40408a4f458495abf20290e55b P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260130 02:58:23.909400 21000 consensus_queue.cc:1048] T e34d5da0ad864b2eaea0d575c982e447 P d870164263bb44b6b780a557daea3656 [LEADER]: Connected to new peer: Peer: permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260130 02:58:23.910033 18935 leader_election.cc:304] T 00000000000000000000000000000000 P 107f274cdda543038832f06a56ab9130 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 107f274cdda543038832f06a56ab9130, 3749baa287ed4f249329a74071781edd; no voters:
W20260130 02:58:23.911031 19314 leader_election.cc:343] T 1129f72158ec4bae89da3c94cdb70839 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:23.911038 21150 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 107f274cdda543038832f06a56ab9130 [term 3 FOLLOWER]: Leader election won for term 3
I20260130 02:58:23.916610 20899 log.cc:1199] T 2c79a3aecab844698e70834999e1caac P 78c2c46c7520492db35d51c244fb4f74: Deleting WAL directory at /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-1/wal/wals/2c79a3aecab844698e70834999e1caac
I20260130 02:58:23.923693 20899 ts_tablet_manager.cc:1950] T 2c79a3aecab844698e70834999e1caac P 78c2c46c7520492db35d51c244fb4f74: Deleting consensus metadata
I20260130 02:58:23.926697 21150 raft_consensus.cc:697] T 00000000000000000000000000000000 P 107f274cdda543038832f06a56ab9130 [term 3 LEADER]: Becoming Leader. State: Replica: 107f274cdda543038832f06a56ab9130, State: Running, Role: LEADER
I20260130 02:58:23.926800 20899 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60110 (request call id 339) took 1260 ms. Trace:
I20260130 02:58:23.927093 20899 rpcz_store.cc:276] 0130 02:58:22.666543 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:58:22.666653 (+ 110us) service_pool.cc:225] Handling call
0130 02:58:23.322060 (+655407us) ts_tablet_manager.cc:1151] Deleting tablet 2c79a3aecab844698e70834999e1caac
0130 02:58:23.322089 (+ 29us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0130 02:58:23.509338 (+187249us) tablet_metadata.cc:641] Metadata flushed
0130 02:58:23.771418 (+262080us) tablet_metadata.cc:641] Metadata flushed
0130 02:58:23.926736 (+155318us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":654886}
I20260130 02:58:23.927799 21150 consensus_queue.cc:237] T 00000000000000000000000000000000 P 107f274cdda543038832f06a56ab9130 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 371, Committed index: 371, Last appended: 2.371, Last appended by leader: 371, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "658299197a424f53a37f14fadf089ae4" member_type: VOTER last_known_addr { host: "127.18.100.126" port: 38191 } } peers { permanent_uuid: "107f274cdda543038832f06a56ab9130" member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39711 } } peers { permanent_uuid: "3749baa287ed4f249329a74071781edd" member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34629 } }
I20260130 02:58:23.936797 21078 ts_tablet_manager.cc:1929] T 43cec19c51ec4f05b63800c07f9534ef P 371a36d219164974abdf848841528152: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260130 02:58:23.937219 21078 log.cc:1199] T 43cec19c51ec4f05b63800c07f9534ef P 371a36d219164974abdf848841528152: Deleting WAL directory at /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-0/wal/wals/43cec19c51ec4f05b63800c07f9534ef
I20260130 02:58:23.938033 21078 ts_tablet_manager.cc:1950] T 43cec19c51ec4f05b63800c07f9534ef P 371a36d219164974abdf848841528152: Deleting consensus metadata
I20260130 02:58:23.938498 21094 consensus_queue.cc:1048] T a3258852aed24ed582a79f77860e6f0d P d870164263bb44b6b780a557daea3656 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260130 02:58:23.940303 21162 sys_catalog.cc:455] T 00000000000000000000000000000000 P 107f274cdda543038832f06a56ab9130 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 107f274cdda543038832f06a56ab9130. Latest consensus state: current_term: 3 leader_uuid: "107f274cdda543038832f06a56ab9130" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "658299197a424f53a37f14fadf089ae4" member_type: VOTER last_known_addr { host: "127.18.100.126" port: 38191 } } peers { permanent_uuid: "107f274cdda543038832f06a56ab9130" member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39711 } } peers { permanent_uuid: "3749baa287ed4f249329a74071781edd" member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34629 } } }
I20260130 02:58:23.943502 21162 sys_catalog.cc:458] T 00000000000000000000000000000000 P 107f274cdda543038832f06a56ab9130 [sys.catalog]: This master's current role is: LEADER
I20260130 02:58:23.946784 19524 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6d1f121f49cd47c2b8247daaf6874e01" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
I20260130 02:58:23.950538 21094 consensus_queue.cc:1048] T 133e9bfe175f44aab1a53114abe67e4d P d870164263bb44b6b780a557daea3656 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260130 02:58:23.957244 19625 tablet_bootstrap.cc:654] T dd377fab8c544bf69121dc9e9a060b29 P d870164263bb44b6b780a557daea3656: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:23.959367 18960 raft_consensus.cc:1240] T 00000000000000000000000000000000 P 107f274cdda543038832f06a56ab9130 [term 3 LEADER]: Rejecting Update request from peer 658299197a424f53a37f14fadf089ae4 for earlier term 2. Current term is 3. Ops: []
I20260130 02:58:23.893210 18885 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "107f274cdda543038832f06a56ab9130" candidate_term: 3 candidate_status { last_received { term: 2 index: 371 } } ignore_live_leader: false dest_uuid: "658299197a424f53a37f14fadf089ae4"
I20260130 02:58:23.963634 18870 catalog_manager.cc:5746] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021) does not have the latest schema for tablet 9fe0448bcc4a4346b3c67fc3f626ed3c (table default.Table_5c1c447e85524ac5a61ffd8a48509d36 [id=70aaf21ad9bd4d959f46939c6b4a4880]). Expected version 1 got 0
I20260130 02:58:23.968269 19462 leader_election.cc:304] T eb7a755ab62b4552bb8588264c5f7b75 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 78c2c46c7520492db35d51c244fb4f74, d870164263bb44b6b780a557daea3656; no voters: 371a36d219164974abdf848841528152
I20260130 02:58:23.971726 21094 raft_consensus.cc:2804] T eb7a755ab62b4552bb8588264c5f7b75 P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Leader election won for term 1
I20260130 02:58:23.829879 19807 tablet_bootstrap.cc:654] T 0855d88806914a36b4bf6b49036f0ad0 P 371a36d219164974abdf848841528152: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:23.839269 21142 raft_consensus.cc:2749] T 2b0edaad6877460cb7982efc62b00b66 P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260130 02:58:23.865712 19159 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.100.67:39571, user_credentials={real_user=slave}} blocked reactor thread for 84572.6us
I20260130 02:58:23.998802 18861 catalog_manager.cc:4985] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021): tablet 43cec19c51ec4f05b63800c07f9534ef (table default.Table_8f632ff84d624b72a10cb90fb9e78636 [id=3cf79022ea954f6397f0a0497e6e6a6c]) successfully deleted
I20260130 02:58:24.000931 21164 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260130 02:58:24.003548 21078 tablet_replica.cc:333] T ba91a508b1cb4430a4d8ca26946aecc8 P 371a36d219164974abdf848841528152: stopping tablet replica
I20260130 02:58:24.008555 21094 raft_consensus.cc:697] T eb7a755ab62b4552bb8588264c5f7b75 P d870164263bb44b6b780a557daea3656 [term 1 LEADER]: Becoming Leader. State: Replica: d870164263bb44b6b780a557daea3656, State: Running, Role: LEADER
I20260130 02:58:24.010282 21094 consensus_queue.cc:237] T eb7a755ab62b4552bb8588264c5f7b75 P d870164263bb44b6b780a557daea3656 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:24.016762 20777 ts_manager.cc:194] Registered new tserver with Master: 371a36d219164974abdf848841528152 (127.18.100.65:42021)
I20260130 02:58:24.024962 21160 consensus_queue.cc:1059] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "107f274cdda543038832f06a56ab9130" member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39711 }, Status: INVALID_TERM, Last received: 2.371, Next index: 372, Last known committed idx: 371, Time since last communication: 0.001s
I20260130 02:58:24.030939 21078 raft_consensus.cc:2243] T ba91a508b1cb4430a4d8ca26946aecc8 P 371a36d219164974abdf848841528152 [term 1 LEADER]: Raft consensus shutting down.
I20260130 02:58:24.031976 21078 raft_consensus.cc:2272] T ba91a508b1cb4430a4d8ca26946aecc8 P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260130 02:58:24.034082 21078 ts_tablet_manager.cc:1916] T ba91a508b1cb4430a4d8ca26946aecc8 P 371a36d219164974abdf848841528152: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260130 02:58:24.057327 19225 raft_consensus.cc:1240] T ca06f07bb98f45a7bf63aebc28348e08 P 371a36d219164974abdf848841528152 [term 2 FOLLOWER]: Rejecting Update request from peer d870164263bb44b6b780a557daea3656 for earlier term 1. Current term is 2. Ops: []
I20260130 02:58:23.955714 19840 tablet_bootstrap.cc:654] T 8b47653c7c3945a68e9bf4680bfc0496 P 78c2c46c7520492db35d51c244fb4f74: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:24.034845 18870 catalog_manager.cc:5654] T 13f0ace4d9de43abab08eca32cdb30c1 P 371a36d219164974abdf848841528152 reported cstate change: term changed from 0 to 1, leader changed from <none> to 371a36d219164974abdf848841528152 (127.18.100.65). New cstate: current_term: 1 leader_uuid: "371a36d219164974abdf848841528152" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } health_report { overall_health: UNKNOWN } } }
I20260130 02:58:24.059657 18870 catalog_manager.cc:5746] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021) does not have the latest schema for tablet 274530359b174ff79f046c2d4a6afa76 (table default.Table_c4f4904b6fac4b9880979cf0314e71c1 [id=c456c6418c93499a9d08f652720854ca]). Expected version 1 got 0
I20260130 02:58:24.060765 18870 catalog_manager.cc:5746] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021) does not have the latest schema for tablet 25fd48e693424537b91ba7c3fcc96878 (table default.Table_ebc8b8379c274c18b0a06a919dbe0ee0 [id=28156e0843674fce88f5914a3e4195ae]). Expected version 1 got 0
I20260130 02:58:24.061721 18870 catalog_manager.cc:5654] T 320dbe98a32a43b9899bbeed80b20f55 P 371a36d219164974abdf848841528152 reported cstate change: term changed from 0 to 1, leader changed from <none> to 371a36d219164974abdf848841528152 (127.18.100.65). New cstate: current_term: 1 leader_uuid: "371a36d219164974abdf848841528152" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } health_report { overall_health: UNKNOWN } } }
I20260130 02:58:24.062300 21000 consensus_queue.cc:1059] T ca06f07bb98f45a7bf63aebc28348e08 P d870164263bb44b6b780a557daea3656 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 }, Status: INVALID_TERM, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260130 02:58:24.063303 21000 raft_consensus.cc:3055] T ca06f07bb98f45a7bf63aebc28348e08 P d870164263bb44b6b780a557daea3656 [term 1 LEADER]: Stepping down as leader of term 1
I20260130 02:58:24.063633 21000 raft_consensus.cc:740] T ca06f07bb98f45a7bf63aebc28348e08 P d870164263bb44b6b780a557daea3656 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d870164263bb44b6b780a557daea3656, State: Running, Role: LEADER
I20260130 02:58:24.063752 18870 catalog_manager.cc:5654] T 838e49e5e8874db393bdc15a1c92f87e P 371a36d219164974abdf848841528152 reported cstate change: term changed from 0 to 1, leader changed from <none> to 371a36d219164974abdf848841528152 (127.18.100.65). New cstate: current_term: 1 leader_uuid: "371a36d219164974abdf848841528152" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } health_report { overall_health: HEALTHY } } }
I20260130 02:58:24.064453 21000 consensus_queue.cc:260] T ca06f07bb98f45a7bf63aebc28348e08 P d870164263bb44b6b780a557daea3656 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:24.065661 21000 raft_consensus.cc:3060] T ca06f07bb98f45a7bf63aebc28348e08 P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Advancing to term 2
I20260130 02:58:23.992043 20704 raft_consensus.cc:493] T 920f52b975ab480fbf42656171222f09 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:24.067457 20704 raft_consensus.cc:515] T 920f52b975ab480fbf42656171222f09 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:24.067845 18860 catalog_manager.cc:4985] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767): tablet 2c79a3aecab844698e70834999e1caac (table default.Table_12c2fe20cc3e45d29b7be37f900f68a9 [id=67ac4f4eda324e799b5867b2b011a16b]) successfully deleted
I20260130 02:58:24.068440 20899 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60110 (request call id 340) took 1298 ms. Trace:
I20260130 02:58:24.068805 20899 rpcz_store.cc:276] 0130 02:58:22.770399 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:58:22.770497 (+ 98us) service_pool.cc:225] Handling call
0130 02:58:24.068179 (+1297682us) ts_tablet_manager.cc:1151] Deleting tablet c7c621b99deb457998c44ab1644d3939
0130 02:58:24.068225 (+ 46us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0130 02:58:24.068379 (+ 154us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1297102}
I20260130 02:58:24.069172 20704 leader_election.cc:290] T 920f52b975ab480fbf42656171222f09 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d870164263bb44b6b780a557daea3656 (127.18.100.67:39571), 371a36d219164974abdf848841528152 (127.18.100.65:42021)
W20260130 02:58:24.069921 18860 catalog_manager.cc:4952] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767): delete failed for tablet c7c621b99deb457998c44ab1644d3939 because the tablet was not found. No further retry: Not found: Tablet not found: c7c621b99deb457998c44ab1644d3939
I20260130 02:58:24.070065 20899 tablet_replica.cc:333] T 43cec19c51ec4f05b63800c07f9534ef P 78c2c46c7520492db35d51c244fb4f74: stopping tablet replica
I20260130 02:58:24.070390 19225 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "920f52b975ab480fbf42656171222f09" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152" is_pre_election: true
I20260130 02:58:24.070909 20899 raft_consensus.cc:2243] T 43cec19c51ec4f05b63800c07f9534ef P 78c2c46c7520492db35d51c244fb4f74 [term 1 LEADER]: Raft consensus shutting down.
I20260130 02:58:24.071759 20899 raft_consensus.cc:2272] T 43cec19c51ec4f05b63800c07f9534ef P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260130 02:58:24.071696 19314 leader_election.cc:343] T 920f52b975ab480fbf42656171222f09 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:24.077090 19462 leader_election.cc:304] T 0855d88806914a36b4bf6b49036f0ad0 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 78c2c46c7520492db35d51c244fb4f74, d870164263bb44b6b780a557daea3656; no voters: 371a36d219164974abdf848841528152
I20260130 02:58:24.082724 19523 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c76f0dc7762c49c389fa6fef41f02141" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
I20260130 02:58:24.083570 19523 raft_consensus.cc:2468] T c76f0dc7762c49c389fa6fef41f02141 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78c2c46c7520492db35d51c244fb4f74 in term 0.
I20260130 02:58:24.084810 21167 raft_consensus.cc:2804] T 0855d88806914a36b4bf6b49036f0ad0 P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Leader election won for term 1
I20260130 02:58:24.085266 21167 raft_consensus.cc:697] T 0855d88806914a36b4bf6b49036f0ad0 P d870164263bb44b6b780a557daea3656 [term 1 LEADER]: Becoming Leader. State: Replica: d870164263bb44b6b780a557daea3656, State: Running, Role: LEADER
I20260130 02:58:24.085355 21085 raft_consensus.cc:493] T cd30d7adcc4545848e3501a205a10293 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:24.086301 19807 tablet_bootstrap.cc:492] T 0855d88806914a36b4bf6b49036f0ad0 P 371a36d219164974abdf848841528152: No bootstrap required, opened a new log
I20260130 02:58:24.086818 19807 ts_tablet_manager.cc:1403] T 0855d88806914a36b4bf6b49036f0ad0 P 371a36d219164974abdf848841528152: Time spent bootstrapping tablet: real 0.333s user 0.014s sys 0.002s
I20260130 02:58:24.086066 21167 consensus_queue.cc:237] T 0855d88806914a36b4bf6b49036f0ad0 P d870164263bb44b6b780a557daea3656 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:24.085929 21085 raft_consensus.cc:515] T cd30d7adcc4545848e3501a205a10293 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:24.092593 19218 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cd30d7adcc4545848e3501a205a10293" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152" is_pre_election: true
I20260130 02:58:24.092854 21085 leader_election.cc:290] T cd30d7adcc4545848e3501a205a10293 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767)
W20260130 02:58:24.094009 19463 leader_election.cc:343] T cd30d7adcc4545848e3501a205a10293 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:24.107265 19625 tablet_bootstrap.cc:492] T dd377fab8c544bf69121dc9e9a060b29 P d870164263bb44b6b780a557daea3656: No bootstrap required, opened a new log
I20260130 02:58:24.107699 19625 ts_tablet_manager.cc:1403] T dd377fab8c544bf69121dc9e9a060b29 P d870164263bb44b6b780a557daea3656: Time spent bootstrapping tablet: real 0.329s user 0.008s sys 0.005s
I20260130 02:58:24.109826 19625 raft_consensus.cc:359] T dd377fab8c544bf69121dc9e9a060b29 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:24.120306 19807 raft_consensus.cc:359] T 0855d88806914a36b4bf6b49036f0ad0 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:24.120980 19807 raft_consensus.cc:385] T 0855d88806914a36b4bf6b49036f0ad0 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:24.121239 19807 raft_consensus.cc:740] T 0855d88806914a36b4bf6b49036f0ad0 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 371a36d219164974abdf848841528152, State: Initialized, Role: FOLLOWER
I20260130 02:58:24.121908 19807 consensus_queue.cc:260] T 0855d88806914a36b4bf6b49036f0ad0 P 371a36d219164974abdf848841528152 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:24.125248 19840 tablet_bootstrap.cc:492] T 8b47653c7c3945a68e9bf4680bfc0496 P 78c2c46c7520492db35d51c244fb4f74: No bootstrap required, opened a new log
I20260130 02:58:24.125065 19524 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cd30d7adcc4545848e3501a205a10293" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
I20260130 02:58:24.125775 19840 ts_tablet_manager.cc:1403] T 8b47653c7c3945a68e9bf4680bfc0496 P 78c2c46c7520492db35d51c244fb4f74: Time spent bootstrapping tablet: real 0.410s user 0.012s sys 0.003s
I20260130 02:58:24.125644 19524 raft_consensus.cc:2468] T cd30d7adcc4545848e3501a205a10293 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78c2c46c7520492db35d51c244fb4f74 in term 0.
I20260130 02:58:24.128168 19840 raft_consensus.cc:359] T 8b47653c7c3945a68e9bf4680bfc0496 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:24.129129 19840 raft_consensus.cc:385] T 8b47653c7c3945a68e9bf4680bfc0496 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:24.131412 21164 catalog_manager.cc:679] Loaded metadata for table default.Table_5e70ab0e32bc4048bd19388f9d7cd302 [id=030c8ac23aac4a1fb7acc954d09434db]
I20260130 02:58:24.131516 19524 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1fc8f09d6329407c8078542b4123ef4d" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
I20260130 02:58:24.132145 19524 raft_consensus.cc:2468] T 1fc8f09d6329407c8078542b4123ef4d P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78c2c46c7520492db35d51c244fb4f74 in term 0.
I20260130 02:58:24.133009 21164 catalog_manager.cc:679] Loaded metadata for table default.Table_dabf999abe834151a0a5b6747a05cac3 [id=038ba9aa798741f6858fa09002d6c3d6]
I20260130 02:58:24.134481 21164 catalog_manager.cc:679] Loaded metadata for table default.Table_ed6e2e20d89542199e5cbc7842df964d [id=047c39bd14664d1099e043b973766d32]
I20260130 02:58:24.137353 19523 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1129f72158ec4bae89da3c94cdb70839" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
I20260130 02:58:24.143668 21164 catalog_manager.cc:679] Loaded metadata for table default.Table_fec49c37fdd045e7801325ace326f367 [id=06f1da4d6e0e46e4a141914a3aa68c07]
I20260130 02:58:24.064946 18870 catalog_manager.cc:5746] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021) does not have the latest schema for tablet 838e49e5e8874db393bdc15a1c92f87e (table default.Table_da3d581b15a94840898777f74611994e [id=b7e241ced53e4160ac6fa1f65ef3674e]). Expected version 1 got 0
I20260130 02:58:24.145243 21164 catalog_manager.cc:679] Loaded metadata for table default.Table_4ee03db7d0064d96a53d4f261bd12722 [id=07fe2931234d4888a524b52171b03f0a]
I20260130 02:58:24.145985 21160 consensus_queue.cc:1243] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4 [LEADER]: Peer 107f274cdda543038832f06a56ab9130 log is divergent from this leader: its last log entry 3.372 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 371
I20260130 02:58:24.146782 19840 raft_consensus.cc:740] T 8b47653c7c3945a68e9bf4680bfc0496 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78c2c46c7520492db35d51c244fb4f74, State: Initialized, Role: FOLLOWER
I20260130 02:58:24.146847 21164 catalog_manager.cc:679] Loaded metadata for table default.Table_0480357296ac4a27812d802ee4566351 [id=0da6197ab32e4645aa8d8787662ee5e6]
I20260130 02:58:24.147434 19840 consensus_queue.cc:260] T 8b47653c7c3945a68e9bf4680bfc0496 P 78c2c46c7520492db35d51c244fb4f74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:24.148240 21164 catalog_manager.cc:679] Loaded metadata for table default.Table_85611716686c4097bacb64df677d5793 [id=122fe719b1d64ec8b531783539f85df1]
I20260130 02:58:24.149729 21164 catalog_manager.cc:679] Loaded metadata for table default.Table_805e07e3546e447d8c8b8029477013cc [id=1515db389f284e2f9ed1a3cd0061d0f1]
I20260130 02:58:24.150126 19840 ts_tablet_manager.cc:1434] T 8b47653c7c3945a68e9bf4680bfc0496 P 78c2c46c7520492db35d51c244fb4f74: Time spent starting tablet: real 0.024s user 0.004s sys 0.000s
I20260130 02:58:24.151000 19840 tablet_bootstrap.cc:492] T 66a7487a7ab343a4abca5312afe78862 P 78c2c46c7520492db35d51c244fb4f74: Bootstrap starting.
I20260130 02:58:24.151155 21164 catalog_manager.cc:679] Loaded metadata for table default.Table_3a858a3cb886427cb8268f42c953ad4c [id=17150cb73e374ff9a1e92c43e0c3f2c1]
I20260130 02:58:24.152573 21164 catalog_manager.cc:679] Loaded metadata for table default.Table_ea2e0aa925a0486bb292304ca21bb819 [id=18580b9c54e448d1bdaae84f697f7b0b]
I20260130 02:58:24.110410 19625 raft_consensus.cc:385] T dd377fab8c544bf69121dc9e9a060b29 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:24.155805 19625 raft_consensus.cc:740] T dd377fab8c544bf69121dc9e9a060b29 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d870164263bb44b6b780a557daea3656, State: Initialized, Role: FOLLOWER
I20260130 02:58:24.155790 21164 catalog_manager.cc:679] Loaded metadata for table default.Table_9e942a1ce3944e9aa028e75656ce066d [id=18ed268eb9ad4450abf9874310ace074]
I20260130 02:58:24.156512 19625 consensus_queue.cc:260] T dd377fab8c544bf69121dc9e9a060b29 P d870164263bb44b6b780a557daea3656 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:24.106454 20777 ts_manager.cc:194] Registered new tserver with Master: 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767)
I20260130 02:58:24.158630 21173 raft_consensus.cc:3055] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4 [term 2 LEADER]: Stepping down as leader of term 2
I20260130 02:58:24.159055 21173 raft_consensus.cc:740] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 658299197a424f53a37f14fadf089ae4, State: Running, Role: LEADER
I20260130 02:58:24.159094 21164 catalog_manager.cc:679] Loaded metadata for table default.Table_a132cf962cfc4ed8a6a8c177e088daeb [id=1c4937ca6e014fec8596a3dc25779600]
I20260130 02:58:24.159989 21173 consensus_queue.cc:260] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 371, Committed index: 371, Last appended: 2.371, Last appended by leader: 371, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "658299197a424f53a37f14fadf089ae4" member_type: VOTER last_known_addr { host: "127.18.100.126" port: 38191 } } peers { permanent_uuid: "107f274cdda543038832f06a56ab9130" member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39711 } } peers { permanent_uuid: "3749baa287ed4f249329a74071781edd" member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34629 } }
I20260130 02:58:24.160583 21164 catalog_manager.cc:679] Loaded metadata for table default.Table_ddfb5df90fdb40a087ac434d088bbf0e [id=1e72b7dd669d4539b61884ba63ff7183]
I20260130 02:58:24.160918 19625 ts_tablet_manager.cc:1434] T dd377fab8c544bf69121dc9e9a060b29 P d870164263bb44b6b780a557daea3656: Time spent starting tablet: real 0.053s user 0.006s sys 0.000s
I20260130 02:58:24.161271 21173 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4 [term 2 FOLLOWER]: Advancing to term 3
I20260130 02:58:24.161630 21000 raft_consensus.cc:493] T 00ddbfe6218c44d788fbf65245a31dc6 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:24.162048 21000 raft_consensus.cc:515] T 00ddbfe6218c44d788fbf65245a31dc6 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:24.164753 19807 ts_tablet_manager.cc:1434] T 0855d88806914a36b4bf6b49036f0ad0 P 371a36d219164974abdf848841528152: Time spent starting tablet: real 0.078s user 0.005s sys 0.000s
I20260130 02:58:24.164940 19218 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00ddbfe6218c44d788fbf65245a31dc6" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152" is_pre_election: true
I20260130 02:58:24.165654 19807 tablet_bootstrap.cc:492] T 1fc8f09d6329407c8078542b4123ef4d P 371a36d219164974abdf848841528152: Bootstrap starting.
W20260130 02:58:24.166172 19463 leader_election.cc:343] T 00ddbfe6218c44d788fbf65245a31dc6 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:24.167035 21000 leader_election.cc:290] T 00ddbfe6218c44d788fbf65245a31dc6 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767)
I20260130 02:58:24.169735 19524 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "eb7a755ab62b4552bb8588264c5f7b75" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
I20260130 02:58:24.171751 21164 catalog_manager.cc:679] Loaded metadata for table default.Table_e03bdbb221e94bd6b535ddc02c863bad [id=2382eeebdc9540d5b588835705b2a4f1]
I20260130 02:58:24.173566 19625 tablet_bootstrap.cc:492] T 9896ff9364744857968e87e80e776dd2 P d870164263bb44b6b780a557daea3656: Bootstrap starting.
I20260130 02:58:24.174867 21164 catalog_manager.cc:679] Loaded metadata for table default.Table_ebc8b8379c274c18b0a06a919dbe0ee0 [id=28156e0843674fce88f5914a3e4195ae]
I20260130 02:58:24.176551 21164 catalog_manager.cc:679] Loaded metadata for table default.Table_bc168ff8b5f34cacbd398117647ba25a [id=2a21ee52b7d246f1b44eda02bf5a411e]
I20260130 02:58:24.177974 21164 catalog_manager.cc:679] Loaded metadata for table default.Table_f619de75da8e4d23b4d641d30285e0e7 [id=2dc3a57f750e4104b99f4589fdb29e3c]
I20260130 02:58:24.179426 21164 catalog_manager.cc:679] Loaded metadata for table default.Table_fa63c9eff99d4ac1b185a4c4ce2df031 [id=2e8b378422c74abc939955d110737316]
I20260130 02:58:24.181022 21164 catalog_manager.cc:679] Loaded metadata for table default.Table_1e42d571631143088cf33d4c3aafed28 [id=30ba2af6dffd4f7e8dcbfdd2f3798157]
I20260130 02:58:24.182479 21164 catalog_manager.cc:679] Loaded metadata for table default.Table_cda216a3bfc14ec0862b998bedd5b300 [id=35978c9d901c47eb81e565e37493b1ae]
I20260130 02:58:24.183943 21164 catalog_manager.cc:679] Loaded metadata for table default.Table_19677b801d1f4a63b22139b9fc80c4ef [id=3992cda3a7bb49ae86b8f84ee12f1c8b]
I20260130 02:58:24.185432 21164 catalog_manager.cc:679] Loaded metadata for table default.Table_0f3e44ae93bc4c028ed3929b94a49a24 [id=3b9c62fc8bbb44ffb7d66e9604974c30]
I20260130 02:58:24.186929 21164 catalog_manager.cc:679] Loaded metadata for table default.Table_8f632ff84d624b72a10cb90fb9e78636 [id=3cf79022ea954f6397f0a0497e6e6a6c]
I20260130 02:58:24.191020 21164 catalog_manager.cc:679] Loaded metadata for table default.Table_d764cbaecfee476ba5e734dd1e2f50fd [id=3f9236f0631a4e4c94ca7456f0478a45]
I20260130 02:58:24.192483 21164 catalog_manager.cc:679] Loaded metadata for table default.Table_0ca1d7370d2f416782b386c5b47111ab [id=3ffe1a273d4f452c835278a8e7f10102]
I20260130 02:58:24.193948 21164 catalog_manager.cc:679] Loaded metadata for table default.Table_f17d8e4d9f25449b84aa751af93936b2 [id=43254e466e7b4a2fb59fbcec68cfde36]
I20260130 02:58:24.195405 21164 catalog_manager.cc:679] Loaded metadata for table default.Table_6f53c17cfd144031b2d4b112fb2d88eb [id=443cbad5510d463fa8ca878b8628dbda]
I20260130 02:58:24.199709 21164 catalog_manager.cc:679] Loaded metadata for table default.Table_8896365e792f4d24b2c16c2d269b0a93 [id=44816e07033849479a7fea0f51d00fad]
I20260130 02:58:24.201190 21164 catalog_manager.cc:679] Loaded metadata for table default.Table_2d345d0f0957486c86c2cf2e34b00941 [id=4c3e0d3ecd5642a3b7fc7d20e2521418]
I20260130 02:58:24.212659 21164 catalog_manager.cc:679] Loaded metadata for table default.Table_8f311c5878744ef692b496ecfdfc3ede [id=4f499357303d464aafc17679ffcaaa07]
I20260130 02:58:24.214980 21164 catalog_manager.cc:679] Loaded metadata for table default.Table_d2a3438ed66741eab245ae6d6c5c2f37 [id=4fb5c26265e94083a8714be512fcd222]
I20260130 02:58:24.186821 20899 ts_tablet_manager.cc:1916] T 43cec19c51ec4f05b63800c07f9534ef P 78c2c46c7520492db35d51c244fb4f74: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260130 02:58:24.217283 21164 catalog_manager.cc:679] Loaded metadata for table default.Table_21178f657dca43a1912ce596834d1618 [id=53b79067a6034189aac5f1554df542c4]
I20260130 02:58:24.218760 21164 catalog_manager.cc:679] Loaded metadata for table default.Table_12b505700b3d44e28353172e1786e5c8 [id=562776405d6f4c7aab0622f3866bf45b]
I20260130 02:58:24.222098 19807 tablet_bootstrap.cc:654] T 1fc8f09d6329407c8078542b4123ef4d P 371a36d219164974abdf848841528152: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:24.224349 21164 catalog_manager.cc:679] Loaded metadata for table default.Table_aa4e5fd22945475da1dad48e1508595e [id=5b41834c6d8b4d89b773c2ea4c4ca7e6]
I20260130 02:58:24.228858 21164 catalog_manager.cc:679] Loaded metadata for table default.Table_9cb9a9f9e5d145a6a42d7ccbc42aced9 [id=5f1f179fadc247da9382dd7796dcf2c2]
I20260130 02:58:24.231106 21164 catalog_manager.cc:679] Loaded metadata for table default.Table_9d0cd9756483406081c8bdb18565a42f [id=5f6488df8ba94ef293b181328cc3b680]
I20260130 02:58:24.236542 21164 catalog_manager.cc:679] Loaded metadata for table default.Table_d95381c590c94802bdc83acef0f0df96 [id=64a8e0e079f34806aa68795f3d548c5e]
I20260130 02:58:24.239828 21000 raft_consensus.cc:493] T 39bcc41f10ae4241b370529364454586 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:24.240337 21000 raft_consensus.cc:515] T 39bcc41f10ae4241b370529364454586 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:24.242910 19625 tablet_bootstrap.cc:654] T 9896ff9364744857968e87e80e776dd2 P d870164263bb44b6b780a557daea3656: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:24.243211 19218 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "39bcc41f10ae4241b370529364454586" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152" is_pre_election: true
W20260130 02:58:24.244504 19463 leader_election.cc:343] T 39bcc41f10ae4241b370529364454586 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:24.245955 21078 ts_tablet_manager.cc:1929] T ba91a508b1cb4430a4d8ca26946aecc8 P 371a36d219164974abdf848841528152: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260130 02:58:24.246320 21078 log.cc:1199] T ba91a508b1cb4430a4d8ca26946aecc8 P 371a36d219164974abdf848841528152: Deleting WAL directory at /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-0/wal/wals/ba91a508b1cb4430a4d8ca26946aecc8
I20260130 02:58:24.247023 21078 ts_tablet_manager.cc:1950] T ba91a508b1cb4430a4d8ca26946aecc8 P 371a36d219164974abdf848841528152: Deleting consensus metadata
I20260130 02:58:24.248263 18861 catalog_manager.cc:4985] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021): tablet ba91a508b1cb4430a4d8ca26946aecc8 (table default.Table_aa496bf2019748f29823a9a253b1f3ca [id=780188e2335f43fbb11de8212eb38d56]) successfully deleted
I20260130 02:58:24.251870 21000 leader_election.cc:290] T 39bcc41f10ae4241b370529364454586 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767)
I20260130 02:58:24.256379 19372 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "eb7a755ab62b4552bb8588264c5f7b75" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74" is_pre_election: true
I20260130 02:58:24.257174 19372 raft_consensus.cc:2393] T eb7a755ab62b4552bb8588264c5f7b75 P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 371a36d219164974abdf848841528152 in current term 1: Already voted for candidate d870164263bb44b6b780a557daea3656 in this term.
I20260130 02:58:24.258347 20704 raft_consensus.cc:493] T 94355fc1338f479e8798070aff04fcba P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:24.258754 20704 raft_consensus.cc:515] T 94355fc1338f479e8798070aff04fcba P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:24.260144 21164 catalog_manager.cc:679] Loaded metadata for table default.Table_dee034aec0b24982b8dc2a3fa1b326f1 [id=66033e75b3814a8584311cf7e4a07ad3]
I20260130 02:58:24.266078 19218 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "94355fc1338f479e8798070aff04fcba" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152" is_pre_election: true
W20260130 02:58:24.271927 19314 leader_election.cc:343] T 94355fc1338f479e8798070aff04fcba P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:24.275074 21164 catalog_manager.cc:679] Loaded metadata for table default.Table_6a7faee4c75c4eb7862fb4c05afc17cb [id=676ece9dc36b418a8b1400e8406d6b98]
I20260130 02:58:24.284795 19159 leader_election.cc:304] T ca06f07bb98f45a7bf63aebc28348e08 P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 371a36d219164974abdf848841528152, 78c2c46c7520492db35d51c244fb4f74; no voters: d870164263bb44b6b780a557daea3656
I20260130 02:58:24.285549 21152 raft_consensus.cc:2804] T ca06f07bb98f45a7bf63aebc28348e08 P 371a36d219164974abdf848841528152 [term 2 FOLLOWER]: Leader election won for term 2
I20260130 02:58:24.285974 21152 raft_consensus.cc:697] T ca06f07bb98f45a7bf63aebc28348e08 P 371a36d219164974abdf848841528152 [term 2 LEADER]: Becoming Leader. State: Replica: 371a36d219164974abdf848841528152, State: Running, Role: LEADER
I20260130 02:58:24.286733 21152 consensus_queue.cc:237] T ca06f07bb98f45a7bf63aebc28348e08 P 371a36d219164974abdf848841528152 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:24.289063 21164 catalog_manager.cc:679] Loaded metadata for table default.Table_12c2fe20cc3e45d29b7be37f900f68a9 [id=67ac4f4eda324e799b5867b2b011a16b]
I20260130 02:58:24.291981 19807 tablet_bootstrap.cc:492] T 1fc8f09d6329407c8078542b4123ef4d P 371a36d219164974abdf848841528152: No bootstrap required, opened a new log
I20260130 02:58:24.292428 19807 ts_tablet_manager.cc:1403] T 1fc8f09d6329407c8078542b4123ef4d P 371a36d219164974abdf848841528152: Time spent bootstrapping tablet: real 0.127s user 0.011s sys 0.000s
I20260130 02:58:24.294664 19807 raft_consensus.cc:359] T 1fc8f09d6329407c8078542b4123ef4d P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
W20260130 02:58:24.297117 18861 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260130 02:58:24.299213 21164 catalog_manager.cc:679] Loaded metadata for table default.Table_6c46e6995d4741dcaaa6ed7bdc765e29 [id=6d91a161cf484239967af58db160c952]
I20260130 02:58:24.258227 21141 raft_consensus.cc:493] T 306452e208ca4e719609f67345114a13 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:24.299844 21141 raft_consensus.cc:515] T 306452e208ca4e719609f67345114a13 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:24.300840 21164 catalog_manager.cc:679] Loaded metadata for table default.Table_e8fd7516280f4dc4a73eb1f88fe916a2 [id=6efb836791414f3888f714aa8f65f0a4]
I20260130 02:58:24.301638 21141 leader_election.cc:290] T 306452e208ca4e719609f67345114a13 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), d870164263bb44b6b780a557daea3656 (127.18.100.67:39571)
I20260130 02:58:24.302322 21164 catalog_manager.cc:679] Loaded metadata for table default.Table_5c1c447e85524ac5a61ffd8a48509d36 [id=70aaf21ad9bd4d959f46939c6b4a4880]
I20260130 02:58:24.302693 20704 leader_election.cc:290] T 94355fc1338f479e8798070aff04fcba P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), d870164263bb44b6b780a557daea3656 (127.18.100.67:39571)
I20260130 02:58:24.305709 19840 tablet_bootstrap.cc:654] T 66a7487a7ab343a4abca5312afe78862 P 78c2c46c7520492db35d51c244fb4f74: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:24.307090 19218 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "306452e208ca4e719609f67345114a13" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152" is_pre_election: true
W20260130 02:58:24.313100 19314 leader_election.cc:343] T 306452e208ca4e719609f67345114a13 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:24.324061 21164 catalog_manager.cc:679] Loaded metadata for table default.Table_7fe8c354c0dc4cfba0e84ed33e4c2506 [id=7523d671e9ac4814acff5060fd282eea]
I20260130 02:58:24.325594 21164 catalog_manager.cc:679] Loaded metadata for table default.Table_f3e214a955f6447482099175b92ecb5c [id=76a688022fd9421a84de059b3c1da77a]
I20260130 02:58:24.330188 21164 catalog_manager.cc:679] Loaded metadata for table default.Table_dd22e9dc63f84d8688c0071f427f0b25 [id=77979d498aa040828de415129cb15a06]
I20260130 02:58:24.335441 21164 catalog_manager.cc:679] Loaded metadata for table default.Table_007e969f70d345a7bcb3dfbaabbc025d [id=77d4fa8705a3419cb0a84d3ca1bb4ce3]
I20260130 02:58:24.338706 21164 catalog_manager.cc:679] Loaded metadata for table default.Table_aa496bf2019748f29823a9a253b1f3ca [id=780188e2335f43fbb11de8212eb38d56]
I20260130 02:58:24.341857 19625 tablet_bootstrap.cc:492] T 9896ff9364744857968e87e80e776dd2 P d870164263bb44b6b780a557daea3656: No bootstrap required, opened a new log
I20260130 02:58:24.342262 19625 ts_tablet_manager.cc:1403] T 9896ff9364744857968e87e80e776dd2 P d870164263bb44b6b780a557daea3656: Time spent bootstrapping tablet: real 0.169s user 0.008s sys 0.004s
I20260130 02:58:24.342381 21080 catalog_manager.cc:1269] Loaded cluster ID: db1c0e7709154c10a8c9a0ea68ffe38b
I20260130 02:58:24.342716 21080 catalog_manager.cc:1562] T 00000000000000000000000000000000 P 3749baa287ed4f249329a74071781edd: loading cluster ID for follower catalog manager: success
I20260130 02:58:24.344990 21164 catalog_manager.cc:679] Loaded metadata for table default.Table_c1886dd9a7934e7797e0e7e0b065d692 [id=7a2d5367224b492b844199088b01b149]
I20260130 02:58:24.355187 21164 catalog_manager.cc:679] Loaded metadata for table default.Table_69267d31df644f4396a44130fa3d499d [id=7efa53800a974e31a38a4c60c0a1b466]
I20260130 02:58:24.356745 21164 catalog_manager.cc:679] Loaded metadata for table default.Table_b85cb876f25740bfaaab01949ac26b38 [id=839e292e040c4aeb832f3fc0258c32ec]
I20260130 02:58:24.358274 21164 catalog_manager.cc:679] Loaded metadata for table default.Table_1ae68ae3f0404843aca3c72f02678ac6 [id=8a9182f49e18489fafec8dca190c1b2f]
I20260130 02:58:24.361429 21164 catalog_manager.cc:679] Loaded metadata for table default.Table_333d2efbf068485a871f566d50e563ce [id=8b7db40afb894ae99625a756c75ef1de]
I20260130 02:58:24.362843 21164 catalog_manager.cc:679] Loaded metadata for table default.Table_1703c69112744c0e951c1ac9667bbac6 [id=8d76f4328edc491291491c78ba205d30]
I20260130 02:58:24.364135 21164 catalog_manager.cc:679] Loaded metadata for table default.Table_6773a0ad5d6a429086a5818d731f3f4a [id=90b72499c7174fd580914d7fe3714891]
I20260130 02:58:24.368273 21164 catalog_manager.cc:679] Loaded metadata for table default.Table_829d1e187e8b4db5b8f4f835e84246e5 [id=a24b4761037d47a383f454a82517131f]
I20260130 02:58:24.369717 21164 catalog_manager.cc:679] Loaded metadata for table default.Table_f8856f06ce97440bae4abdf34164716d [id=a3303ada20a94f99b2f37a02eec361ef]
I20260130 02:58:24.369889 19807 raft_consensus.cc:385] T 1fc8f09d6329407c8078542b4123ef4d P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:24.370235 19807 raft_consensus.cc:740] T 1fc8f09d6329407c8078542b4123ef4d P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 371a36d219164974abdf848841528152, State: Initialized, Role: FOLLOWER
I20260130 02:58:24.371133 21164 catalog_manager.cc:679] Loaded metadata for table default.Table_2e81f0a0cbc449a397141b717db17025 [id=a9c919f57c0c437cb35a6921103e45a6]
I20260130 02:58:24.371371 19375 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "94355fc1338f479e8798070aff04fcba" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74" is_pre_election: true
I20260130 02:58:24.371929 19375 raft_consensus.cc:2468] T 94355fc1338f479e8798070aff04fcba P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d870164263bb44b6b780a557daea3656 in term 0.
I20260130 02:58:24.372411 21164 catalog_manager.cc:679] Loaded metadata for table default.Table_7c48a0d054354501947d11c5fd7b1864 [id=aa4ad1394ab344df8c54ba0031af5590]
I20260130 02:58:24.370858 19807 consensus_queue.cc:260] T 1fc8f09d6329407c8078542b4123ef4d P 371a36d219164974abdf848841528152 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:24.374421 21164 catalog_manager.cc:679] Loaded metadata for table default.Table_477fb92965284222aa1ecaa0107d1ec2 [id=ad82881a19e748edb569e74d692bb2bc]
I20260130 02:58:24.377046 21164 catalog_manager.cc:679] Loaded metadata for table default.Table_2414288f0dc54e5f835f876060927a9b [id=b0c6f2d9483c47da8ff3e65f0400a593]
I20260130 02:58:24.374446 19625 raft_consensus.cc:359] T 9896ff9364744857968e87e80e776dd2 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:24.377839 19625 raft_consensus.cc:385] T 9896ff9364744857968e87e80e776dd2 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:24.378435 21164 catalog_manager.cc:679] Loaded metadata for table default.Table_526677c8ef4144bd9de7a54a5e78e2be [id=b16908b75587459597a978c71756451c]
I20260130 02:58:24.378806 19625 raft_consensus.cc:740] T 9896ff9364744857968e87e80e776dd2 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d870164263bb44b6b780a557daea3656, State: Initialized, Role: FOLLOWER
I20260130 02:58:24.379768 21164 catalog_manager.cc:679] Loaded metadata for table default.Table_78b080efddd94878b9d8b77b5ef3af35 [id=b654822c58224a859ad2dcdd8d0123c2]
I20260130 02:58:24.381130 21164 catalog_manager.cc:679] Loaded metadata for table default.Table_7d1c27ab3515406ba5b5333c90df093e [id=b6fabeeae79941dc8094d0f2b51704f9]
I20260130 02:58:24.382424 21164 catalog_manager.cc:679] Loaded metadata for table default.Table_95f0bff8e17d4ca995c0af0d0c24e84d [id=b7795905d0c440579c59d6ac00473f5a]
I20260130 02:58:24.382216 19372 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1fc8f09d6329407c8078542b4123ef4d" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74" is_pre_election: true
I20260130 02:58:24.382879 19372 raft_consensus.cc:2468] T 1fc8f09d6329407c8078542b4123ef4d P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d870164263bb44b6b780a557daea3656 in term 0.
I20260130 02:58:24.385229 21164 catalog_manager.cc:679] Loaded metadata for table default.Table_da3d581b15a94840898777f74611994e [id=b7e241ced53e4160ac6fa1f65ef3674e]
I20260130 02:58:24.380745 19625 consensus_queue.cc:260] T 9896ff9364744857968e87e80e776dd2 P d870164263bb44b6b780a557daea3656 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:24.386628 19372 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cfd12a72726846e587cff9d6a4d61bd6" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74" is_pre_election: true
I20260130 02:58:24.387255 21164 catalog_manager.cc:679] Loaded metadata for table default.Table_7e6ab9964f4c4e0f906d875a937a781d [id=bcf41af45d284bd89800977d14194812]
I20260130 02:58:24.387192 19372 raft_consensus.cc:2468] T cfd12a72726846e587cff9d6a4d61bd6 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d870164263bb44b6b780a557daea3656 in term 0.
I20260130 02:58:24.388535 21164 catalog_manager.cc:679] Loaded metadata for table default.Table_d3f82845a18f4a4a830460cb1cbbb2de [id=c3d0c57ad165460eae9c20c32557fd2e]
I20260130 02:58:24.388438 19372 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2b8034a92b2043e5b57cf02f1ee6296d" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74" is_pre_election: true
I20260130 02:58:24.389006 19372 raft_consensus.cc:2468] T 2b8034a92b2043e5b57cf02f1ee6296d P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d870164263bb44b6b780a557daea3656 in term 0.
I20260130 02:58:24.389798 21164 catalog_manager.cc:679] Loaded metadata for table default.Table_c4f4904b6fac4b9880979cf0314e71c1 [id=c456c6418c93499a9d08f652720854ca]
I20260130 02:58:24.391700 21164 catalog_manager.cc:679] Loaded metadata for table default.Table_81553b46d7804fd99ef36f431793b001 [id=cdfc1cfcc48f41f8a7a13f865174c3d8]
I20260130 02:58:24.391917 19840 tablet_bootstrap.cc:492] T 66a7487a7ab343a4abca5312afe78862 P 78c2c46c7520492db35d51c244fb4f74: No bootstrap required, opened a new log
I20260130 02:58:24.392294 19840 ts_tablet_manager.cc:1403] T 66a7487a7ab343a4abca5312afe78862 P 78c2c46c7520492db35d51c244fb4f74: Time spent bootstrapping tablet: real 0.241s user 0.006s sys 0.006s
I20260130 02:58:24.393524 21164 catalog_manager.cc:679] Loaded metadata for table default.Table_aaf139c1a55d4a1783298ca12ae0d5c0 [id=d128767e303145a885eec3c5f714a27f]
I20260130 02:58:24.394847 21164 catalog_manager.cc:679] Loaded metadata for table default.Table_94b23422c82d419d97dc097a3efb92fc [id=d2f35437eae140dab4150079eae46377]
I20260130 02:58:24.394526 19840 raft_consensus.cc:359] T 66a7487a7ab343a4abca5312afe78862 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:24.395592 19840 raft_consensus.cc:385] T 66a7487a7ab343a4abca5312afe78862 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:24.395881 19840 raft_consensus.cc:740] T 66a7487a7ab343a4abca5312afe78862 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78c2c46c7520492db35d51c244fb4f74, State: Initialized, Role: FOLLOWER
I20260130 02:58:24.395983 19625 ts_tablet_manager.cc:1434] T 9896ff9364744857968e87e80e776dd2 P d870164263bb44b6b780a557daea3656: Time spent starting tablet: real 0.053s user 0.006s sys 0.000s
I20260130 02:58:24.396759 21164 catalog_manager.cc:679] Loaded metadata for table default.Table_e894e5560f194692bac730600c6bf5f4 [id=d5e1b2a45122429aabb4fd8653d71f0f]
I20260130 02:58:24.396643 19840 consensus_queue.cc:260] T 66a7487a7ab343a4abca5312afe78862 P 78c2c46c7520492db35d51c244fb4f74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:24.397954 21164 catalog_manager.cc:679] Loaded metadata for table default.Table_22d10332a83f4700ac21892bab67f46b [id=da3a9cb1f6d5419abc1afc93659b5324]
I20260130 02:58:24.399241 21164 catalog_manager.cc:679] Loaded metadata for table default.Table_9f83f87711d94e8394abe1925ac84ad1 [id=db23c1be42c44b6f9eb774f14d2b8fd6]
I20260130 02:58:24.398643 19840 ts_tablet_manager.cc:1434] T 66a7487a7ab343a4abca5312afe78862 P 78c2c46c7520492db35d51c244fb4f74: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260130 02:58:24.400255 19840 tablet_bootstrap.cc:492] T 6f5a90b75a0f4851b5db672c41253f7b P 78c2c46c7520492db35d51c244fb4f74: Bootstrap starting.
I20260130 02:58:24.400415 21164 catalog_manager.cc:679] Loaded metadata for table default.Table_109c8b6c17d4442b939bde9441d0476f [id=dcf82ea15d6044c2b023f81f0a579ade]
I20260130 02:58:24.402284 21164 catalog_manager.cc:679] Loaded metadata for table default.Table_baaf376733384fdbbac49ec8f2357432 [id=dea97f19a50141878f97820194afa59e]
I20260130 02:58:24.404187 21164 catalog_manager.cc:679] Loaded metadata for table default.Table_8c27a7d10b7847e78029f7e95321e9b6 [id=e31c1bf0c3b9457fa63efe71138c259c]
I20260130 02:58:24.405418 21164 catalog_manager.cc:679] Loaded metadata for table default.Table_92836e7fcf3a48d8bac2160c2ff7c482 [id=e3ae4b66fa0048668cb1e4d9da7d16fa]
I20260130 02:58:24.406687 21164 catalog_manager.cc:679] Loaded metadata for table default.Table_d24e400f7451443a82d23140a90adc57 [id=e478df970dac40f7a576ce584c3abf2c]
W20260130 02:58:24.405488 19159 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.100.66:40767, user_credentials={real_user=slave}} blocked reactor thread for 46223.8us
I20260130 02:58:24.407353 19807 ts_tablet_manager.cc:1434] T 1fc8f09d6329407c8078542b4123ef4d P 371a36d219164974abdf848841528152: Time spent starting tablet: real 0.115s user 0.005s sys 0.000s
I20260130 02:58:24.407882 21164 catalog_manager.cc:679] Loaded metadata for table default.Table_2f60fd0292ab40c8a3ae9b842cae8faf [id=e648e521ef5044a0b1967452cc3a31b2]
I20260130 02:58:24.408205 19807 tablet_bootstrap.cc:492] T 306452e208ca4e719609f67345114a13 P 371a36d219164974abdf848841528152: Bootstrap starting.
I20260130 02:58:24.409107 21164 catalog_manager.cc:679] Loaded metadata for table default.Table_ba19837ced4c45d79be506e93308449b [id=e732719fbe544e38a15b35017010c7b3]
I20260130 02:58:24.410310 21169 raft_consensus.cc:493] T 0855d88806914a36b4bf6b49036f0ad0 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:24.410791 21169 raft_consensus.cc:515] T 0855d88806914a36b4bf6b49036f0ad0 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:24.412675 21169 leader_election.cc:290] T 0855d88806914a36b4bf6b49036f0ad0 P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d870164263bb44b6b780a557daea3656 (127.18.100.67:39571), 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767)
I20260130 02:58:24.414023 21164 catalog_manager.cc:679] Loaded metadata for table default.Table_c238448062ad4cef8ddd0bd199d5e7f2 [id=e8d3be5336974867a5d65d1ceb4433d6]
I20260130 02:58:24.415355 21164 catalog_manager.cc:679] Loaded metadata for table default.Table_a3ecda6d64704b7fad065a5845683ad8 [id=e93360fd2af141a28d92401ed449c19d]
I20260130 02:58:24.417284 21164 catalog_manager.cc:679] Loaded metadata for table default.Table_3046889233c24f0a8e8058e1591d63b8 [id=edd6505c0bf3462cbf65c3df5649c16c]
I20260130 02:58:24.419961 19625 tablet_bootstrap.cc:492] T 0d14fb95ace9447f810ccbed4d5befdf P d870164263bb44b6b780a557daea3656: Bootstrap starting.
I20260130 02:58:24.423337 20807 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 3749baa287ed4f249329a74071781edd [term 3 FOLLOWER]: Refusing update from remote peer 107f274cdda543038832f06a56ab9130: Log matching property violated. Preceding OpId in replica: term: 2 index: 333. Preceding OpId from leader: term: 3 index: 372. (index mismatch)
I20260130 02:58:24.425784 19807 tablet_bootstrap.cc:654] T 306452e208ca4e719609f67345114a13 P 371a36d219164974abdf848841528152: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:24.429325 19372 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "306452e208ca4e719609f67345114a13" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74" is_pre_election: true
I20260130 02:58:24.430080 19372 raft_consensus.cc:2468] T 306452e208ca4e719609f67345114a13 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d870164263bb44b6b780a557daea3656 in term 0.
I20260130 02:58:24.431808 21162 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 107f274cdda543038832f06a56ab9130 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3749baa287ed4f249329a74071781edd" member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 372, Last known committed idx: 333, Time since last communication: 0.000s
I20260130 02:58:24.441345 21164 catalog_manager.cc:679] Loaded metadata for table default.Table_d0b4516d24384e5386c1debaf2543adb [id=f1342f9497b74e7a9ecad1de8082578d]
I20260130 02:58:24.447980 21164 catalog_manager.cc:679] Loaded metadata for table default.Table_5d5e26f93a5a4fb084831add317d90f9 [id=f237b1c90a1743d597d87c5cf1abc89a]
I20260130 02:58:24.450040 21164 catalog_manager.cc:679] Loaded metadata for table default.Table_1e9e20a1bfec4bcf81f9eb6219c33d59 [id=f675b7a3ec744533903d8834eeed51d7]
I20260130 02:58:24.463819 21164 catalog_manager.cc:679] Loaded metadata for table default.Table_f246882197f44c69b72ca421cbdf201f [id=f778784dc88a4e4890113916098ad20f]
I20260130 02:58:24.463792 21000 consensus_queue.cc:1048] T ab6afa40408a4f458495abf20290e55b P d870164263bb44b6b780a557daea3656 [LEADER]: Connected to new peer: Peer: permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260130 02:58:24.470010 21164 catalog_manager.cc:679] Loaded metadata for table default.Table_2c737f14bf854cfca58d4a9f83fb2e9e [id=fa4e8955bebb40b6adf6ed4ce0e43bfb]
I20260130 02:58:24.471421 21164 catalog_manager.cc:679] Loaded metadata for table default.Table_e2480022cdae4429b710395eebae7fec [id=fa72411018324474aa9a7265d9f1a2e4]
I20260130 02:58:24.472563 19364 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cd30d7adcc4545848e3501a205a10293" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74" is_pre_election: true
I20260130 02:58:24.472764 21164 catalog_manager.cc:679] Loaded metadata for table default.Table_e4d07634d84a4678b1c94374e3adc476 [id=fcbd7e3a7fc64d89aa740494d7e1a0f7]
I20260130 02:58:24.472889 19370 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00ddbfe6218c44d788fbf65245a31dc6" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74" is_pre_election: true
I20260130 02:58:24.473130 19364 raft_consensus.cc:2468] T cd30d7adcc4545848e3501a205a10293 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d870164263bb44b6b780a557daea3656 in term 0.
I20260130 02:58:24.473469 19370 raft_consensus.cc:2468] T 00ddbfe6218c44d788fbf65245a31dc6 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d870164263bb44b6b780a557daea3656 in term 0.
I20260130 02:58:24.473968 21164 catalog_manager.cc:679] Loaded metadata for table default.Table_203b3a960d3e4b5cb978b6168435f3a4 [id=ffe264c47e31462a8deaf81a8b96d34c]
I20260130 02:58:24.474835 19370 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "39bcc41f10ae4241b370529364454586" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74" is_pre_election: true
I20260130 02:58:24.475364 19370 raft_consensus.cc:2468] T 39bcc41f10ae4241b370529364454586 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d870164263bb44b6b780a557daea3656 in term 0.
I20260130 02:58:24.487679 21000 raft_consensus.cc:493] T dd377fab8c544bf69121dc9e9a060b29 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:24.488131 21000 raft_consensus.cc:515] T dd377fab8c544bf69121dc9e9a060b29 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:24.490621 21000 leader_election.cc:290] T dd377fab8c544bf69121dc9e9a060b29 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767)
I20260130 02:58:24.492120 19218 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dd377fab8c544bf69121dc9e9a060b29" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152" is_pre_election: true
I20260130 02:58:24.497164 19527 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "920f52b975ab480fbf42656171222f09" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
I20260130 02:58:24.516040 19524 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "94355fc1338f479e8798070aff04fcba" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
I20260130 02:58:24.516481 18885 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4 [term 3 FOLLOWER]: Refusing update from remote peer 107f274cdda543038832f06a56ab9130: Log matching property violated. Preceding OpId in replica: term: 2 index: 371. Preceding OpId from leader: term: 3 index: 372. (index mismatch)
I20260130 02:58:24.516595 19524 raft_consensus.cc:2468] T 94355fc1338f479e8798070aff04fcba P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78c2c46c7520492db35d51c244fb4f74 in term 0.
I20260130 02:58:24.517935 19462 leader_election.cc:304] T 94355fc1338f479e8798070aff04fcba P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 78c2c46c7520492db35d51c244fb4f74, d870164263bb44b6b780a557daea3656; no voters: 371a36d219164974abdf848841528152
I20260130 02:58:24.519508 19462 leader_election.cc:304] T 1fc8f09d6329407c8078542b4123ef4d P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 78c2c46c7520492db35d51c244fb4f74, d870164263bb44b6b780a557daea3656; no voters: 371a36d219164974abdf848841528152
I20260130 02:58:24.520666 21167 raft_consensus.cc:2804] T 94355fc1338f479e8798070aff04fcba P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260130 02:58:24.520958 21167 raft_consensus.cc:493] T 94355fc1338f479e8798070aff04fcba P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260130 02:58:24.521250 21167 raft_consensus.cc:3060] T 94355fc1338f479e8798070aff04fcba P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:24.527508 21094 consensus_queue.cc:1048] T e34d5da0ad864b2eaea0d575c982e447 P d870164263bb44b6b780a557daea3656 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260130 02:58:24.528730 21094 raft_consensus.cc:2804] T 1fc8f09d6329407c8078542b4123ef4d P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260130 02:58:24.529039 21094 raft_consensus.cc:493] T 1fc8f09d6329407c8078542b4123ef4d P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260130 02:58:24.529335 21094 raft_consensus.cc:3060] T 1fc8f09d6329407c8078542b4123ef4d P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:24.532771 21162 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 107f274cdda543038832f06a56ab9130 [LEADER]: Connected to new peer: Peer: permanent_uuid: "658299197a424f53a37f14fadf089ae4" member_type: VOTER last_known_addr { host: "127.18.100.126" port: 38191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 372, Last known committed idx: 371, Time since last communication: 0.000s
I20260130 02:58:24.536516 19527 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "306452e208ca4e719609f67345114a13" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
I20260130 02:58:24.537078 19527 raft_consensus.cc:2468] T 306452e208ca4e719609f67345114a13 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78c2c46c7520492db35d51c244fb4f74 in term 0.
I20260130 02:58:24.542874 21173 sys_catalog.cc:455] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 107f274cdda543038832f06a56ab9130. Latest consensus state: current_term: 3 leader_uuid: "107f274cdda543038832f06a56ab9130" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "658299197a424f53a37f14fadf089ae4" member_type: VOTER last_known_addr { host: "127.18.100.126" port: 38191 } } peers { permanent_uuid: "107f274cdda543038832f06a56ab9130" member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39711 } } peers { permanent_uuid: "3749baa287ed4f249329a74071781edd" member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34629 } } }
I20260130 02:58:24.549919 21173 sys_catalog.cc:458] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4 [sys.catalog]: This master's current role is: FOLLOWER
I20260130 02:58:24.521420 21188 tablet.cc:1722] T f2d01bfdc9c4494386bd02bed655fe63 P 78c2c46c7520492db35d51c244fb4f74: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20260130 02:58:24.554404 19840 tablet_bootstrap.cc:654] T 6f5a90b75a0f4851b5db672c41253f7b P 78c2c46c7520492db35d51c244fb4f74: Neither blocks nor log segments found. Creating new log.
W20260130 02:58:24.493382 19463 leader_election.cc:343] T dd377fab8c544bf69121dc9e9a060b29 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Illegal state: must be running to vote when last-logged opid is not known
W20260130 02:58:24.559954 19463 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.100.65:42021, user_credentials={real_user=slave}} blocked reactor thread for 66601.2us
I20260130 02:58:24.560494 21164 tablet_loader.cc:96] loaded metadata for tablet 00ddbfe6218c44d788fbf65245a31dc6 (table default.Table_9cb9a9f9e5d145a6a42d7ccbc42aced9 [id=5f1f179fadc247da9382dd7796dcf2c2])
I20260130 02:58:24.563449 21164 tablet_loader.cc:96] loaded metadata for tablet 03025d13f0e545cfa6288677a2715343 (table default.Table_21178f657dca43a1912ce596834d1618 [id=53b79067a6034189aac5f1554df542c4])
I20260130 02:58:24.566774 21164 tablet_loader.cc:96] loaded metadata for tablet 0855d88806914a36b4bf6b49036f0ad0 (table default.Table_6773a0ad5d6a429086a5818d731f3f4a [id=90b72499c7174fd580914d7fe3714891])
I20260130 02:58:24.567850 21164 tablet_loader.cc:96] loaded metadata for tablet 0a763e48583c4e748a46a325746ce69a (table default.Table_94b23422c82d419d97dc097a3efb92fc [id=d2f35437eae140dab4150079eae46377])
I20260130 02:58:24.568708 19372 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0855d88806914a36b4bf6b49036f0ad0" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74" is_pre_election: true
I20260130 02:58:24.568929 21164 tablet_loader.cc:96] loaded metadata for tablet 0b1877e5f9244913bcd639f1336df7ee (table default.Table_19677b801d1f4a63b22139b9fc80c4ef [id=3992cda3a7bb49ae86b8f84ee12f1c8b])
I20260130 02:58:24.569455 19625 tablet_bootstrap.cc:654] T 0d14fb95ace9447f810ccbed4d5befdf P d870164263bb44b6b780a557daea3656: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:24.569536 19372 raft_consensus.cc:2393] T 0855d88806914a36b4bf6b49036f0ad0 P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 371a36d219164974abdf848841528152 in current term 1: Already voted for candidate d870164263bb44b6b780a557daea3656 in this term.
I20260130 02:58:24.570035 21164 tablet_loader.cc:96] loaded metadata for tablet 0d14fb95ace9447f810ccbed4d5befdf (table default.Table_c238448062ad4cef8ddd0bd199d5e7f2 [id=e8d3be5336974867a5d65d1ceb4433d6])
I20260130 02:58:24.571173 21164 tablet_loader.cc:96] loaded metadata for tablet 0e8018c094024c4788c484e8f5466e00 (table default.Table_f17d8e4d9f25449b84aa751af93936b2 [id=43254e466e7b4a2fb59fbcec68cfde36])
I20260130 02:58:24.572235 21164 tablet_loader.cc:96] loaded metadata for tablet 111287b5557d4abda1c5cd560fb0d019 (table default.Table_fec49c37fdd045e7801325ace326f367 [id=06f1da4d6e0e46e4a141914a3aa68c07])
I20260130 02:58:24.575484 21164 tablet_loader.cc:96] loaded metadata for tablet 127a84346b9f473d9e7f739b1e65feb8 (table default.Table_805e07e3546e447d8c8b8029477013cc [id=1515db389f284e2f9ed1a3cd0061d0f1])
I20260130 02:58:24.576597 21164 tablet_loader.cc:96] loaded metadata for tablet 12c3f3ec523040e7b7cd356e3da57e0d (table default.Table_81553b46d7804fd99ef36f431793b001 [id=cdfc1cfcc48f41f8a7a13f865174c3d8])
I20260130 02:58:24.577631 21164 tablet_loader.cc:96] loaded metadata for tablet 133e9bfe175f44aab1a53114abe67e4d (table default.Table_d3f82845a18f4a4a830460cb1cbbb2de [id=c3d0c57ad165460eae9c20c32557fd2e])
I20260130 02:58:24.578697 21164 tablet_loader.cc:96] loaded metadata for tablet 13f0ace4d9de43abab08eca32cdb30c1 (table default.Table_9d0cd9756483406081c8bdb18565a42f [id=5f6488df8ba94ef293b181328cc3b680])
I20260130 02:58:24.579732 21164 tablet_loader.cc:96] loaded metadata for tablet 155ff0c35fc44cb292a28407e6ed937d (table default.Table_dabf999abe834151a0a5b6747a05cac3 [id=038ba9aa798741f6858fa09002d6c3d6])
I20260130 02:58:24.580792 21164 tablet_loader.cc:96] loaded metadata for tablet 15918c90ffdc43a5a38b2950e9834188 (table default.Table_333d2efbf068485a871f566d50e563ce [id=8b7db40afb894ae99625a756c75ef1de])
I20260130 02:58:24.581822 21164 tablet_loader.cc:96] loaded metadata for tablet 16691c9315e94253bb9b3fb5058e5636 (table default.Table_8f311c5878744ef692b496ecfdfc3ede [id=4f499357303d464aafc17679ffcaaa07])
W20260130 02:58:24.583320 19313 leader_election.cc:343] T 6d1f121f49cd47c2b8247daaf6874e01 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:24.583698 19313 leader_election.cc:304] T 6d1f121f49cd47c2b8247daaf6874e01 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78c2c46c7520492db35d51c244fb4f74; no voters: 371a36d219164974abdf848841528152, d870164263bb44b6b780a557daea3656
I20260130 02:58:24.584355 21164 tablet_loader.cc:96] loaded metadata for tablet 1fc8f09d6329407c8078542b4123ef4d (table default.Table_7d1c27ab3515406ba5b5333c90df093e [id=b6fabeeae79941dc8094d0f2b51704f9])
I20260130 02:58:24.585049 21141 raft_consensus.cc:2749] T 6d1f121f49cd47c2b8247daaf6874e01 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260130 02:58:24.585547 21141 consensus_queue.cc:1048] T 49efcfced0b74c48962393149ac2a500 P 78c2c46c7520492db35d51c244fb4f74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260130 02:58:24.585887 19313 leader_election.cc:304] T c76f0dc7762c49c389fa6fef41f02141 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 78c2c46c7520492db35d51c244fb4f74, d870164263bb44b6b780a557daea3656; no voters: 371a36d219164974abdf848841528152
I20260130 02:58:24.586097 21164 tablet_loader.cc:96] loaded metadata for tablet 25fd48e693424537b91ba7c3fcc96878 (table default.Table_ebc8b8379c274c18b0a06a919dbe0ee0 [id=28156e0843674fce88f5914a3e4195ae])
I20260130 02:58:24.586319 20704 consensus_queue.cc:1048] T d480bf18140245e3870a62582d5a8318 P 78c2c46c7520492db35d51c244fb4f74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260130 02:58:24.587157 21121 raft_consensus.cc:2804] T c76f0dc7762c49c389fa6fef41f02141 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260130 02:58:24.587502 21121 raft_consensus.cc:493] T c76f0dc7762c49c389fa6fef41f02141 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260130 02:58:24.587805 21121 raft_consensus.cc:3060] T c76f0dc7762c49c389fa6fef41f02141 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:24.587899 21164 tablet_loader.cc:96] loaded metadata for tablet 274530359b174ff79f046c2d4a6afa76 (table default.Table_c4f4904b6fac4b9880979cf0314e71c1 [id=c456c6418c93499a9d08f652720854ca])
I20260130 02:58:24.589926 21164 tablet_loader.cc:96] loaded metadata for tablet 2b0edaad6877460cb7982efc62b00b66 (table default.Table_d95381c590c94802bdc83acef0f0df96 [id=64a8e0e079f34806aa68795f3d548c5e])
I20260130 02:58:24.591030 21164 tablet_loader.cc:96] loaded metadata for tablet 2b8034a92b2043e5b57cf02f1ee6296d (table default.Table_e2480022cdae4429b710395eebae7fec [id=fa72411018324474aa9a7265d9f1a2e4])
I20260130 02:58:24.592777 21164 tablet_loader.cc:96] loaded metadata for tablet 2ca0df71836145748cada4e4559a5656 (table default.Table_d764cbaecfee476ba5e734dd1e2f50fd [id=3f9236f0631a4e4c94ca7456f0478a45])
I20260130 02:58:24.594391 21164 tablet_loader.cc:96] loaded metadata for tablet 306452e208ca4e719609f67345114a13 (table default.Table_fa63c9eff99d4ac1b185a4c4ce2df031 [id=2e8b378422c74abc939955d110737316])
I20260130 02:58:24.595470 21164 tablet_loader.cc:96] loaded metadata for tablet 30cc52136bb3422895d95c0fed0c6798 (table default.Table_f619de75da8e4d23b4d641d30285e0e7 [id=2dc3a57f750e4104b99f4589fdb29e3c])
I20260130 02:58:24.594792 21121 raft_consensus.cc:515] T c76f0dc7762c49c389fa6fef41f02141 P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:24.597237 21164 tablet_loader.cc:96] loaded metadata for tablet 320dbe98a32a43b9899bbeed80b20f55 (table default.Table_cda216a3bfc14ec0862b998bedd5b300 [id=35978c9d901c47eb81e565e37493b1ae])
I20260130 02:58:24.598170 21164 tablet_loader.cc:96] loaded metadata for tablet 3379641837df4b6ea360e8e27dfe4f8b (table default.Table_8f632ff84d624b72a10cb90fb9e78636 [id=3cf79022ea954f6397f0a0497e6e6a6c])
I20260130 02:58:24.598636 21121 leader_election.cc:290] T c76f0dc7762c49c389fa6fef41f02141 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 election: Requested vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), d870164263bb44b6b780a557daea3656 (127.18.100.67:39571)
I20260130 02:58:24.599225 19224 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c76f0dc7762c49c389fa6fef41f02141" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152"
I20260130 02:58:24.599910 21164 tablet_loader.cc:96] loaded metadata for tablet 341041cb0be2456e88403367c0814704 (table default.Table_1703c69112744c0e951c1ac9667bbac6 [id=8d76f4328edc491291491c78ba205d30])
W20260130 02:58:24.600289 19314 leader_election.cc:343] T c76f0dc7762c49c389fa6fef41f02141 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:24.600934 21164 tablet_loader.cc:96] loaded metadata for tablet 345eaaa5625c48d099280b7fbb2e514b (table default.Table_c1886dd9a7934e7797e0e7e0b065d692 [id=7a2d5367224b492b844199088b01b149])
I20260130 02:58:24.602034 21164 tablet_loader.cc:96] loaded metadata for tablet 3593f8b89da24ff9b6cf549d5ac58bc6 (table default.Table_2c737f14bf854cfca58d4a9f83fb2e9e [id=fa4e8955bebb40b6adf6ed4ce0e43bfb])
I20260130 02:58:24.603091 21164 tablet_loader.cc:96] loaded metadata for tablet 3738385130c044e0a5a5803da74c567d (table default.Table_ddfb5df90fdb40a087ac434d088bbf0e [id=1e72b7dd669d4539b61884ba63ff7183])
E20260130 02:58:24.222615 18870 catalog_manager.cc:5809] Error updating tablets from 371a36d219164974abdf848841528152: Illegal state: replica 658299197a424f53a37f14fadf089ae4 is not leader of this config: current role FOLLOWER. Tablet report was: is_incremental: true updated_tablets { tablet_id: "12c3f3ec523040e7b7cd356e3da57e0d" state: RUNNING consensus_state { current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "320dbe98a32a43b9899bbeed80b20f55" state: RUNNING consensus_state { current_term: 1 leader_uuid: "371a36d219164974abdf848841528152" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } health_report { overall_health: UNKNOWN } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY stats { } } updated_tablets { tablet_id: "13f0ace4d9de43abab08eca32cdb30c1" state: RUNNING consensus_state { current_term: 1 leader_uuid: "371a36d219164974abdf848841528152" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } health_report { overall_health: UNKNOWN } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY stats { on_disk_size: 27281 live_row_count: 0 } } updated_tablets { tablet_id: "3aa76993beb94b12bc706a76db26a03b" state: RUNNING consensus_state { current_term: 1 leader_uuid: "d870164263bb44b6b780a557daea3656" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "e12d130bcdf34965888753f12c1ff848" state: RUNNING consensus_state { current_term: 1 leader_uuid: "78c2c46c7520492db35d51c244fb4f74" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "838e49e5e8874db393bdc15a1c92f87e" state: RUNNING consensus_state { current_term: 1 leader_uuid: "371a36d219164974abdf848841528152" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } health_report { overall_health: HEALTHY } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY stats { on_disk_size: 27281 live_row_count: 0 } } updated_tablets { tablet_id: "f9e5f01e69254d658c369f3e21cef523" state: RUNNING consensus_state { current_term: 1 leader_uuid: "d870164263bb44b6b780a557daea3656" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "a90712651a604b83ba187f893ab17711" state: RUNNING consensus_state { current_term: 1 leader_uuid: "371a36d219164974abdf848841528152" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } health_report { overall_health: HEALTHY } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY stats { on_disk_size: 27356 live_row_count: 0 } } updated_tablets { tablet_id: "78e9bc3e55fe4e4a98e8eb01da59d365" state: RUNNING consensus_state { current_term: 1 leader_uuid: "d870164263bb44b6b780a557daea3656" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "2c79a3aecab844698e70834999e1caac" state: RUNNING consensus_state { current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "f41f4cff7dd4489f948700a45e9a7057" state: RUNNING consensus_state { current_term: 1 leader_uuid: "d870164263bb44b6b780a557daea3656" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "341041cb0be2456e88403367c0814704" state: RUNNING consensus_state { current_term: 1 leader_uuid: "d870164263bb44b6b780a557daea3656" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } } } schema_version: 1 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "5fbe92a2227d44d88b4531a7f8aa0e08" state: RUNNING consensus_state { current_term: 1 leader_uuid: "371a36d219164974abdf848841528152" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } health_report { overall_health: HEALTHY } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY stats { on_disk_size: 27356 live_row_count: 0 } } updated_tablets { tablet_id: "7caf7cbad5054396b5617094e4d536a0" state: RUNNING consensus_state { current_term: 1 leader_uuid: "d870164263bb44b6b780a557daea3656" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } } } schema_version: 1 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "ba91a508b1cb4430a4d8ca26946aecc8" state: RUNNING consensus_state { current_term: 1 leader_uuid: "371a36d219164974abdf848841528152" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } health_report { overall_health: HEALTHY } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY stats { on_disk_size: 27356 live_row_count: 0 } } updated_tablets { tablet_id: "4907294442c145c6b03eb1de057c66a8" state: RUNNING consensus_state { current_term: 1 leader_uuid: "78c2c46c7520492db35d51c244fb4f74" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "03025d13f0e545cfa6288677a2715343" state: RUNNING consensus_state { current_term: 1 leader_uuid: "d870164263bb44b6b780a557daea3656" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "9fe0448bcc4a4346b3c67fc3f626ed3c" state: RUNNING consensus_state { current_term: 1 leader_uuid: "78c2c46c7520492db35d51c244fb4f74" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "4c139aa8cd184f488cec4e05ef7dcada" state: RUNNING consensus_state { current_term: 1 leader_uuid: "78c2c46c7520492db35d51c244fb4f74" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } } } schema_version: 1 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "f2d01bfdc9c4494386bd02bed655fe63" state: RUNNING consensus_state { current_term: 1 leader_uuid: "d870164263bb44b6b780a557daea3656" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } } } schema_version: 3 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "16691c9315e94253bb9b3fb5058e5636" state: RUNNING consensus_state { current_term: 1 leader_uuid: "78c2c46c7520492db35d51c244fb4f74" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } } } schema_version: 1 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "753344737b5941d9a93b6779e72f63d4" state: RUNNING consensus_state { current_term: 1 leader_uuid: "78c2c46c7520492db35d51c244fb4f74" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "f7607df5efd543438fb2dff3667cc8c8" state: RUNNING consensus_state { current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "43cec19c51ec4f05b63800c07f9534ef" state: RUNNING consensus_state { current_term: 1 leader_uuid: "78c2c46c7520492db35d51c244fb4f74" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "8e57ae31ab3941f789ac69627384e3b0" state: STOPPING consensus_state { current_term: 1 leader_uuid: "78c2c46c7520492db35d51c244fb4f74" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } } } schema_version: 1 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "25fd48e693424537b91ba7c3fcc96878" state: RUNNING consensus_state { current_term: 1 leader_uuid: "371a36d219164974abdf848841528152" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } health_report { overall_health: HEALTHY } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY stats { on_disk_size: 27325 live_row_count: 0 } } updated_tablets { tablet_id: "3ae5472cba1e4613abe4cda42ccb2496" state: RUNNING consensus_state { current_term: 1 leader_uuid: "371a36d219164974abdf848841528152" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } health_report { overall_health: HEALTHY } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY stats { on_disk_size: 27356 live_row_count: 0 } } updated_tablets { tablet_id: "6218f2f7d73a42428b57a45858f53afd" state: RUNNING consensus_state { current_term: 1 leader_uuid: "371a36d219164974abdf848841528152" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } health_report { overall_health: HEALTHY } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY stats { on_disk_size: 27325 live_row_count: 0 } } updated_tablets { tablet_id: "e2aaaf00f53d418b87d5c7af286e4d84" state: RUNNING consensus_state { current_term: 1 leader_uuid: "371a36d219164974abdf848841528152" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } health_report { overall_health: HEALTHY } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY stats { on_disk_size: 27356 live_row_count: 0 } } updated_tablets { tablet_id: "274530359b174ff79f046c2d4a6afa76" state: RUNNING consensus_state { current_term: 1 leader_uuid: "371a36d219164974abdf848841528152" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } health_report { overall_health: HEALTHY } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY stats { on_disk_size: 27356 live_row_count: 0 } } updated_tablets { tablet_id: "3593f8b89da24ff9b6cf549d5ac58bc6" state: RUNNING consensus_state { current_term: 1 leader_uuid: "78c2c46c7520492db35d51c244fb4f74" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "ee7c8ff92ca9465fa18be6eae63d78ca" state: RUNNING consensus_state { current_term: 1 leader_uuid: "78c2c46c7520492db35d51c244fb4f74" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } } } schema_version: 1 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "7783db3568f4434e91e9e456799cc312" state: RUNNING consensus_state { current_term: 1 leader_uuid: "371a36d219164974abdf848841528152" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } health_report { overall_health: HEALTHY } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY stats { on_disk_size: 27356 live_row_count: 0 } } updated_tablets { tablet_id: "54be22352e404ff9b798a075693f7f21" state: RUNNING consensus_state { current_term: 1 leader_uuid: "371a36d219164974abdf848841528152" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } health_report { overall_health: HEALTHY } } } } schema_version: 1 tablet_data_state: TABLET_DATA_READY stats { on_disk_size: 27578 live_row_count: 0 } } updated_tablets { tablet_id: "b8c579ff5fa84651ba9125457d927162" state: RUNNING consensus_state { current_term: 1 leader_uuid: "78c2c46c7520492db35d51c244fb4f74" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "91f954f3ef4244c9a575a7421a7e75eb" state: RUNNING consensus_state { current_term: 1 leader_uuid: "d870164263bb44b6b780a557daea3656" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } } } schema_version: 1 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "111287b5557d4abda1c5cd560fb0d019" state: RUNNING consensus_state { current_term: 1 leader_uuid: "371a36d219164974abdf848841528152" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } health_report { overall_health: HEALTHY } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY stats { on_disk_size: 27356 live_row_count: 0 } } updated_tablets { tablet_id: "127a84346b9f473d9e7f739b1e65feb8" state: RUNNING consensus_state { current_term: 1 leader_uuid: "371a36d219164974abdf848841528152" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } health_report { overall_health: HEALTHY } } } } schema_version: 7 tablet_data_state: TABLET_DATA_READY stats { on_disk_size: 28022 live_row_count: 0 } } updated_tablets { tablet_id: "0a763e48583c4e748a46a325746ce69a" state: RUNNING consensus_state { current_term: 1 leader_uuid: "371a36d219164974abdf848841528152" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } health_report { overall_health: HEALTHY } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY stats { on_disk_size: 27356 live_row_count: 0 } } updated_tablets { tablet_id: "a90e92d569254bbf8a5fbf286baf5cce" state: RUNNING consensus_state { current_term: 1 leader_uuid: "78c2c46c7520492db35d51c244fb4f74" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port
I20260130 02:58:24.606292 21164 tablet_loader.cc:96] loaded metadata for tablet 39bcc41f10ae4241b370529364454586 (table default.Table_a132cf962cfc4ed8a6a8c177e088daeb [id=1c4937ca6e014fec8596a3dc25779600])
I20260130 02:58:24.608062 19807 tablet_bootstrap.cc:492] T 306452e208ca4e719609f67345114a13 P 371a36d219164974abdf848841528152: No bootstrap required, opened a new log
I20260130 02:58:24.608517 19807 ts_tablet_manager.cc:1403] T 306452e208ca4e719609f67345114a13 P 371a36d219164974abdf848841528152: Time spent bootstrapping tablet: real 0.200s user 0.009s sys 0.000s
W20260130 02:58:24.608626 18859 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet 657e9b0d8c05413f904129dce745710b (table default.Table_526677c8ef4144bd9de7a54a5e78e2be [id=b16908b75587459597a978c71756451c], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 657e9b0d8c05413f904129dce745710b consensus state has no leader
I20260130 02:58:24.609050 21164 tablet_loader.cc:96] loaded metadata for tablet 3aa76993beb94b12bc706a76db26a03b (table default.Table_477fb92965284222aa1ecaa0107d1ec2 [id=ad82881a19e748edb569e74d692bb2bc])
I20260130 02:58:24.610090 21164 tablet_loader.cc:96] loaded metadata for tablet 3ae5472cba1e4613abe4cda42ccb2496 (table default.Table_69267d31df644f4396a44130fa3d499d [id=7efa53800a974e31a38a4c60c0a1b466])
I20260130 02:58:24.611166 21164 tablet_loader.cc:96] loaded metadata for tablet 3b02480d9c6f4328af83f7940716d698 (table default.Table_1e9e20a1bfec4bcf81f9eb6219c33d59 [id=f675b7a3ec744533903d8834eeed51d7])
I20260130 02:58:24.613965 20899 ts_tablet_manager.cc:1929] T 43cec19c51ec4f05b63800c07f9534ef P 78c2c46c7520492db35d51c244fb4f74: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260130 02:58:24.614096 19313 leader_election.cc:304] T cd30d7adcc4545848e3501a205a10293 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 78c2c46c7520492db35d51c244fb4f74, d870164263bb44b6b780a557daea3656; no voters: 371a36d219164974abdf848841528152
I20260130 02:58:24.614488 20899 log.cc:1199] T 43cec19c51ec4f05b63800c07f9534ef P 78c2c46c7520492db35d51c244fb4f74: Deleting WAL directory at /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-1/wal/wals/43cec19c51ec4f05b63800c07f9534ef
I20260130 02:58:24.624747 19840 tablet_bootstrap.cc:492] T 6f5a90b75a0f4851b5db672c41253f7b P 78c2c46c7520492db35d51c244fb4f74: No bootstrap required, opened a new log
I20260130 02:58:24.624567 21192 sys_catalog.cc:455] T 00000000000000000000000000000000 P 107f274cdda543038832f06a56ab9130 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "107f274cdda543038832f06a56ab9130" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "658299197a424f53a37f14fadf089ae4" member_type: VOTER last_known_addr { host: "127.18.100.126" port: 38191 } } peers { permanent_uuid: "107f274cdda543038832f06a56ab9130" member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39711 } } peers { permanent_uuid: "3749baa287ed4f249329a74071781edd" member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34629 } } }
I20260130 02:58:24.625161 19840 ts_tablet_manager.cc:1403] T 6f5a90b75a0f4851b5db672c41253f7b P 78c2c46c7520492db35d51c244fb4f74: Time spent bootstrapping tablet: real 0.225s user 0.008s sys 0.001s
I20260130 02:58:24.625561 21192 sys_catalog.cc:458] T 00000000000000000000000000000000 P 107f274cdda543038832f06a56ab9130 [sys.catalog]: This master's current role is: LEADER
I20260130 02:58:24.627373 19840 raft_consensus.cc:359] T 6f5a90b75a0f4851b5db672c41253f7b P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:24.627974 19840 raft_consensus.cc:385] T 6f5a90b75a0f4851b5db672c41253f7b P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:24.628207 19840 raft_consensus.cc:740] T 6f5a90b75a0f4851b5db672c41253f7b P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78c2c46c7520492db35d51c244fb4f74, State: Initialized, Role: FOLLOWER
I20260130 02:58:24.629001 21141 raft_consensus.cc:2804] T cd30d7adcc4545848e3501a205a10293 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260130 02:58:24.629349 21141 raft_consensus.cc:493] T cd30d7adcc4545848e3501a205a10293 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260130 02:58:24.629650 21141 raft_consensus.cc:3060] T cd30d7adcc4545848e3501a205a10293 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:24.631521 21164 tablet_loader.cc:96] loaded metadata for tablet 475b435b2967418f927fa172d2ff04ea (table default.Table_dee034aec0b24982b8dc2a3fa1b326f1 [id=66033e75b3814a8584311cf7e4a07ad3])
I20260130 02:58:24.632560 21164 tablet_loader.cc:96] loaded metadata for tablet 47f574674aa7414588605ca38eb7200a (table default.Table_d2a3438ed66741eab245ae6d6c5c2f37 [id=4fb5c26265e94083a8714be512fcd222])
I20260130 02:58:24.634438 21164 tablet_loader.cc:96] loaded metadata for tablet 4907294442c145c6b03eb1de057c66a8 (table default.Table_ea2e0aa925a0486bb292304ca21bb819 [id=18580b9c54e448d1bdaae84f697f7b0b])
I20260130 02:58:24.635509 21164 tablet_loader.cc:96] loaded metadata for tablet 49efcfced0b74c48962393149ac2a500 (table default.Table_2e81f0a0cbc449a397141b717db17025 [id=a9c919f57c0c437cb35a6921103e45a6])
I20260130 02:58:24.636565 21164 tablet_loader.cc:96] loaded metadata for tablet 4c139aa8cd184f488cec4e05ef7dcada (table default.Table_e4d07634d84a4678b1c94374e3adc476 [id=fcbd7e3a7fc64d89aa740494d7e1a0f7])
I20260130 02:58:24.636806 21141 raft_consensus.cc:515] T cd30d7adcc4545848e3501a205a10293 P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:24.637655 21164 tablet_loader.cc:96] loaded metadata for tablet 532e867fd5944d468d7672d2a5c92f3c (table default.Table_0ca1d7370d2f416782b386c5b47111ab [id=3ffe1a273d4f452c835278a8e7f10102])
I20260130 02:58:24.639071 19313 leader_election.cc:304] T 1fc8f09d6329407c8078542b4123ef4d P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 78c2c46c7520492db35d51c244fb4f74, d870164263bb44b6b780a557daea3656; no voters: 371a36d219164974abdf848841528152
W20260130 02:58:24.640102 19313 leader_election.cc:343] T 1129f72158ec4bae89da3c94cdb70839 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:24.640478 19313 leader_election.cc:304] T 1129f72158ec4bae89da3c94cdb70839 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78c2c46c7520492db35d51c244fb4f74; no voters: 371a36d219164974abdf848841528152, d870164263bb44b6b780a557daea3656
I20260130 02:58:24.641291 20704 raft_consensus.cc:2804] T 1fc8f09d6329407c8078542b4123ef4d P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260130 02:58:24.641592 20704 raft_consensus.cc:493] T 1fc8f09d6329407c8078542b4123ef4d P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260130 02:58:24.641908 20704 raft_consensus.cc:3060] T 1fc8f09d6329407c8078542b4123ef4d P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:24.646054 21164 tablet_loader.cc:96] loaded metadata for tablet 54be22352e404ff9b798a075693f7f21 (table default.Table_3a858a3cb886427cb8268f42c953ad4c [id=17150cb73e374ff9a1e92c43e0c3f2c1])
I20260130 02:58:24.647347 18833 master-stress-test.cc:577] Killing test thread 1/10
I20260130 02:58:24.647718 20704 raft_consensus.cc:515] T 1fc8f09d6329407c8078542b4123ef4d P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:24.649143 21164 tablet_loader.cc:96] loaded metadata for tablet 5fbe92a2227d44d88b4531a7f8aa0e08 (table default.Table_a3ecda6d64704b7fad065a5845683ad8 [id=e93360fd2af141a28d92401ed449c19d])
I20260130 02:58:24.649371 20704 leader_election.cc:290] T 1fc8f09d6329407c8078542b4123ef4d P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 election: Requested vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), d870164263bb44b6b780a557daea3656 (127.18.100.67:39571)
I20260130 02:58:24.649807 20704 raft_consensus.cc:2749] T 1129f72158ec4bae89da3c94cdb70839 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260130 02:58:24.650249 21164 tablet_loader.cc:96] loaded metadata for tablet 6218f2f7d73a42428b57a45858f53afd (table default.Table_95f0bff8e17d4ca995c0af0d0c24e84d [id=b7795905d0c440579c59d6ac00473f5a])
I20260130 02:58:24.651944 19840 consensus_queue.cc:260] T 6f5a90b75a0f4851b5db672c41253f7b P 78c2c46c7520492db35d51c244fb4f74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:24.612547 19159 leader_election.cc:304] T eb7a755ab62b4552bb8588264c5f7b75 P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 371a36d219164974abdf848841528152; no voters: 78c2c46c7520492db35d51c244fb4f74, d870164263bb44b6b780a557daea3656
I20260130 02:58:24.653574 21152 raft_consensus.cc:3060] T eb7a755ab62b4552bb8588264c5f7b75 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:24.654078 19840 ts_tablet_manager.cc:1434] T 6f5a90b75a0f4851b5db672c41253f7b P 78c2c46c7520492db35d51c244fb4f74: Time spent starting tablet: real 0.029s user 0.001s sys 0.004s
I20260130 02:58:24.656579 21164 tablet_loader.cc:96] loaded metadata for tablet 657e9b0d8c05413f904129dce745710b (table default.Table_526677c8ef4144bd9de7a54a5e78e2be [id=b16908b75587459597a978c71756451c])
I20260130 02:58:24.657714 21164 tablet_loader.cc:96] loaded metadata for tablet 66a7487a7ab343a4abca5312afe78862 (table default.Table_6c46e6995d4741dcaaa6ed7bdc765e29 [id=6d91a161cf484239967af58db160c952])
I20260130 02:58:24.658689 21164 tablet_loader.cc:96] loaded metadata for tablet 6ad6bb8c986a433d94c745f2b47266ff (table default.Table_bc168ff8b5f34cacbd398117647ba25a [id=2a21ee52b7d246f1b44eda02bf5a411e])
I20260130 02:58:24.659759 21164 tablet_loader.cc:96] loaded metadata for tablet 6d1f121f49cd47c2b8247daaf6874e01 (table default.Table_0480357296ac4a27812d802ee4566351 [id=0da6197ab32e4645aa8d8787662ee5e6])
I20260130 02:58:24.660395 21152 raft_consensus.cc:2749] T eb7a755ab62b4552bb8588264c5f7b75 P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260130 02:58:24.661269 19159 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.100.67:39571, user_credentials={real_user=slave}} blocked reactor thread for 48799.8us
I20260130 02:58:24.664009 21164 tablet_loader.cc:96] loaded metadata for tablet 6e12a8eeb3204325a9a8b7779f72ab29 (table default.Table_8c27a7d10b7847e78029f7e95321e9b6 [id=e31c1bf0c3b9457fa63efe71138c259c])
I20260130 02:58:24.670842 21164 tablet_loader.cc:96] loaded metadata for tablet 6f5a90b75a0f4851b5db672c41253f7b (table default.Table_6f53c17cfd144031b2d4b112fb2d88eb [id=443cbad5510d463fa8ca878b8628dbda])
I20260130 02:58:24.672755 21164 tablet_loader.cc:96] loaded metadata for tablet 753344737b5941d9a93b6779e72f63d4 (table default.Table_8896365e792f4d24b2c16c2d269b0a93 [id=44816e07033849479a7fea0f51d00fad])
I20260130 02:58:24.673839 21164 tablet_loader.cc:96] loaded metadata for tablet 7783db3568f4434e91e9e456799cc312 (table default.Table_2d345d0f0957486c86c2cf2e34b00941 [id=4c3e0d3ecd5642a3b7fc7d20e2521418])
I20260130 02:58:24.680574 21164 tablet_loader.cc:96] loaded metadata for tablet 7caf7cbad5054396b5617094e4d536a0 (table default.Table_007e969f70d345a7bcb3dfbaabbc025d [id=77d4fa8705a3419cb0a84d3ca1bb4ce3])
I20260130 02:58:24.686697 21141 leader_election.cc:290] T cd30d7adcc4545848e3501a205a10293 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 election: Requested vote from peers d870164263bb44b6b780a557daea3656 (127.18.100.67:39571), 371a36d219164974abdf848841528152 (127.18.100.65:42021)
I20260130 02:58:24.691596 21164 tablet_loader.cc:96] loaded metadata for tablet 838e49e5e8874db393bdc15a1c92f87e (table default.Table_da3d581b15a94840898777f74611994e [id=b7e241ced53e4160ac6fa1f65ef3674e])
W20260130 02:58:24.691689 18870 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.18.100.65:51021 (request call id 42) took 3129 ms (client timeout 1000 ms). Trace:
I20260130 02:58:24.650818 21000 raft_consensus.cc:493] T e2aaaf00f53d418b87d5c7af286e4d84 P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 371a36d219164974abdf848841528152)
I20260130 02:58:24.692313 21000 raft_consensus.cc:515] T e2aaaf00f53d418b87d5c7af286e4d84 P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
W20260130 02:58:24.691996 18870 rpcz_store.cc:269] 0130 02:58:21.562092 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:58:22.233949 (+671857us) service_pool.cc:225] Handling call
0130 02:58:24.178321 (+1944372us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:58:24.178355 (+ 34us) write_op.cc:454] Released schema lock
0130 02:58:24.691609 (+513254us) inbound_call.cc:173] Queueing failure response
Related trace 'op':
0130 02:58:24.177910 (+ 0us) op_driver.cc:389] HandleFailure(Illegal state: replica 658299197a424f53a37f14fadf089ae4 is not leader of this config: current role FOLLOWER)
0130 02:58:24.177950 (+ 40us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:58:24.178156 (+ 206us) write_op.cc:454] Released schema lock
0130 02:58:24.178177 (+ 21us) write_op.cc:335] FINISH: Op aborted
Metrics: {"reported_tablets":60,"child_traces":[["op",{}]]}
I20260130 02:58:24.693444 21164 tablet_loader.cc:96] loaded metadata for tablet 8b47653c7c3945a68e9bf4680bfc0496 (table default.Table_1ae68ae3f0404843aca3c72f02678ac6 [id=8a9182f49e18489fafec8dca190c1b2f])
W20260130 02:58:24.693423 18870 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.18.100.66:59691 (request call id 40) took 1090 ms (client timeout 1000 ms). Trace:
W20260130 02:58:24.693724 18870 rpcz_store.cc:269] 0130 02:58:23.603362 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:58:24.693278 (+1089916us) service_pool.cc:212] Skipping call since client already timed out
0130 02:58:24.693372 (+ 94us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20260130 02:58:24.694007 21000 leader_election.cc:290] T e2aaaf00f53d418b87d5c7af286e4d84 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767)
I20260130 02:58:24.690326 21094 raft_consensus.cc:515] T 1fc8f09d6329407c8078542b4123ef4d P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:24.696215 21094 leader_election.cc:290] T 1fc8f09d6329407c8078542b4123ef4d P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 election: Requested vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767)
I20260130 02:58:24.696522 21164 tablet_loader.cc:96] loaded metadata for tablet 91f954f3ef4244c9a575a7421a7e75eb (table default.Table_9f83f87711d94e8394abe1925ac84ad1 [id=db23c1be42c44b6f9eb774f14d2b8fd6])
I20260130 02:58:24.696789 20899 ts_tablet_manager.cc:1950] T 43cec19c51ec4f05b63800c07f9534ef P 78c2c46c7520492db35d51c244fb4f74: Deleting consensus metadata
I20260130 02:58:24.697606 21164 tablet_loader.cc:96] loaded metadata for tablet 920f52b975ab480fbf42656171222f09 (table default.Table_78b080efddd94878b9d8b77b5ef3af35 [id=b654822c58224a859ad2dcdd8d0123c2])
I20260130 02:58:24.696959 21167 raft_consensus.cc:515] T 94355fc1338f479e8798070aff04fcba P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:24.699570 19519 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0855d88806914a36b4bf6b49036f0ad0" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
I20260130 02:58:24.700814 19218 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1fc8f09d6329407c8078542b4123ef4d" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152"
I20260130 02:58:24.701362 19218 raft_consensus.cc:3060] T 1fc8f09d6329407c8078542b4123ef4d P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:24.701745 21167 leader_election.cc:290] T 94355fc1338f479e8798070aff04fcba P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 election: Requested vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767)
I20260130 02:58:24.702399 19224 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cd30d7adcc4545848e3501a205a10293" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152"
I20260130 02:58:24.705602 21164 tablet_loader.cc:96] loaded metadata for tablet 94355fc1338f479e8798070aff04fcba (table default.Table_12c2fe20cc3e45d29b7be37f900f68a9 [id=67ac4f4eda324e799b5867b2b011a16b])
I20260130 02:58:24.710934 19218 raft_consensus.cc:2468] T 1fc8f09d6329407c8078542b4123ef4d P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78c2c46c7520492db35d51c244fb4f74 in term 1.
I20260130 02:58:24.713230 20899 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60110 (request call id 341) took 1493 ms. Trace:
I20260130 02:58:24.713748 20899 rpcz_store.cc:276] 0130 02:58:23.219966 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:58:23.220094 (+ 128us) service_pool.cc:225] Handling call
0130 02:58:24.069927 (+849833us) ts_tablet_manager.cc:1151] Deleting tablet 43cec19c51ec4f05b63800c07f9534ef
0130 02:58:24.069966 (+ 39us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0130 02:58:24.433060 (+363094us) tablet_metadata.cc:641] Metadata flushed
0130 02:58:24.603901 (+170841us) tablet_metadata.cc:641] Metadata flushed
0130 02:58:24.713175 (+109274us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":814564}
I20260130 02:58:24.714819 21164 tablet_loader.cc:96] loaded metadata for tablet 955b9547f49f434699fe1d86b260638c (table default.Table_f246882197f44c69b72ca421cbdf201f [id=f778784dc88a4e4890113916098ad20f])
W20260130 02:58:24.699085 19313 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.100.67:39571, user_credentials={real_user=slave}} blocked reactor thread for 59004.8us
I20260130 02:58:24.716212 20899 tablet_replica.cc:333] T ba91a508b1cb4430a4d8ca26946aecc8 P 78c2c46c7520492db35d51c244fb4f74: stopping tablet replica
W20260130 02:58:24.716398 19462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.100.66:40767, user_credentials={real_user=slave}} blocked reactor thread for 196939us
I20260130 02:58:24.716286 21133 consensus_queue.cc:1048] T aae68390dbbe4dc58ce3d3bd01725ae6 P 78c2c46c7520492db35d51c244fb4f74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260130 02:58:24.716977 21164 tablet_loader.cc:96] loaded metadata for tablet 9896ff9364744857968e87e80e776dd2 (table default.Table_5d5e26f93a5a4fb084831add317d90f9 [id=f237b1c90a1743d597d87c5cf1abc89a])
I20260130 02:58:24.717792 21167 consensus_queue.cc:1048] T dfbbe9045e7c4aa5b520218687e7e4c1 P d870164263bb44b6b780a557daea3656 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260130 02:58:24.718089 21164 tablet_loader.cc:96] loaded metadata for tablet 9fe0448bcc4a4346b3c67fc3f626ed3c (table default.Table_5c1c447e85524ac5a61ffd8a48509d36 [id=70aaf21ad9bd4d959f46939c6b4a4880])
I20260130 02:58:24.719197 21164 tablet_loader.cc:96] loaded metadata for tablet a3258852aed24ed582a79f77860e6f0d (table default.Table_7c48a0d054354501947d11c5fd7b1864 [id=aa4ad1394ab344df8c54ba0031af5590])
W20260130 02:58:24.723172 19313 leader_election.cc:343] T 920f52b975ab480fbf42656171222f09 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:24.723620 19313 leader_election.cc:304] T 920f52b975ab480fbf42656171222f09 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78c2c46c7520492db35d51c244fb4f74; no voters: 371a36d219164974abdf848841528152, d870164263bb44b6b780a557daea3656
I20260130 02:58:24.724279 21133 raft_consensus.cc:2749] T 920f52b975ab480fbf42656171222f09 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260130 02:58:24.724386 19462 leader_election.cc:304] T cfd12a72726846e587cff9d6a4d61bd6 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 78c2c46c7520492db35d51c244fb4f74, d870164263bb44b6b780a557daea3656; no voters: 371a36d219164974abdf848841528152
I20260130 02:58:24.699720 19840 tablet_bootstrap.cc:492] T e691c238199b4914bcc3fd248f31420c P 78c2c46c7520492db35d51c244fb4f74: Bootstrap starting.
I20260130 02:58:24.722502 21164 tablet_loader.cc:96] loaded metadata for tablet a90712651a604b83ba187f893ab17711 (table default.Table_0f3e44ae93bc4c028ed3929b94a49a24 [id=3b9c62fc8bbb44ffb7d66e9604974c30])
W20260130 02:58:24.712280 19314 leader_election.cc:343] T cd30d7adcc4545848e3501a205a10293 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:24.727728 21164 tablet_loader.cc:96] loaded metadata for tablet a90e92d569254bbf8a5fbf286baf5cce (table default.Table_1e42d571631143088cf33d4c3aafed28 [id=30ba2af6dffd4f7e8dcbfdd2f3798157])
I20260130 02:58:24.727687 19314 leader_election.cc:304] T 1fc8f09d6329407c8078542b4123ef4d P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 371a36d219164974abdf848841528152, 78c2c46c7520492db35d51c244fb4f74; no voters:
I20260130 02:58:24.728797 21164 tablet_loader.cc:96] loaded metadata for tablet aae68390dbbe4dc58ce3d3bd01725ae6 (table default.Table_f3e214a955f6447482099175b92ecb5c [id=76a688022fd9421a84de059b3c1da77a])
I20260130 02:58:24.728858 21124 raft_consensus.cc:2804] T 1fc8f09d6329407c8078542b4123ef4d P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Leader election won for term 1
I20260130 02:58:24.729270 21124 raft_consensus.cc:697] T 1fc8f09d6329407c8078542b4123ef4d P 78c2c46c7520492db35d51c244fb4f74 [term 1 LEADER]: Becoming Leader. State: Replica: 78c2c46c7520492db35d51c244fb4f74, State: Running, Role: LEADER
I20260130 02:58:24.730003 21124 consensus_queue.cc:237] T 1fc8f09d6329407c8078542b4123ef4d P 78c2c46c7520492db35d51c244fb4f74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:24.736380 21164 tablet_loader.cc:96] loaded metadata for tablet aedf72892b7248e2ace8772283c2b476 (table default.Table_203b3a960d3e4b5cb978b6168435f3a4 [id=ffe264c47e31462a8deaf81a8b96d34c])
I20260130 02:58:24.736434 21173 sys_catalog.cc:455] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "107f274cdda543038832f06a56ab9130" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "658299197a424f53a37f14fadf089ae4" member_type: VOTER last_known_addr { host: "127.18.100.126" port: 38191 } } peers { permanent_uuid: "107f274cdda543038832f06a56ab9130" member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39711 } } peers { permanent_uuid: "3749baa287ed4f249329a74071781edd" member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34629 } } }
I20260130 02:58:24.737133 21173 sys_catalog.cc:458] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4 [sys.catalog]: This master's current role is: FOLLOWER
I20260130 02:58:24.737504 21164 tablet_loader.cc:96] loaded metadata for tablet af0f90071a334016aa17fae2e3a97fc8 (table default.Table_22d10332a83f4700ac21892bab67f46b [id=da3a9cb1f6d5419abc1afc93659b5324])
I20260130 02:58:24.739295 21164 tablet_loader.cc:96] loaded metadata for tablet b1e8b78eff0d4a0b92059adb2de4a3ce (table default.Table_d0b4516d24384e5386c1debaf2543adb [id=f1342f9497b74e7a9ecad1de8082578d])
I20260130 02:58:24.673883 19807 raft_consensus.cc:359] T 306452e208ca4e719609f67345114a13 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:24.741094 19807 raft_consensus.cc:385] T 306452e208ca4e719609f67345114a13 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:24.745848 19625 tablet_bootstrap.cc:492] T 0d14fb95ace9447f810ccbed4d5befdf P d870164263bb44b6b780a557daea3656: No bootstrap required, opened a new log
I20260130 02:58:24.749828 19807 raft_consensus.cc:740] T 306452e208ca4e719609f67345114a13 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 371a36d219164974abdf848841528152, State: Initialized, Role: FOLLOWER
I20260130 02:58:24.750761 21164 tablet_loader.cc:96] loaded metadata for tablet b89ed298e3dc49d789d166c613ad7fb8 (table default.Table_f8856f06ce97440bae4abdf34164716d [id=a3303ada20a94f99b2f37a02eec361ef])
I20260130 02:58:24.751930 21164 tablet_loader.cc:96] loaded metadata for tablet b8c579ff5fa84651ba9125457d927162 (table default.Table_3046889233c24f0a8e8058e1591d63b8 [id=edd6505c0bf3462cbf65c3df5649c16c])
I20260130 02:58:24.752436 19218 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1fc8f09d6329407c8078542b4123ef4d" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152"
I20260130 02:58:24.753243 19218 raft_consensus.cc:2393] T 1fc8f09d6329407c8078542b4123ef4d P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate d870164263bb44b6b780a557daea3656 in current term 1: Already voted for candidate 78c2c46c7520492db35d51c244fb4f74 in this term.
I20260130 02:58:24.753744 21164 tablet_loader.cc:96] loaded metadata for tablet bac739bf4cac4272975f5587ad9c71ff (table default.Table_9e942a1ce3944e9aa028e75656ce066d [id=18ed268eb9ad4450abf9874310ace074])
W20260130 02:58:24.754230 19159 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.100.67:39571, user_credentials={real_user=slave}} blocked reactor thread for 91129.5us
I20260130 02:58:24.754813 21164 tablet_loader.cc:96] loaded metadata for tablet bc04be1284414a8699d3c4b88bf62813 (table default.Table_2414288f0dc54e5f835f876060927a9b [id=b0c6f2d9483c47da8ff3e65f0400a593])
I20260130 02:58:24.755448 19625 ts_tablet_manager.cc:1403] T 0d14fb95ace9447f810ccbed4d5befdf P d870164263bb44b6b780a557daea3656: Time spent bootstrapping tablet: real 0.336s user 0.000s sys 0.011s
I20260130 02:58:24.756788 19218 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "94355fc1338f479e8798070aff04fcba" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152"
I20260130 02:58:24.757903 19840 tablet_bootstrap.cc:654] T e691c238199b4914bcc3fd248f31420c P 78c2c46c7520492db35d51c244fb4f74: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:24.759040 19807 consensus_queue.cc:260] T 306452e208ca4e719609f67345114a13 P 371a36d219164974abdf848841528152 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:24.759123 19313 leader_election.cc:304] T 94355fc1338f479e8798070aff04fcba P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 78c2c46c7520492db35d51c244fb4f74, d870164263bb44b6b780a557daea3656; no voters: 371a36d219164974abdf848841528152
I20260130 02:58:24.760309 19313 leader_election.cc:304] T 306452e208ca4e719609f67345114a13 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 78c2c46c7520492db35d51c244fb4f74, d870164263bb44b6b780a557daea3656; no voters: 371a36d219164974abdf848841528152
I20260130 02:58:24.760929 21121 raft_consensus.cc:2804] T 94355fc1338f479e8798070aff04fcba P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260130 02:58:24.761245 21121 raft_consensus.cc:493] T 94355fc1338f479e8798070aff04fcba P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260130 02:58:24.748687 19462 leader_election.cc:304] T 2b8034a92b2043e5b57cf02f1ee6296d P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 78c2c46c7520492db35d51c244fb4f74, d870164263bb44b6b780a557daea3656; no voters: 371a36d219164974abdf848841528152
I20260130 02:58:24.761554 21121 raft_consensus.cc:3060] T 94355fc1338f479e8798070aff04fcba P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:24.748097 21190 raft_consensus.cc:2804] T cfd12a72726846e587cff9d6a4d61bd6 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260130 02:58:24.761945 21190 raft_consensus.cc:493] T cfd12a72726846e587cff9d6a4d61bd6 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260130 02:58:24.762691 21190 raft_consensus.cc:3060] T cfd12a72726846e587cff9d6a4d61bd6 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:24.765324 21164 tablet_loader.cc:96] loaded metadata for tablet c2c927e461f04fb48bfa9e3a685b077d (table default.Table_aa496bf2019748f29823a9a253b1f3ca [id=780188e2335f43fbb11de8212eb38d56])
I20260130 02:58:24.765879 21094 raft_consensus.cc:2804] T 2b8034a92b2043e5b57cf02f1ee6296d P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260130 02:58:24.766387 21094 raft_consensus.cc:493] T 2b8034a92b2043e5b57cf02f1ee6296d P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260130 02:58:24.766814 21094 raft_consensus.cc:3060] T 2b8034a92b2043e5b57cf02f1ee6296d P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:24.767311 21164 tablet_loader.cc:96] loaded metadata for tablet c76f0dc7762c49c389fa6fef41f02141 (table default.Table_e894e5560f194692bac730600c6bf5f4 [id=d5e1b2a45122429aabb4fd8653d71f0f])
I20260130 02:58:24.769228 21164 tablet_loader.cc:96] loaded metadata for tablet ca06f07bb98f45a7bf63aebc28348e08 (table default.Table_829d1e187e8b4db5b8f4f835e84246e5 [id=a24b4761037d47a383f454a82517131f])
I20260130 02:58:24.770280 21164 tablet_loader.cc:96] loaded metadata for tablet cd30d7adcc4545848e3501a205a10293 (table default.Table_b85cb876f25740bfaaab01949ac26b38 [id=839e292e040c4aeb832f3fc0258c32ec])
I20260130 02:58:24.770691 20704 raft_consensus.cc:2804] T 306452e208ca4e719609f67345114a13 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260130 02:58:24.770403 21186 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3749baa287ed4f249329a74071781edd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 107f274cdda543038832f06a56ab9130. Latest consensus state: current_term: 3 leader_uuid: "107f274cdda543038832f06a56ab9130" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "658299197a424f53a37f14fadf089ae4" member_type: VOTER last_known_addr { host: "127.18.100.126" port: 38191 } } peers { permanent_uuid: "107f274cdda543038832f06a56ab9130" member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39711 } } peers { permanent_uuid: "3749baa287ed4f249329a74071781edd" member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34629 } } }
I20260130 02:58:24.771039 20704 raft_consensus.cc:493] T 306452e208ca4e719609f67345114a13 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260130 02:58:24.771119 21186 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3749baa287ed4f249329a74071781edd [sys.catalog]: This master's current role is: FOLLOWER
I20260130 02:58:24.771304 20704 raft_consensus.cc:3060] T 306452e208ca4e719609f67345114a13 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:24.715337 18860 catalog_manager.cc:4985] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767): tablet 43cec19c51ec4f05b63800c07f9534ef (table default.Table_8f632ff84d624b72a10cb90fb9e78636 [id=3cf79022ea954f6397f0a0497e6e6a6c]) successfully deleted
W20260130 02:58:24.779850 18860 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.18.100.66:40767, user_credentials={real_user=slave}} blocked reactor thread for 64534.3us
I20260130 02:58:24.783243 20899 raft_consensus.cc:2243] T ba91a508b1cb4430a4d8ca26946aecc8 P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260130 02:58:24.783722 21164 tablet_loader.cc:96] loaded metadata for tablet cf1b66a20fce4c4d96bc0bc32a7bafa8 (table default.Table_d24e400f7451443a82d23140a90adc57 [id=e478df970dac40f7a576ce584c3abf2c])
I20260130 02:58:24.784083 20899 raft_consensus.cc:2272] T ba91a508b1cb4430a4d8ca26946aecc8 P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260130 02:58:24.784837 21164 tablet_loader.cc:96] loaded metadata for tablet cfd12a72726846e587cff9d6a4d61bd6 (table default.Table_4ee03db7d0064d96a53d4f261bd12722 [id=07fe2931234d4888a524b52171b03f0a])
I20260130 02:58:24.786640 21164 tablet_loader.cc:96] loaded metadata for tablet d3ba2df0b15c44f882070570f91b254d (table default.Table_92836e7fcf3a48d8bac2160c2ff7c482 [id=e3ae4b66fa0048668cb1e4d9da7d16fa])
I20260130 02:58:24.787684 21164 tablet_loader.cc:96] loaded metadata for tablet d480bf18140245e3870a62582d5a8318 (table default.Table_109c8b6c17d4442b939bde9441d0476f [id=dcf82ea15d6044c2b023f81f0a579ade])
I20260130 02:58:24.799153 21164 tablet_loader.cc:96] loaded metadata for tablet dc09a0f0cecf49738bab04596944c0f3 (table default.Table_6a7faee4c75c4eb7862fb4c05afc17cb [id=676ece9dc36b418a8b1400e8406d6b98])
I20260130 02:58:24.800204 21164 tablet_loader.cc:96] loaded metadata for tablet dc4bdd8e4f104d4a86c0af6ac6accda0 (table default.Table_ba19837ced4c45d79be506e93308449b [id=e732719fbe544e38a15b35017010c7b3])
I20260130 02:58:24.800691 19807 ts_tablet_manager.cc:1434] T 306452e208ca4e719609f67345114a13 P 371a36d219164974abdf848841528152: Time spent starting tablet: real 0.130s user 0.004s sys 0.001s
I20260130 02:58:24.801369 21164 tablet_loader.cc:96] loaded metadata for tablet dcba6b534f9d406dad4fb502be40921c (table default.Table_baaf376733384fdbbac49ec8f2357432 [id=dea97f19a50141878f97820194afa59e])
I20260130 02:58:24.802454 21164 tablet_loader.cc:96] loaded metadata for tablet dd377fab8c544bf69121dc9e9a060b29 (table default.Table_5e70ab0e32bc4048bd19388f9d7cd302 [id=030c8ac23aac4a1fb7acc954d09434db])
I20260130 02:58:24.802771 21167 consensus_queue.cc:1048] T 657e9b0d8c05413f904129dce745710b P d870164263bb44b6b780a557daea3656 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260130 02:58:24.804059 21167 consensus_queue.cc:1048] T 2b0edaad6877460cb7982efc62b00b66 P d870164263bb44b6b780a557daea3656 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260130 02:58:24.804920 21164 tablet_loader.cc:96] loaded metadata for tablet dfbbe9045e7c4aa5b520218687e7e4c1 (table default.Table_e03bdbb221e94bd6b535ddc02c863bad [id=2382eeebdc9540d5b588835705b2a4f1])
I20260130 02:58:24.804908 19515 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c76f0dc7762c49c389fa6fef41f02141" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656"
I20260130 02:58:24.805020 19462 leader_election.cc:304] T 306452e208ca4e719609f67345114a13 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 78c2c46c7520492db35d51c244fb4f74, d870164263bb44b6b780a557daea3656; no voters: 371a36d219164974abdf848841528152
I20260130 02:58:24.805464 19515 raft_consensus.cc:3060] T c76f0dc7762c49c389fa6fef41f02141 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:24.805994 21164 tablet_loader.cc:96] loaded metadata for tablet e026dee22493424692d00f14848ab57c (table default.Table_85611716686c4097bacb64df677d5793 [id=122fe719b1d64ec8b531783539f85df1])
I20260130 02:58:24.812180 19515 raft_consensus.cc:2468] T c76f0dc7762c49c389fa6fef41f02141 P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78c2c46c7520492db35d51c244fb4f74 in term 1.
I20260130 02:58:24.815093 19515 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cd30d7adcc4545848e3501a205a10293" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656"
I20260130 02:58:24.815693 19515 raft_consensus.cc:3060] T cd30d7adcc4545848e3501a205a10293 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:24.819502 20899 ts_tablet_manager.cc:1916] T ba91a508b1cb4430a4d8ca26946aecc8 P 78c2c46c7520492db35d51c244fb4f74: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260130 02:58:24.820187 21164 tablet_loader.cc:96] loaded metadata for tablet e34d5da0ad864b2eaea0d575c982e447 (table default.Table_aa4e5fd22945475da1dad48e1508595e [id=5b41834c6d8b4d89b773c2ea4c4ca7e6])
I20260130 02:58:24.821058 19514 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1fc8f09d6329407c8078542b4123ef4d" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656"
I20260130 02:58:24.821785 19514 raft_consensus.cc:2393] T 1fc8f09d6329407c8078542b4123ef4d P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 78c2c46c7520492db35d51c244fb4f74 in current term 1: Already voted for candidate d870164263bb44b6b780a557daea3656 in this term.
I20260130 02:58:24.822752 21164 tablet_loader.cc:96] loaded metadata for tablet e691c238199b4914bcc3fd248f31420c (table default.Table_7fe8c354c0dc4cfba0e84ed33e4c2506 [id=7523d671e9ac4814acff5060fd282eea])
I20260130 02:58:24.824337 21164 tablet_loader.cc:96] loaded metadata for tablet eb7a755ab62b4552bb8588264c5f7b75 (table default.Table_7e6ab9964f4c4e0f906d875a937a781d [id=bcf41af45d284bd89800977d14194812])
I20260130 02:58:24.824795 21000 raft_consensus.cc:2804] T 306452e208ca4e719609f67345114a13 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260130 02:58:24.825143 21000 raft_consensus.cc:493] T 306452e208ca4e719609f67345114a13 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260130 02:58:24.825446 21000 raft_consensus.cc:3060] T 306452e208ca4e719609f67345114a13 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:24.826134 21164 tablet_loader.cc:96] loaded metadata for tablet ee7c8ff92ca9465fa18be6eae63d78ca (table default.Table_dd22e9dc63f84d8688c0071f427f0b25 [id=77979d498aa040828de415129cb15a06])
I20260130 02:58:24.827248 21164 tablet_loader.cc:96] loaded metadata for tablet f2d01bfdc9c4494386bd02bed655fe63 (table default.Table_aaf139c1a55d4a1783298ca12ae0d5c0 [id=d128767e303145a885eec3c5f714a27f])
I20260130 02:58:24.828311 21164 tablet_loader.cc:96] loaded metadata for tablet f41f4cff7dd4489f948700a45e9a7057 (table default.Table_ed6e2e20d89542199e5cbc7842df964d [id=047c39bd14664d1099e043b973766d32])
I20260130 02:58:24.834995 21164 tablet_loader.cc:96] loaded metadata for tablet f7607df5efd543438fb2dff3667cc8c8 (table default.Table_e8fd7516280f4dc4a73eb1f88fe916a2 [id=6efb836791414f3888f714aa8f65f0a4])
I20260130 02:58:24.836139 21164 tablet_loader.cc:96] loaded metadata for tablet f9e5f01e69254d658c369f3e21cef523 (table default.Table_12b505700b3d44e28353172e1786e5c8 [id=562776405d6f4c7aab0622f3866bf45b])
I20260130 02:58:24.837245 21164 tablet_loader.cc:96] loaded metadata for tablet ffa1142fc7274aecbdbee1a0802b95da (table default.Table_2f60fd0292ab40c8a3ae9b842cae8faf [id=e648e521ef5044a0b1967452cc3a31b2])
I20260130 02:58:24.801556 19807 tablet_bootstrap.cc:492] T cd30d7adcc4545848e3501a205a10293 P 371a36d219164974abdf848841528152: Bootstrap starting.
I20260130 02:58:24.813812 21200 raft_consensus.cc:493] T 306452e208ca4e719609f67345114a13 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:24.839743 21200 raft_consensus.cc:515] T 306452e208ca4e719609f67345114a13 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:24.841878 21200 leader_election.cc:290] T 306452e208ca4e719609f67345114a13 P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d870164263bb44b6b780a557daea3656 (127.18.100.67:39571), 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767)
I20260130 02:58:24.844075 19462 leader_election.cc:304] T cd30d7adcc4545848e3501a205a10293 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 78c2c46c7520492db35d51c244fb4f74, d870164263bb44b6b780a557daea3656; no voters: 371a36d219164974abdf848841528152
I20260130 02:58:24.845113 19462 leader_election.cc:304] T 00ddbfe6218c44d788fbf65245a31dc6 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 78c2c46c7520492db35d51c244fb4f74, d870164263bb44b6b780a557daea3656; no voters: 371a36d219164974abdf848841528152
I20260130 02:58:24.859596 19374 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dd377fab8c544bf69121dc9e9a060b29" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74" is_pre_election: true
I20260130 02:58:24.860227 19374 raft_consensus.cc:2468] T dd377fab8c544bf69121dc9e9a060b29 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d870164263bb44b6b780a557daea3656 in term 0.
I20260130 02:58:24.863744 20998 raft_consensus.cc:2804] T 00ddbfe6218c44d788fbf65245a31dc6 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260130 02:58:24.864081 20998 raft_consensus.cc:493] T 00ddbfe6218c44d788fbf65245a31dc6 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260130 02:58:24.864387 20998 raft_consensus.cc:3060] T 00ddbfe6218c44d788fbf65245a31dc6 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:24.867352 19462 leader_election.cc:304] T 39bcc41f10ae4241b370529364454586 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 78c2c46c7520492db35d51c244fb4f74, d870164263bb44b6b780a557daea3656; no voters: 371a36d219164974abdf848841528152
I20260130 02:58:24.867887 19313 leader_election.cc:304] T c76f0dc7762c49c389fa6fef41f02141 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 78c2c46c7520492db35d51c244fb4f74, d870164263bb44b6b780a557daea3656; no voters: 371a36d219164974abdf848841528152
I20260130 02:58:24.873673 19625 raft_consensus.cc:359] T 0d14fb95ace9447f810ccbed4d5befdf P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:24.874274 19625 raft_consensus.cc:385] T 0d14fb95ace9447f810ccbed4d5befdf P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:24.874531 19625 raft_consensus.cc:740] T 0d14fb95ace9447f810ccbed4d5befdf P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d870164263bb44b6b780a557daea3656, State: Initialized, Role: FOLLOWER
I20260130 02:58:24.875172 19625 consensus_queue.cc:260] T 0d14fb95ace9447f810ccbed4d5befdf P d870164263bb44b6b780a557daea3656 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
W20260130 02:58:24.876180 18861 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260130 02:58:23.124641 20810 rpcz_store.cc:276] 0130 02:58:17.960175 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:58:17.960338 (+ 163us) service_pool.cc:225] Handling call
0130 02:58:18.207757 (+247419us) raft_consensus.cc:1471] Updating replica for 328 ops
0130 02:58:18.498824 (+291067us) raft_consensus.cc:1534] Early marking committed up to index 0
0130 02:58:18.498897 (+ 73us) raft_consensus.cc:1539] Triggering prepare for 328 ops
0130 02:58:18.499696 (+ 799us) write_op.cc:270] Start()
0130 02:58:18.499852 (+ 156us) write_op.cc:276] Timestamp: P: 1769741872888141 usec, L: 0
0130 02:58:18.716243 (+216391us) write_op.cc:270] Start()
0130 02:58:18.716346 (+ 103us) write_op.cc:276] Timestamp: P: 1769741872903268 usec, L: 0
0130 02:58:18.716911 (+ 565us) write_op.cc:270] Start()
0130 02:58:18.717026 (+ 115us) write_op.cc:276] Timestamp: P: 1769741872906262 usec, L: 0
0130 02:58:18.717718 (+ 692us) write_op.cc:270] Start()
0130 02:58:18.717798 (+ 80us) write_op.cc:276] Timestamp: P: 1769741872909306 usec, L: 0
0130 02:58:18.718282 (+ 484us) write_op.cc:270] Start()
0130 02:58:18.718359 (+ 77us) write_op.cc:276] Timestamp: P: 1769741872942902 usec, L: 0
0130 02:58:18.718799 (+ 440us) write_op.cc:270] Start()
0130 02:58:18.718886 (+ 87us) write_op.cc:276] Timestamp: P: 1769741872957583 usec, L: 0
0130 02:58:18.864346 (+145460us) write_op.cc:270] Start()
0130 02:58:18.864518 (+ 172us) write_op.cc:276] Timestamp: P: 1769741872974246 usec, L: 0
0130 02:58:18.865023 (+ 505us) write_op.cc:270] Start()
0130 02:58:18.865108 (+ 85us) write_op.cc:276] Timestamp: P: 1769741872980094 usec, L: 0
0130 02:58:18.865789 (+ 681us) write_op.cc:270] Start()
0130 02:58:18.865872 (+ 83us) write_op.cc:276] Timestamp: P: 1769741872983463 usec, L: 0
0130 02:58:18.866350 (+ 478us) write_op.cc:270] Start()
0130 02:58:18.866434 (+ 84us) write_op.cc:276] Timestamp: P: 1769741873041321 usec, L: 0
0130 02:58:18.866922 (+ 488us) write_op.cc:270] Start()
0130 02:58:18.867007 (+ 85us) write_op.cc:276] Timestamp: P: 1769741873140800 usec, L: 0
0130 02:58:18.867493 (+ 486us) write_op.cc:270] Start()
0130 02:58:18.867598 (+ 105us) write_op.cc:276] Timestamp: P: 1769741873160050 usec, L: 0
0130 02:58:18.868080 (+ 482us) write_op.cc:270] Start()
0130 02:58:18.868159 (+ 79us) write_op.cc:276] Timestamp: P: 1769741873184923 usec, L: 0
0130 02:58:18.926418 (+ 58259us) write_op.cc:270] Start()
0130 02:58:18.926625 (+ 207us) write_op.cc:276] Timestamp: P: 1769741873192883 usec, L: 0
0130 02:58:18.927292 (+ 667us) write_op.cc:270] Start()
0130 02:58:18.927380 (+ 88us) write_op.cc:276] Timestamp: P: 1769741873244977 usec, L: 0
0130 02:58:18.927872 (+ 492us) write_op.cc:270] Start()
0130 02:58:18.927952 (+ 80us) write_op.cc:276] Timestamp: P: 1769741873249387 usec, L: 0
0130 02:58:18.928473 (+ 521us) write_op.cc:270] Start()
0130 02:58:18.928566 (+ 93us) write_op.cc:276] Timestamp: P: 1769741873252574 usec, L: 0
0130 02:58:18.928998 (+ 432us) write_op.cc:270] Start()
0130 02:58:18.929074 (+ 76us) write_op.cc:276] Timestamp: P: 1769741873294748 usec, L: 0
0130 02:58:18.973948 (+ 44874us) write_op.cc:270] Start()
0130 02:58:18.974171 (+ 223us) write_op.cc:276] Timestamp: P: 1769741873323033 usec, L: 0
0130 02:58:18.979677 (+ 5506us) write_op.cc:270] Start()
0130 02:58:18.979776 (+ 99us) write_op.cc:276] Timestamp: P: 1769741873342067 usec, L: 0
0130 02:58:18.981047 (+ 1271us) write_op.cc:270] Start()
0130 02:58:18.981136 (+ 89us) write_op.cc:276] Timestamp: P: 1769741873379500 usec, L: 0
0130 02:58:18.981649 (+ 513us) write_op.cc:270] Start()
0130 02:58:19.018619 (+ 36970us) write_op.cc:276] Timestamp: P: 1769741873391962 usec, L: 0
0130 02:58:19.019684 (+ 1065us) write_op.cc:270] Start()
0130 02:58:19.019789 (+ 105us) write_op.cc:276] Timestamp: P: 1769741873428434 usec, L: 0
0130 02:58:19.022547 (+ 2758us) write_op.cc:270] Start()
0130 02:58:19.022709 (+ 162us) write_op.cc:276] Timestamp: P: 1769741873486212 usec, L: 0
0130 02:58:19.023366 (+ 657us) write_op.cc:270] Start()
0130 02:58:19.023452 (+ 86us) write_op.cc:276] Timestamp: P: 1769741873493090 usec, L: 0
0130 02:58:19.024003 (+ 551us) write_op.cc:270] Start()
0130 02:58:19.024080 (+ 77us) write_op.cc:276] Timestamp: P: 1769741873514766 usec, L: 0
0130 02:58:19.024558 (+ 478us) write_op.cc:270] Start()
0130 02:58:19.024643 (+ 85us) write_op.cc:276] Timestamp: P: 1769741873528282 usec, L: 0
0130 02:58:19.072454 (+ 47811us) write_op.cc:270] Start()
0130 02:58:19.072574 (+ 120us) write_op.cc:276] Timestamp: P: 1769741873532011 usec, L: 0
0130 02:58:19.083224 (+ 10650us) write_op.cc:270] Start()
0130 02:58:19.083327 (+ 103us) write_op.cc:276] Timestamp: P: 1769741873569739 usec, L: 0
0130 02:58:19.083906 (+ 579us) write_op.cc:270] Start()
0130 02:58:19.083991 (+ 85us) write_op.cc:276] Timestamp: P: 1769741873615947 usec, L: 0
0130 02:58:19.084407 (+ 416us) write_op.cc:270] Start()
0130 02:58:19.084470 (+ 63us) write_op.cc:276] Timestamp: P: 1769741873630904 usec, L: 0
0130 02:58:19.084912 (+ 442us) write_op.cc:270] Start()
0130 02:58:19.084988 (+ 76us) write_op.cc:276] Timestamp: P: 1769741873671574 usec, L: 0
0130 02:58:19.098628 (+ 13640us) write_op.cc:270] Start()
0130 02:58:19.098725 (+ 97us) write_op.cc:276] Timestamp: P: 1769741873716606 usec, L: 0
0130 02:58:19.099256 (+ 531us) write_op.cc:270] Start()
0130 02:58:19.099333 (+ 77us) write_op.cc:276] Timestamp: P: 1769741873727862 usec, L: 0
0130 02:58:19.099795 (+ 462us) write_op.cc:270] Start()
0130 02:58:19.099877 (+ 82us) write_op.cc:276] Timestamp: P: 1769741873738064 usec, L: 0
0130 02:58:19.100519 (+ 642us) write_op.cc:270] Start()
0130 02:58:19.100603 (+ 84us) write_op.cc:276] Timestamp: P: 1769741873744156 usec, L: 0
0130 02:58:19.101051 (+ 448us) write_op.cc:270] Start()
0130 02:58:19.101121 (+ 70us) write_op.cc:276] Timestamp: P: 1769741873787860 usec, L: 0
0130 02:58:19.101563 (+ 442us) write_op.cc:270] Start()
0130 02:58:19.101641 (+ 78us) write_op.cc:276] Timestamp: P: 1769741873810047 usec, L: 0
0130 02:58:19.102110 (+ 469us) write_op.cc:270] Start()
0130 02:58:19.102187 (+ 77us) write_op.cc:276] Timestamp: P: 1769741874066447 usec, L: 0
0130 02:58:19.137836 (+ 35649us) write_op.cc:270] Start()
0130 02:58:19.137943 (+ 107us) write_op.cc:276] Timestamp: P: 1769741874071330 usec, L: 0
0130 02:58:19.138539 (+ 596us) write_op.cc:270] Start()
0130 02:58:19.138641 (+ 102us) write_op.cc:276] Timestamp: P: 1769741874074941 usec, L: 0
0130 02:58:19.139137 (+ 496us) write_op.cc:270] Start()
0130 02:58:19.139226 (+ 89us) write_op.cc:276] Timestamp: P: 1769741874086837 usec, L: 0
0130 02:58:19.139703 (+ 477us) write_op.cc:270] Start()
0130 02:58:19.139787 (+ 84us) write_op.cc:276] Timestamp: P: 1769741874088972 usec, L: 0
0130 02:58:19.140362 (+ 575us) write_op.cc:270] Start()
0130 02:58:19.140439 (+ 77us) write_op.cc:276] Timestamp: P: 1769741874092911 usec, L: 0
0130 02:58:19.140889 (+ 450us) write_op.cc:270] Start()
0130 02:58:19.140962 (+ 73us) write_op.cc:276] Timestamp: P: 1769741874178272 usec, L: 0
0130 02:58:19.141399 (+ 437us) write_op.cc:270] Start()
0130 02:58:19.141477 (+ 78us) write_op.cc:276] Timestamp: P: 1769741874208171 usec, L: 0
0130 02:58:19.141973 (+ 496us) write_op.cc:270] Start()
0130 02:58:19.142058 (+ 85us) write_op.cc:276] Timestamp: P: 1769741874249505 usec, L: 0
0130 02:58:19.142514 (+ 456us) write_op.cc:270] Start()
0130 02:58:19.170738 (+ 28224us) write_op.cc:276] Timestamp: P: 1769741874363926 usec, L: 0
0130 02:58:19.171566 (+ 828us) write_op.cc:270] Start()
0130 02:58:19.171651 (+ 85us) write_op.cc:276] Timestamp: P: 1769741874369121 usec, L: 0
0130 02:58:19.172141 (+ 490us) write_op.cc:270] Start()
0130 02:58:19.172779 (+ 638us) write_op.cc:276] Timestamp: P: 1769741874407448 usec, L: 0
0130 02:58:19.173288 (+ 509us) write_op.cc:270] Start()
0130 02:58:19.173364 (+ 76us) write_op.cc:276] Timestamp: P: 1769741874426581 usec, L: 0
0130 02:58:19.173817 (+ 453us) write_op.cc:270] Start()
0130 02:58:19.173892 (+ 75us) write_op.cc:276] Timestamp: P: 1769741874468318 usec, L: 0
0130 02:58:19.174428 (+ 536us) write_op.cc:270] Start()
0130 02:58:19.174498 (+ 70us) write_op.cc:276] Timestamp: P: 1769741874482131 usec, L: 0
0130 02:58:19.195230 (+ 20732us) write_op.cc:270] Start()
0130 02:58:19.195333 (+ 103us) write_op.cc:276] Timestamp: P: 1769741874606582 usec, L: 0
0130 02:58:19.202877 (+ 7544us) write_op.cc:270] Start()
0130 02:58:19.202976 (+ 99us) write_op.cc:276] Timestamp: P: 1769741874625227 usec, L: 0
0130 02:58:19.203534 (+ 558us) write_op.cc:270] Start()
0130 02:58:19.203614 (+ 80us) write_op.cc:276] Timestamp: P: 1769741874638925 usec, L: 0
0130 02:58:19.204536 (+ 922us) write_op.cc:270] Start()
0130 02:58:19.204621 (+ 85us) write_op.cc:276] Timestamp: P: 1769741874816441 usec, L: 0
0130 02:58:19.205082 (+ 461us) write_op.cc:270] Start()
0130 02:58:19.205154 (+ 72us) write_op.cc:276] Timestamp: P: 1769741874954068 usec, L: 0
0130 02:58:19.205707 (+ 553us) write_op.cc:270] Start()
0130 02:58:19.205784 (+ 77us) write_op.cc:276] Timestamp: P: 1769741874958839 usec, L: 0
0130 02:58:19.206288 (+ 504us) write_op.cc:270] Start()
0130 02:58:19.206442 (+ 154us) write_op.cc:276] Timestamp: P: 1769741874962199 usec, L: 0
0130 02:58:19.231213 (+ 24771us) write_op.cc:270] Start()
0130 02:58:19.231307 (+ 94us) write_op.cc:276] Timestamp: P: 1769741875017713 usec, L: 0
0130 02:58:19.231812 (+ 505us) write_op.cc:270] Start()
0130 02:58:19.231894 (+ 82us) write_op.cc:276] Timestamp: P: 1769741875111769 usec, L: 0
0130 02:58:19.232376 (+ 482us) write_op.cc:270] Start()
0130 02:58:19.232450 (+ 74us) write_op.cc:276] Timestamp: P: 1769741875159981 usec, L: 0
0130 02:58:19.232920 (+ 470us) write_op.cc:270] Start()
0130 02:58:19.233003 (+ 83us) write_op.cc:276] Timestamp: P: 1769741875277786 usec, L: 0
0130 02:58:19.234192 (+ 1189us) write_op.cc:270] Start()
0130 02:58:19.234270 (+ 78us) write_op.cc:276] Timestamp: P: 1769741875483815 usec, L: 0
0130 02:58:19.244811 (+ 10541us) write_op.cc:270] Start()
0130 02:58:19.244926 (+ 115us) write_op.cc:276] Timestamp: P: 1769741875646250 usec, L: 0
0130 02:58:19.245514 (+ 588us) write_op.cc:270] Start()
0130 02:58:19.245598 (+ 84us) write_op.cc:276] Timestamp: P: 1769741875649355 usec, L: 0
0130 02:58:19.246292 (+ 694us) write_op.cc:270] Start()
0130 02:58:19.246382 (+ 90us) write_op.cc:276] Timestamp: P: 1769741875653187 usec, L: 0
0130 02:58:19.246907 (+ 525us) write_op.cc:270] Start()
0130 02:58:19.247626 (+ 719us) write_op.cc:276] Timestamp: P: 1769741875702799 usec, L: 0
0130 02:58:19.248093 (+ 467us) write_op.cc:270] Start()
0130 02:58:19.248167 (+ 74us) write_op.cc:276] Timestamp: P: 1769741875819116 usec, L: 0
0130 02:58:19.248724 (+ 557us) write_op.cc:270] Start()
0130 02:58:19.248809 (+ 85us) write_op.cc:276] Timestamp: P: 1769741875927739 usec, L: 0
0130 02:58:19.249274 (+ 465us) write_op.cc:270] Start()
0130 02:58:19.249361 (+ 87us) write_op.cc:276] Timestamp: P: 1769741875950143 usec, L: 0
0130 02:58:19.249898 (+ 537us) write_op.cc:270] Start()
0130 02:58:19.249977 (+ 79us) write_op.cc:276] Timestamp: P: 1769741875967492 usec, L: 0
0130 02:58:19.250439 (+ 462us) write_op.cc:270] Start()
0130 02:58:19.250518 (+ 79us) write_op.cc:276] Timestamp: P: 1769741875971160 usec, L: 0
0130 02:58:19.349508 (+ 98990us) write_op.cc:270] Start()
0130 02:58:19.349600 (+ 92us) write_op.cc:276] Timestamp: P: 1769741875989616 usec, L: 0
0130 02:58:19.350107 (+ 507us) write_op.cc:270] Start()
0130 02:58:19.350200 (+ 93us) write_op.cc:276] Timestamp: P: 1769741876012470 usec, L: 0
0130 02:58:19.351107 (+ 907us) write_op.cc:270] Start()
0130 02:58:19.351207 (+ 100us) write_op.cc:276] Timestamp: P: 1769741876052557 usec, L: 0
0130 02:58:19.351692 (+ 485us) write_op.cc:270] Start()
0130 02:58:19.351769 (+ 77us) write_op.cc:276] Timestamp: P: 1769741876080546 usec, L: 0
0130 02:58:19.352212 (+ 443us) write_op.cc:270] Start()
0130 02:58:19.352285 (+ 73us) write_op.cc:276] Timestamp: P: 1769741876127196 usec, L: 0
0130 02:58:19.353302 (+ 1017us) write_op.cc:270] Start()
0130 02:58:19.353382 (+ 80us) write_op.cc:276] Timestamp: P: 1769741876252123 usec, L: 0
0130 02:58:19.447915 (+ 94533us) write_op.cc:270] Start()
0130 02:58:19.448023 (+ 108us) write_op.cc:276] Timestamp: P: 1769741876287055 usec, L: 0
0130 02:58:19.448635 (+ 612us) write_op.cc:270] Start()
0130 02:58:19.448719 (+ 84us) write_op.cc:276] Timestamp: P: 1769741876343885 usec, L: 0
0130 02:58:19.449197 (+ 478us) write_op.cc:270] Start()
0130 02:58:19.449278 (+ 81us) write_op.cc:276] Timestamp: P: 1769741876347012 usec, L: 0
0130 02:58:19.449877 (+ 599us) write_op.cc:270] Start()
0130 02:58:19.449953 (+ 76us) write_op.cc:276] Timestamp: P: 1769741876458092 usec, L: 0
0130 02:58:19.450400 (+ 447us) write_op.cc:270] Start()
0130 02:58:19.450498 (+ 98us) write_op.cc:276] Timestamp: P: 1769741876580009 usec, L: 0
0130 02:58:19.451029 (+ 531us) write_op.cc:270] Start()
0130 02:58:19.451110 (+ 81us) write_op.cc:276] Timestamp: P: 1769741876603347 usec, L: 0
0130 02:58:19.451590 (+ 480us) write_op.cc:270] Start()
0130 02:58:19.451662 (+ 72us) write_op.cc:276] Timestamp: P: 1769741876605385 usec, L: 0
0130 02:58:19.458825 (+ 7163us) write_op.cc:270] Start()
0130 02:58:19.458920 (+ 95us) write_op.cc:276] Timestamp: P: 1769741876607012 usec, L: 0
0130 02:58:19.459461 (+ 541us) write_op.cc:270] Start()
0130 02:58:19.459545 (+ 84us) write_op.cc:276] Timestamp: P: 1769741876623579 usec, L: 0
0130 02:58:19.460002 (+ 457us) write_op.cc:270] Start()
0130 02:58:19.460082 (+ 80us) write_op.cc:276] Timestamp: P: 1769741876656763 usec, L: 0
0130 02:58:19.460556 (+ 474us) write_op.cc:270] Start()
0130 02:58:19.460637 (+ 81us) write_op.cc:276] Timestamp: P: 1769741876678131 usec, L: 0
0130 02:58:19.461073 (+ 436us) write_op.cc:270] Start()
0130 02:58:19.461148 (+ 75us) write_op.cc:276] Timestamp: P: 1769741876700897 usec, L: 0
0130 02:58:19.461613 (+ 465us) write_op.cc:270] Start()
0130 02:58:19.461687 (+ 74us) write_op.cc:276] Timestamp: P: 1769741876813873 usec, L: 0
0130 02:58:19.462115 (+ 428us) write_op.cc:270] Start()
0130 02:58:19.462188 (+ 73us) write_op.cc:276] Timestamp: P: 1769741876915278 usec, L: 0
0130 02:58:19.626937 (+164749us) write_op.cc:270] Start()
0130 02:58:19.627033 (+ 96us) write_op.cc:276] Timestamp: P: 1769741876963260 usec, L: 0
0130 02:58:19.627654 (+ 621us) write_op.cc:270] Start()
0130 02:58:19.627730 (+ 76us) write_op.cc:276] Timestamp: P: 1769741876967134 usec, L: 0
0130 02:58:19.628175 (+ 445us) write_op.cc:270] Start()
0130 02:58:19.628245 (+ 70us) write_op.cc:276] Timestamp: P: 1769741877093264 usec, L: 0
0130 02:58:19.628693 (+ 448us) write_op.cc:270] Start()
0130 02:58:19.628884 (+ 191us) write_op.cc:276] Timestamp: P: 1769741877106311 usec, L: 0
0130 02:58:19.629452 (+ 568us) write_op.cc:270] Start()
0130 02:58:19.629530 (+ 78us) write_op.cc:276] Timestamp: P: 1769741877108163 usec, L: 0
0130 02:58:19.630025 (+ 495us) write_op.cc:270] Start()
0130 02:58:19.630101 (+ 76us) write_op.cc:276] Timestamp: P: 1769741877238043 usec, L: 0
0130 02:58:19.680205 (+ 50104us) write_op.cc:270] Start()
0130 02:58:19.680303 (+ 98us) write_op.cc:276] Timestamp: P: 1769741877332074 usec, L: 0
0130 02:58:19.680908 (+ 605us) write_op.cc:270] Start()
0130 02:58:19.680987 (+ 79us) write_op.cc:276] Timestamp: P: 1769741877442322 usec, L: 0
0130 02:58:19.681748 (+ 761us) write_op.cc:270] Start()
0130 02:58:19.681839 (+ 91us) write_op.cc:276] Timestamp: P: 1769741877452387 usec, L: 0
0130 02:58:19.682316 (+ 477us) write_op.cc:270] Start()
0130 02:58:19.682392 (+ 76us) write_op.cc:276] Timestamp: P: 1769741877526267 usec, L: 0
0130 02:58:19.682974 (+ 582us) write_op.cc:270] Start()
0130 02:58:19.683055 (+ 81us) write_op.cc:276] Timestamp: P: 1769741877528595 usec, L: 0
0130 02:58:19.683501 (+ 446us) write_op.cc:270] Start()
0130 02:58:19.683575 (+ 74us) write_op.cc:276] Timestamp: P: 1769741877617820 usec, L: 0
0130 02:58:19.844046 (+160471us) write_op.cc:270] Start()
0130 02:58:19.844149 (+ 103us) write_op.cc:276] Timestamp: P: 1769741877625892 usec, L: 0
0130 02:58:19.844806 (+ 657us) write_op.cc:270] Start()
0130 02:58:19.844899 (+ 93us) write_op.cc:276] Timestamp: P: 1769741877651855 usec, L: 0
0130 02:58:19.845405 (+ 506us) write_op.cc:270] Start()
0130 02:58:19.845479 (+ 74us) write_op.cc:276] Timestamp: P: 1769741877656349 usec, L: 0
0130 02:58:19.845939 (+ 460us) write_op.cc:270] Start()
0130 02:58:19.846022 (+ 83us) write_op.cc:276] Timestamp: P: 1769741877786002 usec, L: 0
0130 02:58:19.846510 (+ 488us) write_op.cc:270] Start()
0130 02:58:19.898629 (+ 52119us) write_op.cc:276] Timestamp: P: 1769741877876037 usec, L: 0
0130 02:58:19.899230 (+ 601us) write_op.cc:270] Start()
0130 02:58:19.899337 (+ 107us) write_op.cc:276] Timestamp: P: 1769741878081974 usec, L: 0
0130 02:58:19.899827 (+ 490us) write_op.cc:270] Start()
0130 02:58:19.899918 (+ 91us) write_op.cc:276] Timestamp: P: 1769741878084013 usec, L: 0
0130 02:58:19.900390 (+ 472us) write_op.cc:270] Start()
0130 02:58:19.900468 (+ 78us) write_op.cc:276] Timestamp: P: 1769741878095957 usec, L: 0
0130 02:58:19.900914 (+ 446us) write_op.cc:270] Start()
0130 02:58:19.900989 (+ 75us) write_op.cc:276] Timestamp: P: 1769741878116118 usec, L: 0
0130 02:58:19.901442 (+ 453us) write_op.cc:270] Start()
0130 02:58:19.901518 (+ 76us) write_op.cc:276] Timestamp: P: 1769741878128361 usec, L: 0
0130 02:58:19.902009 (+ 491us) write_op.cc:270] Start()
0130 02:58:19.902085 (+ 76us) write_op.cc:276] Timestamp: P: 1769741878290780 usec, L: 0
0130 02:58:19.902553 (+ 468us) write_op.cc:270] Start()
0130 02:58:19.955359 (+ 52806us) write_op.cc:276] Timestamp: P: 1769741878304712 usec, L: 0
0130 02:58:19.955971 (+ 612us) write_op.cc:270] Start()
0130 02:58:19.956061 (+ 90us) write_op.cc:276] Timestamp: P: 1769741878320155 usec, L: 0
0130 02:58:19.956544 (+ 483us) write_op.cc:270] Start()
0130 02:58:19.956617 (+ 73us) write_op.cc:276] Timestamp: P: 1769741878451298 usec, L: 0
0130 02:58:19.957574 (+ 957us) write_op.cc:270] Start()
0130 02:58:19.957649 (+ 75us) write_op.cc:276] Timestamp: P: 1769741878456643 usec, L: 0
0130 02:58:19.958047 (+ 398us) write_op.cc:270] Start()
0130 02:58:19.958110 (+ 63us) write_op.cc:276] Timestamp: P: 1769741878599755 usec, L: 0
0130 02:58:19.958491 (+ 381us) write_op.cc:270] Start()
0130 02:58:19.958552 (+ 61us) write_op.cc:276] Timestamp: P: 1769741878672591 usec, L: 0
0130 02:58:19.991371 (+ 32819us) write_op.cc:270] Start()
0130 02:58:19.991455 (+ 84us) write_op.cc:276] Timestamp: P: 1769741878692454 usec, L: 0
0130 02:58:19.991919 (+ 464us) write_op.cc:270] Start()
0130 02:58:19.991986 (+ 67us) write_op.cc:276] Timestamp: P: 1769741878752467 usec, L: 0
0130 02:58:19.994862 (+ 2876us) write_op.cc:270] Start()
0130 02:58:19.994935 (+ 73us) write_op.cc:276] Timestamp: P: 1769741878766234 usec, L: 0
0130 02:58:20.012114 (+ 17179us) write_op.cc:270] Start()
0130 02:58:20.012200 (+ 86us) write_op.cc:276] Timestamp: P: 1769741878810121 usec, L: 0
0130 02:58:20.012734 (+ 534us) write_op.cc:270] Start()
0130 02:58:20.012809 (+ 75us) write_op.cc:276] Timestamp: P: 1769741878889257 usec, L: 0
0130 02:58:20.013412 (+ 603us) write_op.cc:270] Start()
0130 02:58:20.013481 (+ 69us) write_op.cc:276] Timestamp: P: 1769741878891319 usec, L: 0
0130 02:58:20.013916 (+ 435us) write_op.cc:270] Start()
0130 02:58:20.013983 (+ 67us) write_op.cc:276] Timestamp: P: 1769741878905933 usec, L: 0
0130 02:58:20.014603 (+ 620us) write_op.cc:270] Start()
0130 02:58:20.078810 (+ 64207us) write_op.cc:276] Timestamp: P: 1769741879005570 usec, L: 0
0130 02:58:20.079500 (+ 690us) write_op.cc:270] Start()
0130 02:58:20.079587 (+ 87us) write_op.cc:276] Timestamp: P: 1769741879070734 usec, L: 0
0130 02:58:20.080052 (+ 465us) write_op.cc:270] Start()
0130 02:58:20.080119 (+ 67us) write_op.cc:276] Timestamp: P: 1769741879185043 usec, L: 0
0130 02:58:20.146195 (+ 66076us) write_op.cc:270] Start()
0130 02:58:20.146322 (+ 127us) write_op.cc:276] Timestamp: P: 1769741879270398 usec, L: 0
0130 02:58:20.179437 (+ 33115us) write_op.cc:270] Start()
0130 02:58:20.179552 (+ 115us) write_op.cc:276] Timestamp: P: 1769741879539889 usec, L: 0
0130 02:58:20.180053 (+ 501us) write_op.cc:270] Start()
0130 02:58:20.180122 (+ 69us) write_op.cc:276] Timestamp: P: 1769741879550645 usec, L: 0
0130 02:58:20.180520 (+ 398us) write_op.cc:270] Start()
0130 02:58:20.180598 (+ 78us) write_op.cc:276] Timestamp: P: 1769741879564302 usec, L: 0
0130 02:58:20.181008 (+ 410us) write_op.cc:270] Start()
0130 02:58:20.181072 (+ 64us) write_op.cc:276] Timestamp: P: 1769741879569574 usec, L: 0
0130 02:58:20.181474 (+ 402us) write_op.cc:270] Start()
0130 02:58:20.181543 (+ 69us) write_op.cc:276] Timestamp: P: 1769741879665665 usec, L: 0
0130 02:58:20.181944 (+ 401us) write_op.cc:270] Start()
0130 02:58:20.182007 (+ 63us) write_op.cc:276] Timestamp: P: 1769741879699482 usec, L: 0
0130 02:58:20.182394 (+ 387us) write_op.cc:270] Start()
0130 02:58:20.182455 (+ 61us) write_op.cc:276] Timestamp: P: 1769741879701318 usec, L: 0
0130 02:58:20.183115 (+ 660us) write_op.cc:270] Start()
0130 02:58:20.183189 (+ 74us) write_op.cc:276] Timestamp: P: 1769741880030654 usec, L: 0
0130 02:58:20.183599 (+ 410us) write_op.cc:270] Start()
0130 02:58:20.183676 (+ 77us) write_op.cc:276] Timestamp: P: 1769741880071629 usec, L: 0
0130 02:58:20.184053 (+ 377us) write_op.cc:270] Start()
0130 02:58:20.184115 (+ 62us) write_op.cc:276] Timestamp: P: 1769741880179703 usec, L: 0
0130 02:58:20.184727 (+ 612us) write_op.cc:270] Start()
0130 02:58:20.184793 (+ 66us) write_op.cc:276] Timestamp: P: 1769741880201995 usec, L: 0
0130 02:58:20.185181 (+ 388us) write_op.cc:270] Start()
0130 02:58:20.185242 (+ 61us) write_op.cc:276] Timestamp: P: 1769741880261485 usec, L: 0
0130 02:58:20.185630 (+ 388us) write_op.cc:270] Start()
0130 02:58:20.185694 (+ 64us) write_op.cc:276] Timestamp: P: 1769741880380263 usec, L: 0
0130 02:58:20.186046 (+ 352us) write_op.cc:270] Start()
0130 02:58:20.186105 (+ 59us) write_op.cc:276] Timestamp: P: 1769741880406752 usec, L: 0
0130 02:58:20.186512 (+ 407us) write_op.cc:270] Start()
0130 02:58:20.186620 (+ 108us) write_op.cc:276] Timestamp: P: 1769741880487941 usec, L: 0
0130 02:58:20.187024 (+ 404us) write_op.cc:270] Start()
0130 02:58:20.187090 (+ 66us) write_op.cc:276] Timestamp: P: 1769741880565639 usec, L: 0
0130 02:58:20.187472 (+ 382us) write_op.cc:270] Start()
0130 02:58:20.187543 (+ 71us) write_op.cc:276] Timestamp: P: 1769741880568319 usec, L: 0
0130 02:58:20.188084 (+ 541us) write_op.cc:270] Start()
0130 02:58:20.188154 (+ 70us) write_op.cc:276] Timestamp: P: 1769741880702617 usec, L: 0
0130 02:58:20.188550 (+ 396us) write_op.cc:270] Start()
0130 02:58:20.188639 (+ 89us) write_op.cc:276] Timestamp: P: 1769741880761687 usec, L: 0
0130 02:58:20.188999 (+ 360us) write_op.cc:270] Start()
0130 02:58:20.189059 (+ 60us) write_op.cc:276] Timestamp: P: 1769741881010305 usec, L: 0
0130 02:58:20.189407 (+ 348us) write_op.cc:270] Start()
0130 02:58:20.189495 (+ 88us) write_op.cc:276] Timestamp: P: 1769741881169818 usec, L: 0
0130 02:58:20.189860 (+ 365us) write_op.cc:270] Start()
0130 02:58:20.189920 (+ 60us) write_op.cc:276] Timestamp: P: 1769741881223718 usec, L: 0
0130 02:58:20.190294 (+ 374us) write_op.cc:270] Start()
0130 02:58:20.190356 (+ 62us) write_op.cc:276] Timestamp: P: 1769741881253547 usec, L: 0
0130 02:58:20.190736 (+ 380us) write_op.cc:270] Start()
0130 02:58:20.190803 (+ 67us) write_op.cc:276] Timestamp: P: 1769741881296229 usec, L: 0
0130 02:58:20.191152 (+ 349us) write_op.cc:270] Start()
0130 02:58:20.191217 (+ 65us) write_op.cc:276] Timestamp: P: 1769741881311299 usec, L: 0
0130 02:58:20.191578 (+ 361us) write_op.cc:270] Start()
0130 02:58:20.191638 (+ 60us) write_op.cc:276] Timestamp: P: 1769741881314012 usec, L: 0
0130 02:58:20.192004 (+ 366us) write_op.cc:270] Start()
0130 02:58:20.192064 (+ 60us) write_op.cc:276] Timestamp: P: 1769741881354191 usec, L: 0
0130 02:58:20.192422 (+ 358us) write_op.cc:270] Start()
0130 02:58:20.192480 (+ 58us) write_op.cc:276] Timestamp: P: 1769741881431505 usec, L: 0
0130 02:58:20.192828 (+ 348us) write_op.cc:270] Start()
0130 02:58:20.192896 (+ 68us) write_op.cc:276] Timestamp: P: 1769741881437948 usec, L: 0
0130 02:58:20.193267 (+ 371us) write_op.cc:270] Start()
0130 02:58:20.193330 (+ 63us) write_op.cc:276] Timestamp: P: 1769741881442113 usec, L: 0
0130 02:58:20.193851 (+ 521us) write_op.cc:270] Start()
0130 02:58:20.193931 (+ 80us) write_op.cc:276] Timestamp: P: 1769741881596326 usec, L: 0
0130 02:58:20.194352 (+ 421us) write_op.cc:270] Start()
0130 02:58:20.194416 (+ 64us) write_op.cc:276] Timestamp: P: 1769741881602460 usec, L: 0
0130 02:58:20.194931 (+ 515us) write_op.cc:270] Start()
0130 02:58:20.195026 (+ 95us) write_op.cc:276] Timestamp: P: 1769741881721302 usec, L: 0
0130 02:58:20.195416 (+ 390us) write_op.cc:270] Start()
0130 02:58:20.195481 (+ 65us) write_op.cc:276] Timestamp: P: 1769741881756070 usec, L: 0
0130 02:58:20.195873 (+ 392us) write_op.cc:270] Start()
0130 02:58:20.195935 (+ 62us) write_op.cc:276] Timestamp: P: 1769741881806211 usec, L: 0
0130 02:58:20.196325 (+ 390us) write_op.cc:270] Start()
0130 02:58:20.196391 (+ 66us) write_op.cc:276] Timestamp: P: 1769741881942256 usec, L: 0
0130 02:58:20.196792 (+ 401us) write_op.cc:270] Start()
0130 02:58:20.196861 (+ 69us) write_op.cc:276] Timestamp: P: 1769741881946272 usec, L: 0
0130 02:58:20.197245 (+ 384us) write_op.cc:270] Start()
0130 02:58:20.197313 (+ 68us) write_op.cc:276] Timestamp: P: 1769741881949566 usec, L: 0
0130 02:58:20.197708 (+ 395us) write_op.cc:270] Start()
0130 02:58:20.197774 (+ 66us) write_op.cc:276] Timestamp: P: 1769741881976338 usec, L: 0
0130 02:58:20.198160 (+ 386us) write_op.cc:270] Start()
0130 02:58:20.198224 (+ 64us) write_op.cc:276] Timestamp: P: 1769741881994211 usec, L: 0
0130 02:58:20.198700 (+ 476us) write_op.cc:270] Start()
0130 02:58:20.198777 (+ 77us) write_op.cc:276] Timestamp: P: 1769741882023582 usec, L: 0
0130 02:58:20.199216 (+ 439us) write_op.cc:270] Start()
0130 02:58:20.199280 (+ 64us) write_op.cc:276] Timestamp: P: 1769741882051968 usec, L: 0
0130 02:58:20.199687 (+ 407us) write_op.cc:270] Start()
0130 02:58:20.199753 (+ 66us) write_op.cc:276] Timestamp: P: 1769741882061364 usec, L: 0
0130 02:58:20.200115 (+ 362us) write_op.cc:270] Start()
0130 02:58:20.200175 (+ 60us) write_op.cc:276] Timestamp: P: 1769741882097650 usec, L: 0
0130 02:58:20.200547 (+ 372us) write_op.cc:270] Start()
0130 02:58:20.200622 (+ 75us) write_op.cc:276] Timestamp: P: 1769741882119475 usec, L: 0
0130 02:58:20.200978 (+ 356us) write_op.cc:270] Start()
0130 02:58:20.201043 (+ 65us) write_op.cc:276] Timestamp: P: 1769741882199829 usec, L: 0
0130 02:58:20.201412 (+ 369us) write_op.cc:270] Start()
0130 02:58:20.201476 (+ 64us) write_op.cc:276] Timestamp: P: 1769741882244439 usec, L: 0
0130 02:58:20.201880 (+ 404us) write_op.cc:270] Start()
0130 02:58:20.201945 (+ 65us) write_op.cc:276] Timestamp: P: 1769741882266039 usec, L: 0
0130 02:58:20.202320 (+ 375us) write_op.cc:270] Start()
0130 02:58:20.202390 (+ 70us) write_op.cc:276] Timestamp: P: 1769741882415988 usec, L: 0
0130 02:58:20.202816 (+ 426us) write_op.cc:270] Start()
0130 02:58:20.202881 (+ 65us) write_op.cc:276] Timestamp: P: 1769741882566839 usec, L: 0
0130 02:58:20.203283 (+ 402us) write_op.cc:270] Start()
0130 02:58:20.203348 (+ 65us) write_op.cc:276] Timestamp: P: 1769741882617960 usec, L: 0
0130 02:58:20.203758 (+ 410us) write_op.cc:270] Start()
0130 02:58:20.203822 (+ 64us) write_op.cc:276] Timestamp: P: 1769741883164730 usec, L: 0
0130 02:58:20.204221 (+ 399us) write_op.cc:270] Start()
0130 02:58:20.204298 (+ 77us) write_op.cc:276] Timestamp: P: 1769741883206789 usec, L: 0
0130 02:58:20.204860 (+ 562us) write_op.cc:270] Start()
0130 02:58:20.204934 (+ 74us) write_op.cc:276] Timestamp: P: 1769741883209987 usec, L: 0
0130 02:58:20.205345 (+ 411us) write_op.cc:270] Start()
0130 02:58:20.205408 (+ 63us) write_op.cc:276] Timestamp: P: 1769741883212859 usec, L: 0
0130 02:58:20.205788 (+ 380us) write_op.cc:270] Start()
0130 02:58:20.205849 (+ 61us) write_op.cc:276] Timestamp: P: 1769741883234114 usec, L: 0
0130 02:58:20.206237 (+ 388us) write_op.cc:270] Start()
0130 02:58:20.206304 (+ 67us) write_op.cc:276] Timestamp: P: 1769741883387089 usec, L: 0
0130 02:58:20.206755 (+ 451us) write_op.cc:270] Start()
0130 02:58:20.206820 (+ 65us) write_op.cc:276] Timestamp: P: 1769741883426671 usec, L: 0
0130 02:58:20.207246 (+ 426us) write_op.cc:270] Start()
0130 02:58:20.207310 (+ 64us) write_op.cc:276] Timestamp: P: 1769741883450551 usec, L: 0
0130 02:58:20.208125 (+ 815us) write_op.cc:270] Start()
0130 02:58:20.208202 (+ 77us) write_op.cc:276] Timestamp: P: 1769741883476785 usec, L: 0
0130 02:58:20.208600 (+ 398us) write_op.cc:270] Start()
0130 02:58:20.208667 (+ 67us) write_op.cc:276] Timestamp: P: 1769741883482215 usec, L: 0
0130 02:58:20.209063 (+ 396us) write_op.cc:270] Start()
0130 02:58:20.209131 (+ 68us) write_op.cc:276] Timestamp: P: 1769741883506036 usec, L: 0
0130 02:58:20.210063 (+ 932us) write_op.cc:270] Start()
0130 02:58:20.210128 (+ 65us) write_op.cc:276] Timestamp: P: 17697418
I20260130 02:58:24.841907 21121 raft_consensus.cc:515] T 94355fc1338f479e8798070aff04fcba P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:24.884641 21121 leader_election.cc:290] T 94355fc1338f479e8798070aff04fcba P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 election: Requested vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), d870164263bb44b6b780a557daea3656 (127.18.100.67:39571)
I20260130 02:58:24.886866 20704 raft_consensus.cc:515] T 306452e208ca4e719609f67345114a13 P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:24.886529 19218 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "94355fc1338f479e8798070aff04fcba" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152"
I20260130 02:58:24.888715 20704 leader_election.cc:290] T 306452e208ca4e719609f67345114a13 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 election: Requested vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), d870164263bb44b6b780a557daea3656 (127.18.100.67:39571)
I20260130 02:58:24.889240 19224 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "306452e208ca4e719609f67345114a13" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152"
I20260130 02:58:24.889760 19224 raft_consensus.cc:3060] T 306452e208ca4e719609f67345114a13 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:24.890805 19840 tablet_bootstrap.cc:492] T e691c238199b4914bcc3fd248f31420c P 78c2c46c7520492db35d51c244fb4f74: No bootstrap required, opened a new log
I20260130 02:58:24.891216 19840 ts_tablet_manager.cc:1403] T e691c238199b4914bcc3fd248f31420c P 78c2c46c7520492db35d51c244fb4f74: Time spent bootstrapping tablet: real 0.192s user 0.001s sys 0.009s
I20260130 02:58:24.893568 19840 raft_consensus.cc:359] T e691c238199b4914bcc3fd248f31420c P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:24.894186 19840 raft_consensus.cc:385] T e691c238199b4914bcc3fd248f31420c P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:24.896337 19224 raft_consensus.cc:2468] T 306452e208ca4e719609f67345114a13 P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78c2c46c7520492db35d51c244fb4f74 in term 1.
I20260130 02:58:24.897346 19314 leader_election.cc:304] T 306452e208ca4e719609f67345114a13 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 371a36d219164974abdf848841528152, 78c2c46c7520492db35d51c244fb4f74; no voters:
I20260130 02:58:24.901755 21124 raft_consensus.cc:2804] T 306452e208ca4e719609f67345114a13 P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Leader election won for term 1
I20260130 02:58:24.901958 21133 raft_consensus.cc:2804] T c76f0dc7762c49c389fa6fef41f02141 P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Leader election won for term 1
I20260130 02:58:24.902179 21124 raft_consensus.cc:697] T 306452e208ca4e719609f67345114a13 P 78c2c46c7520492db35d51c244fb4f74 [term 1 LEADER]: Becoming Leader. State: Replica: 78c2c46c7520492db35d51c244fb4f74, State: Running, Role: LEADER
I20260130 02:58:24.902408 21133 raft_consensus.cc:697] T c76f0dc7762c49c389fa6fef41f02141 P 78c2c46c7520492db35d51c244fb4f74 [term 1 LEADER]: Becoming Leader. State: Replica: 78c2c46c7520492db35d51c244fb4f74, State: Running, Role: LEADER
I20260130 02:58:24.902563 21164 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260130 02:58:24.903124 21124 consensus_queue.cc:237] T 306452e208ca4e719609f67345114a13 P 78c2c46c7520492db35d51c244fb4f74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:24.903280 21133 consensus_queue.cc:237] T c76f0dc7762c49c389fa6fef41f02141 P 78c2c46c7520492db35d51c244fb4f74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
W20260130 02:58:24.909663 19314 leader_election.cc:343] T 94355fc1338f479e8798070aff04fcba P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:24.916406 19807 tablet_bootstrap.cc:654] T cd30d7adcc4545848e3501a205a10293 P 371a36d219164974abdf848841528152: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:24.921211 21167 raft_consensus.cc:2804] T 39bcc41f10ae4241b370529364454586 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260130 02:58:24.921602 21167 raft_consensus.cc:493] T 39bcc41f10ae4241b370529364454586 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260130 02:58:24.921927 21167 raft_consensus.cc:3060] T 39bcc41f10ae4241b370529364454586 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:24.894430 19840 raft_consensus.cc:740] T e691c238199b4914bcc3fd248f31420c P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78c2c46c7520492db35d51c244fb4f74, State: Initialized, Role: FOLLOWER
W20260130 02:58:24.944715 19462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.100.66:40767, user_credentials={real_user=slave}} blocked reactor thread for 77435.4us
I20260130 02:58:24.943238 19840 consensus_queue.cc:260] T e691c238199b4914bcc3fd248f31420c P 78c2c46c7520492db35d51c244fb4f74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:24.946280 19520 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "94355fc1338f479e8798070aff04fcba" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656"
I20260130 02:58:24.947010 19520 raft_consensus.cc:2393] T 94355fc1338f479e8798070aff04fcba P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 78c2c46c7520492db35d51c244fb4f74 in current term 1: Already voted for candidate d870164263bb44b6b780a557daea3656 in this term.
I20260130 02:58:24.885509 21000 raft_consensus.cc:515] T 306452e208ca4e719609f67345114a13 P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:24.948066 19313 leader_election.cc:304] T 94355fc1338f479e8798070aff04fcba P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78c2c46c7520492db35d51c244fb4f74; no voters: 371a36d219164974abdf848841528152, d870164263bb44b6b780a557daea3656
I20260130 02:58:24.948736 21141 raft_consensus.cc:2749] T 94355fc1338f479e8798070aff04fcba P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260130 02:58:24.949796 19524 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "306452e208ca4e719609f67345114a13" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656"
I20260130 02:58:24.953583 21080 catalog_manager.cc:1584] T 00000000000000000000000000000000 P 3749baa287ed4f249329a74071781edd: acquiring CA information for follower catalog manager: success
I20260130 02:58:24.956117 20998 raft_consensus.cc:515] T 00ddbfe6218c44d788fbf65245a31dc6 P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
E20260130 02:58:24.957218 19102 catalog_manager.cc:826] Error processing pending assignments: Illegal state: error persisting updated tablet metadata: replica 658299197a424f53a37f14fadf089ae4 is not leader of this config: current role FOLLOWER
I20260130 02:58:24.960163 19515 raft_consensus.cc:2468] T cd30d7adcc4545848e3501a205a10293 P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78c2c46c7520492db35d51c244fb4f74 in term 1.
I20260130 02:58:24.960768 21085 raft_consensus.cc:2764] T cd30d7adcc4545848e3501a205a10293 P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20260130 02:58:24.964716 19524 raft_consensus.cc:2393] T 306452e208ca4e719609f67345114a13 P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 78c2c46c7520492db35d51c244fb4f74 in current term 1: Already voted for candidate d870164263bb44b6b780a557daea3656 in this term.
I20260130 02:58:24.965714 19840 ts_tablet_manager.cc:1434] T e691c238199b4914bcc3fd248f31420c P 78c2c46c7520492db35d51c244fb4f74: Time spent starting tablet: real 0.074s user 0.005s sys 0.000s
I20260130 02:58:24.966866 19313 leader_election.cc:304] T cd30d7adcc4545848e3501a205a10293 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 78c2c46c7520492db35d51c244fb4f74, d870164263bb44b6b780a557daea3656; no voters: 371a36d219164974abdf848841528152
I20260130 02:58:24.967557 21141 raft_consensus.cc:2804] T cd30d7adcc4545848e3501a205a10293 P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Leader election won for term 1
I20260130 02:58:24.968000 21141 raft_consensus.cc:697] T cd30d7adcc4545848e3501a205a10293 P 78c2c46c7520492db35d51c244fb4f74 [term 1 LEADER]: Becoming Leader. State: Replica: 78c2c46c7520492db35d51c244fb4f74, State: Running, Role: LEADER
I20260130 02:58:24.968544 21000 leader_election.cc:290] T 306452e208ca4e719609f67345114a13 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 election: Requested vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767)
I20260130 02:58:24.968797 21141 consensus_queue.cc:237] T cd30d7adcc4545848e3501a205a10293 P 78c2c46c7520492db35d51c244fb4f74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:24.978085 20998 leader_election.cc:290] T 00ddbfe6218c44d788fbf65245a31dc6 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 election: Requested vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767)
I20260130 02:58:24.919364 21190 raft_consensus.cc:515] T cfd12a72726846e587cff9d6a4d61bd6 P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:24.982221 19840 tablet_bootstrap.cc:492] T 532e867fd5944d468d7672d2a5c92f3c P 78c2c46c7520492db35d51c244fb4f74: Bootstrap starting.
I20260130 02:58:24.987040 21186 raft_consensus.cc:493] T 00000000000000000000000000000000 P 3749baa287ed4f249329a74071781edd [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 107f274cdda543038832f06a56ab9130)
W20260130 02:58:24.987303 19463 leader_election.cc:343] T e2aaaf00f53d418b87d5c7af286e4d84 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Not found: Tablet not found: e2aaaf00f53d418b87d5c7af286e4d84
I20260130 02:58:24.987625 21186 raft_consensus.cc:515] T 00000000000000000000000000000000 P 3749baa287ed4f249329a74071781edd [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "658299197a424f53a37f14fadf089ae4" member_type: VOTER last_known_addr { host: "127.18.100.126" port: 38191 } } peers { permanent_uuid: "107f274cdda543038832f06a56ab9130" member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39711 } } peers { permanent_uuid: "3749baa287ed4f249329a74071781edd" member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34629 } }
I20260130 02:58:24.988940 21190 leader_election.cc:290] T cfd12a72726846e587cff9d6a4d61bd6 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 election: Requested vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767)
I20260130 02:58:24.989506 21186 leader_election.cc:290] T 00000000000000000000000000000000 P 3749baa287ed4f249329a74071781edd [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 658299197a424f53a37f14fadf089ae4 (127.18.100.126:38191), 107f274cdda543038832f06a56ab9130 (127.18.100.125:39711)
I20260130 02:58:24.991600 18960 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3749baa287ed4f249329a74071781edd" candidate_term: 4 candidate_status { last_received { term: 3 index: 372 } } ignore_live_leader: false dest_uuid: "107f274cdda543038832f06a56ab9130" is_pre_election: true
I20260130 02:58:24.992660 18885 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3749baa287ed4f249329a74071781edd" candidate_term: 4 candidate_status { last_received { term: 3 index: 372 } } ignore_live_leader: false dest_uuid: "658299197a424f53a37f14fadf089ae4" is_pre_election: true
I20260130 02:58:24.994146 20613 leader_election.cc:304] T 00000000000000000000000000000000 P 3749baa287ed4f249329a74071781edd [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3749baa287ed4f249329a74071781edd; no voters: 107f274cdda543038832f06a56ab9130, 658299197a424f53a37f14fadf089ae4
I20260130 02:58:24.995332 21186 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 3749baa287ed4f249329a74071781edd [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260130 02:58:24.995937 19462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.100.66:40767, user_credentials={real_user=slave}} blocked reactor thread for 50612.5us
I20260130 02:58:24.996068 20998 consensus_queue.cc:1048] T ab6afa40408a4f458495abf20290e55b P d870164263bb44b6b780a557daea3656 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260130 02:58:24.997071 19462 leader_election.cc:304] T dd377fab8c544bf69121dc9e9a060b29 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 78c2c46c7520492db35d51c244fb4f74, d870164263bb44b6b780a557daea3656; no voters: 371a36d219164974abdf848841528152
I20260130 02:58:24.998632 19625 ts_tablet_manager.cc:1434] T 0d14fb95ace9447f810ccbed4d5befdf P d870164263bb44b6b780a557daea3656: Time spent starting tablet: real 0.237s user 0.008s sys 0.001s
W20260130 02:58:24.999179 19463 leader_election.cc:343] T 94355fc1338f479e8798070aff04fcba P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:25.000603 21133 raft_consensus.cc:493] T 8b47653c7c3945a68e9bf4680bfc0496 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:25.001108 21133 raft_consensus.cc:515] T 8b47653c7c3945a68e9bf4680bfc0496 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:25.003299 21133 leader_election.cc:290] T 8b47653c7c3945a68e9bf4680bfc0496 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), d870164263bb44b6b780a557daea3656 (127.18.100.67:39571)
I20260130 02:58:25.007486 21094 raft_consensus.cc:515] T 2b8034a92b2043e5b57cf02f1ee6296d P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:25.009367 21094 leader_election.cc:290] T 2b8034a92b2043e5b57cf02f1ee6296d P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 election: Requested vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767)
I20260130 02:58:25.013561 19218 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8b47653c7c3945a68e9bf4680bfc0496" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152" is_pre_election: true
I20260130 02:58:25.014153 19159 leader_election.cc:304] T 0855d88806914a36b4bf6b49036f0ad0 P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 371a36d219164974abdf848841528152; no voters: 78c2c46c7520492db35d51c244fb4f74, d870164263bb44b6b780a557daea3656
W20260130 02:58:25.014665 19314 leader_election.cc:343] T 8b47653c7c3945a68e9bf4680bfc0496 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:25.015489 19523 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8b47653c7c3945a68e9bf4680bfc0496" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
I20260130 02:58:25.016721 19840 tablet_bootstrap.cc:654] T 532e867fd5944d468d7672d2a5c92f3c P 78c2c46c7520492db35d51c244fb4f74: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:25.017455 19625 tablet_bootstrap.cc:492] T 1129f72158ec4bae89da3c94cdb70839 P d870164263bb44b6b780a557daea3656: Bootstrap starting.
I20260130 02:58:25.025372 21211 raft_consensus.cc:2804] T dd377fab8c544bf69121dc9e9a060b29 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260130 02:58:25.026312 21211 raft_consensus.cc:493] T dd377fab8c544bf69121dc9e9a060b29 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260130 02:58:25.016764 19313 leader_election.cc:343] T 8b47653c7c3945a68e9bf4680bfc0496 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:25.026671 21211 raft_consensus.cc:3060] T dd377fab8c544bf69121dc9e9a060b29 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:25.026835 19313 leader_election.cc:304] T 8b47653c7c3945a68e9bf4680bfc0496 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78c2c46c7520492db35d51c244fb4f74; no voters: 371a36d219164974abdf848841528152, d870164263bb44b6b780a557daea3656
I20260130 02:58:25.028029 20704 raft_consensus.cc:2749] T 8b47653c7c3945a68e9bf4680bfc0496 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260130 02:58:25.035684 19218 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "306452e208ca4e719609f67345114a13" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152"
I20260130 02:58:25.036448 19218 raft_consensus.cc:2393] T 306452e208ca4e719609f67345114a13 P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate d870164263bb44b6b780a557daea3656 in current term 1: Already voted for candidate 78c2c46c7520492db35d51c244fb4f74 in this term.
I20260130 02:58:25.042559 19374 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1fc8f09d6329407c8078542b4123ef4d" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74"
I20260130 02:58:25.045087 19375 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "94355fc1338f479e8798070aff04fcba" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74"
I20260130 02:58:25.045267 21169 raft_consensus.cc:3060] T 0855d88806914a36b4bf6b49036f0ad0 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:25.045730 19375 raft_consensus.cc:2393] T 94355fc1338f479e8798070aff04fcba P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate d870164263bb44b6b780a557daea3656 in current term 1: Already voted for candidate 78c2c46c7520492db35d51c244fb4f74 in this term.
I20260130 02:58:25.052933 18875 catalog_manager.cc:5746] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021) does not have the latest schema for tablet 345eaaa5625c48d099280b7fbb2e514b (table default.Table_c1886dd9a7934e7797e0e7e0b065d692 [id=7a2d5367224b492b844199088b01b149]). Expected version 1 got 0
I20260130 02:58:25.054443 21169 raft_consensus.cc:2749] T 0855d88806914a36b4bf6b49036f0ad0 P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260130 02:58:25.056274 18875 catalog_manager.cc:5746] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021) does not have the latest schema for tablet b1e8b78eff0d4a0b92059adb2de4a3ce (table default.Table_d0b4516d24384e5386c1debaf2543adb [id=f1342f9497b74e7a9ecad1de8082578d]). Expected version 1 got 0
I20260130 02:58:25.060668 19224 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cfd12a72726846e587cff9d6a4d61bd6" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152"
I20260130 02:58:25.061954 19218 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00ddbfe6218c44d788fbf65245a31dc6" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152"
I20260130 02:58:25.064838 19840 tablet_bootstrap.cc:492] T 532e867fd5944d468d7672d2a5c92f3c P 78c2c46c7520492db35d51c244fb4f74: No bootstrap required, opened a new log
I20260130 02:58:25.065397 19840 ts_tablet_manager.cc:1403] T 532e867fd5944d468d7672d2a5c92f3c P 78c2c46c7520492db35d51c244fb4f74: Time spent bootstrapping tablet: real 0.083s user 0.010s sys 0.000s
I20260130 02:58:25.065182 21167 raft_consensus.cc:515] T 39bcc41f10ae4241b370529364454586 P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:25.067479 21167 leader_election.cc:290] T 39bcc41f10ae4241b370529364454586 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 election: Requested vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767)
I20260130 02:58:25.067683 19840 raft_consensus.cc:359] T 532e867fd5944d468d7672d2a5c92f3c P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:25.068295 19840 raft_consensus.cc:385] T 532e867fd5944d468d7672d2a5c92f3c P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:25.068547 19840 raft_consensus.cc:740] T 532e867fd5944d468d7672d2a5c92f3c P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78c2c46c7520492db35d51c244fb4f74, State: Initialized, Role: FOLLOWER
I20260130 02:58:25.068992 19218 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2b8034a92b2043e5b57cf02f1ee6296d" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152"
I20260130 02:58:25.080156 21173 raft_consensus.cc:493] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 107f274cdda543038832f06a56ab9130)
I20260130 02:58:25.080664 21173 raft_consensus.cc:515] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "658299197a424f53a37f14fadf089ae4" member_type: VOTER last_known_addr { host: "127.18.100.126" port: 38191 } } peers { permanent_uuid: "107f274cdda543038832f06a56ab9130" member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39711 } } peers { permanent_uuid: "3749baa287ed4f249329a74071781edd" member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34629 } }
I20260130 02:58:25.081604 21211 raft_consensus.cc:515] T dd377fab8c544bf69121dc9e9a060b29 P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:25.083268 21173 leader_election.cc:290] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 107f274cdda543038832f06a56ab9130 (127.18.100.125:39711), 3749baa287ed4f249329a74071781edd (127.18.100.124:34629)
I20260130 02:58:25.084904 20810 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "658299197a424f53a37f14fadf089ae4" candidate_term: 4 candidate_status { last_received { term: 3 index: 372 } } ignore_live_leader: false dest_uuid: "3749baa287ed4f249329a74071781edd" is_pre_election: true
I20260130 02:58:25.085403 20810 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 3749baa287ed4f249329a74071781edd [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 658299197a424f53a37f14fadf089ae4 in term 3.
I20260130 02:58:25.085848 18960 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "658299197a424f53a37f14fadf089ae4" candidate_term: 4 candidate_status { last_received { term: 3 index: 372 } } ignore_live_leader: false dest_uuid: "107f274cdda543038832f06a56ab9130" is_pre_election: true
I20260130 02:58:25.086423 18860 leader_election.cc:304] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3749baa287ed4f249329a74071781edd, 658299197a424f53a37f14fadf089ae4; no voters:
I20260130 02:58:25.087311 21173 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260130 02:58:25.087690 21173 raft_consensus.cc:493] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 107f274cdda543038832f06a56ab9130)
I20260130 02:58:25.088001 21173 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4 [term 3 FOLLOWER]: Advancing to term 4
I20260130 02:58:25.096457 18875 catalog_manager.cc:5654] T f7607df5efd543438fb2dff3667cc8c8 P 371a36d219164974abdf848841528152 reported cstate change: term changed from 0 to 1, leader changed from <none> to 371a36d219164974abdf848841528152 (127.18.100.65). New cstate: current_term: 1 leader_uuid: "371a36d219164974abdf848841528152" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } health_report { overall_health: HEALTHY } } }
I20260130 02:58:25.099038 19840 consensus_queue.cc:260] T 532e867fd5944d468d7672d2a5c92f3c P 78c2c46c7520492db35d51c244fb4f74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:25.101835 19840 ts_tablet_manager.cc:1434] T 532e867fd5944d468d7672d2a5c92f3c P 78c2c46c7520492db35d51c244fb4f74: Time spent starting tablet: real 0.036s user 0.005s sys 0.000s
I20260130 02:58:25.106426 19225 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "39bcc41f10ae4241b370529364454586" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152"
I20260130 02:58:25.107627 21141 raft_consensus.cc:493] T 6f5a90b75a0f4851b5db672c41253f7b P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:25.108129 21141 raft_consensus.cc:515] T 6f5a90b75a0f4851b5db672c41253f7b P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:25.112421 21141 leader_election.cc:290] T 6f5a90b75a0f4851b5db672c41253f7b P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d870164263bb44b6b780a557daea3656 (127.18.100.67:39571), 371a36d219164974abdf848841528152 (127.18.100.65:42021)
I20260130 02:58:25.124686 19840 tablet_bootstrap.cc:492] T 155ff0c35fc44cb292a28407e6ed937d P 78c2c46c7520492db35d51c244fb4f74: Bootstrap starting.
I20260130 02:58:25.125480 19372 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "306452e208ca4e719609f67345114a13" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74"
I20260130 02:58:25.125628 19365 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2b8034a92b2043e5b57cf02f1ee6296d" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74"
I20260130 02:58:25.126075 19365 raft_consensus.cc:3060] T 2b8034a92b2043e5b57cf02f1ee6296d P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:25.131033 19378 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cfd12a72726846e587cff9d6a4d61bd6" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74"
I20260130 02:58:25.131559 19378 raft_consensus.cc:3060] T cfd12a72726846e587cff9d6a4d61bd6 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:25.114187 21211 leader_election.cc:290] T dd377fab8c544bf69121dc9e9a060b29 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 election: Requested vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767)
I20260130 02:58:25.130507 19379 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "39bcc41f10ae4241b370529364454586" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74"
I20260130 02:58:25.135136 18875 catalog_manager.cc:5746] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021) does not have the latest schema for tablet 274530359b174ff79f046c2d4a6afa76 (table default.Table_c4f4904b6fac4b9880979cf0314e71c1 [id=c456c6418c93499a9d08f652720854ca]). Expected version 1 got 0
I20260130 02:58:25.135205 19379 raft_consensus.cc:3060] T 39bcc41f10ae4241b370529364454586 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:25.135257 21121 raft_consensus.cc:493] T 66a7487a7ab343a4abca5312afe78862 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:25.135720 18875 catalog_manager.cc:5746] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021) does not have the latest schema for tablet 49efcfced0b74c48962393149ac2a500 (table default.Table_2e81f0a0cbc449a397141b717db17025 [id=a9c919f57c0c437cb35a6921103e45a6]). Expected version 1 got 0
I20260130 02:58:25.135711 21121 raft_consensus.cc:515] T 66a7487a7ab343a4abca5312afe78862 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:25.136268 18875 catalog_manager.cc:5654] T 320dbe98a32a43b9899bbeed80b20f55 P 371a36d219164974abdf848841528152 reported cstate change: term changed from 0 to 1, leader changed from <none> to 371a36d219164974abdf848841528152 (127.18.100.65). New cstate: current_term: 1 leader_uuid: "371a36d219164974abdf848841528152" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } health_report { overall_health: HEALTHY } } }
I20260130 02:58:25.137471 21121 leader_election.cc:290] T 66a7487a7ab343a4abca5312afe78862 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d870164263bb44b6b780a557daea3656 (127.18.100.67:39571), 371a36d219164974abdf848841528152 (127.18.100.65:42021)
I20260130 02:58:25.137693 18875 catalog_manager.cc:5746] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021) does not have the latest schema for tablet 657e9b0d8c05413f904129dce745710b (table default.Table_526677c8ef4144bd9de7a54a5e78e2be [id=b16908b75587459597a978c71756451c]). Expected version 1 got 0
W20260130 02:58:25.139510 19462 leader_election.cc:343] T e2aaaf00f53d418b87d5c7af286e4d84 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767): Not found: Tablet not found: e2aaaf00f53d418b87d5c7af286e4d84
I20260130 02:58:25.139876 19462 leader_election.cc:304] T e2aaaf00f53d418b87d5c7af286e4d84 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d870164263bb44b6b780a557daea3656; no voters: 371a36d219164974abdf848841528152, 78c2c46c7520492db35d51c244fb4f74
I20260130 02:58:25.140199 19366 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00ddbfe6218c44d788fbf65245a31dc6" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74"
I20260130 02:58:25.140723 19366 raft_consensus.cc:3060] T 00ddbfe6218c44d788fbf65245a31dc6 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:25.140996 20998 raft_consensus.cc:493] T 9896ff9364744857968e87e80e776dd2 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:25.141384 20998 raft_consensus.cc:515] T 9896ff9364744857968e87e80e776dd2 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:25.143213 20998 leader_election.cc:290] T 9896ff9364744857968e87e80e776dd2 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767)
I20260130 02:58:25.144068 20998 raft_consensus.cc:2749] T e2aaaf00f53d418b87d5c7af286e4d84 P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260130 02:58:25.145467 19462 leader_election.cc:304] T 1fc8f09d6329407c8078542b4123ef4d P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d870164263bb44b6b780a557daea3656; no voters: 371a36d219164974abdf848841528152, 78c2c46c7520492db35d51c244fb4f74
I20260130 02:58:25.146090 20998 raft_consensus.cc:2749] T 1fc8f09d6329407c8078542b4123ef4d P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260130 02:58:25.147329 19365 raft_consensus.cc:2468] T 2b8034a92b2043e5b57cf02f1ee6296d P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d870164263bb44b6b780a557daea3656 in term 1.
I20260130 02:58:25.115823 19523 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6f5a90b75a0f4851b5db672c41253f7b" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
I20260130 02:58:25.162885 19523 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "66a7487a7ab343a4abca5312afe78862" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
W20260130 02:58:25.128780 19463 leader_election.cc:343] T cfd12a72726846e587cff9d6a4d61bd6 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Illegal state: must be running to vote when last-logged opid is not known
W20260130 02:58:25.165365 19463 leader_election.cc:343] T 00ddbfe6218c44d788fbf65245a31dc6 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Illegal state: must be running to vote when last-logged opid is not known
W20260130 02:58:25.166083 19463 leader_election.cc:343] T 2b8034a92b2043e5b57cf02f1ee6296d P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:25.133872 19625 tablet_bootstrap.cc:654] T 1129f72158ec4bae89da3c94cdb70839 P d870164263bb44b6b780a557daea3656: Neither blocks nor log segments found. Creating new log.
W20260130 02:58:25.168497 19313 leader_election.cc:343] T 6f5a90b75a0f4851b5db672c41253f7b P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:25.169797 19462 leader_election.cc:304] T 94355fc1338f479e8798070aff04fcba P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d870164263bb44b6b780a557daea3656; no voters: 371a36d219164974abdf848841528152, 78c2c46c7520492db35d51c244fb4f74
W20260130 02:58:25.170754 19159 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.100.66:40767, user_credentials={real_user=slave}} blocked reactor thread for 113000us
I20260130 02:58:25.171023 18875 catalog_manager.cc:5746] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021) does not have the latest schema for tablet f2d01bfdc9c4494386bd02bed655fe63 (table default.Table_aaf139c1a55d4a1783298ca12ae0d5c0 [id=d128767e303145a885eec3c5f714a27f]). Expected version 4 got 3
I20260130 02:58:25.172093 18875 catalog_manager.cc:5746] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021) does not have the latest schema for tablet 25fd48e693424537b91ba7c3fcc96878 (table default.Table_ebc8b8379c274c18b0a06a919dbe0ee0 [id=28156e0843674fce88f5914a3e4195ae]). Expected version 1 got 0
I20260130 02:58:25.178653 21094 raft_consensus.cc:2749] T 94355fc1338f479e8798070aff04fcba P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260130 02:58:25.183802 19224 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "66a7487a7ab343a4abca5312afe78862" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152" is_pre_election: true
I20260130 02:58:25.184288 19218 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6f5a90b75a0f4851b5db672c41253f7b" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152" is_pre_election: true
W20260130 02:58:25.187146 19463 leader_election.cc:343] T 39bcc41f10ae4241b370529364454586 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:25.188927 19807 tablet_bootstrap.cc:492] T cd30d7adcc4545848e3501a205a10293 P 371a36d219164974abdf848841528152: No bootstrap required, opened a new log
I20260130 02:58:25.189297 19807 ts_tablet_manager.cc:1403] T cd30d7adcc4545848e3501a205a10293 P 371a36d219164974abdf848841528152: Time spent bootstrapping tablet: real 0.388s user 0.013s sys 0.000s
W20260130 02:58:25.189792 19462 consensus_peers.cc:597] T 2c79a3aecab844698e70834999e1caac P d870164263bb44b6b780a557daea3656 -> Peer 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767): Couldn't send request to peer 78c2c46c7520492db35d51c244fb4f74. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2c79a3aecab844698e70834999e1caac. This is attempt 1: this message will repeat every 5th retry.
I20260130 02:58:25.169226 19378 raft_consensus.cc:2468] T cfd12a72726846e587cff9d6a4d61bd6 P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d870164263bb44b6b780a557daea3656 in term 1.
I20260130 02:58:25.169898 20899 ts_tablet_manager.cc:1929] T ba91a508b1cb4430a4d8ca26946aecc8 P 78c2c46c7520492db35d51c244fb4f74: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260130 02:58:25.194736 20899 log.cc:1199] T ba91a508b1cb4430a4d8ca26946aecc8 P 78c2c46c7520492db35d51c244fb4f74: Deleting WAL directory at /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-1/wal/wals/ba91a508b1cb4430a4d8ca26946aecc8
I20260130 02:58:25.195600 20899 ts_tablet_manager.cc:1950] T ba91a508b1cb4430a4d8ca26946aecc8 P 78c2c46c7520492db35d51c244fb4f74: Deleting consensus metadata
W20260130 02:58:25.186190 19314 leader_election.cc:343] T 66a7487a7ab343a4abca5312afe78862 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:25.195956 19625 tablet_bootstrap.cc:492] T 1129f72158ec4bae89da3c94cdb70839 P d870164263bb44b6b780a557daea3656: No bootstrap required, opened a new log
I20260130 02:58:25.196346 19625 ts_tablet_manager.cc:1403] T 1129f72158ec4bae89da3c94cdb70839 P d870164263bb44b6b780a557daea3656: Time spent bootstrapping tablet: real 0.179s user 0.007s sys 0.001s
W20260130 02:58:25.196300 19314 leader_election.cc:343] T 6f5a90b75a0f4851b5db672c41253f7b P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:25.196637 19314 leader_election.cc:304] T 6f5a90b75a0f4851b5db672c41253f7b P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78c2c46c7520492db35d51c244fb4f74; no voters: 371a36d219164974abdf848841528152, d870164263bb44b6b780a557daea3656
I20260130 02:58:25.197656 21141 raft_consensus.cc:2749] T 6f5a90b75a0f4851b5db672c41253f7b P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260130 02:58:25.191601 19366 raft_consensus.cc:2468] T 00ddbfe6218c44d788fbf65245a31dc6 P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d870164263bb44b6b780a557daea3656 in term 1.
I20260130 02:58:25.174113 19379 raft_consensus.cc:2468] T 39bcc41f10ae4241b370529364454586 P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d870164263bb44b6b780a557daea3656 in term 1.
I20260130 02:58:25.198412 19462 leader_election.cc:304] T 306452e208ca4e719609f67345114a13 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d870164263bb44b6b780a557daea3656; no voters: 371a36d219164974abdf848841528152, 78c2c46c7520492db35d51c244fb4f74
I20260130 02:58:25.180109 19840 tablet_bootstrap.cc:654] T 155ff0c35fc44cb292a28407e6ed937d P 78c2c46c7520492db35d51c244fb4f74: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:25.198508 19625 raft_consensus.cc:359] T 1129f72158ec4bae89da3c94cdb70839 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:25.200918 19625 raft_consensus.cc:385] T 1129f72158ec4bae89da3c94cdb70839 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:25.201184 19625 raft_consensus.cc:740] T 1129f72158ec4bae89da3c94cdb70839 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d870164263bb44b6b780a557daea3656, State: Initialized, Role: FOLLOWER
I20260130 02:58:25.201714 19625 consensus_queue.cc:260] T 1129f72158ec4bae89da3c94cdb70839 P d870164263bb44b6b780a557daea3656 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:25.203047 20998 raft_consensus.cc:2749] T 306452e208ca4e719609f67345114a13 P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260130 02:58:25.194128 19807 raft_consensus.cc:359] T cd30d7adcc4545848e3501a205a10293 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
W20260130 02:58:25.202241 19313 leader_election.cc:343] T 66a7487a7ab343a4abca5312afe78862 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:25.173338 18875 catalog_manager.cc:5654] T 13f0ace4d9de43abab08eca32cdb30c1 P 371a36d219164974abdf848841528152 reported cstate change: term changed from 0 to 1, leader changed from <none> to 371a36d219164974abdf848841528152 (127.18.100.65). New cstate: current_term: 1 leader_uuid: "371a36d219164974abdf848841528152" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } health_report { overall_health: HEALTHY } } }
I20260130 02:58:25.205034 18875 catalog_manager.cc:5654] T 838e49e5e8874db393bdc15a1c92f87e P 371a36d219164974abdf848841528152 reported cstate change: term changed from 0 to 1, leader changed from <none> to 371a36d219164974abdf848841528152 (127.18.100.65). New cstate: current_term: 1 leader_uuid: "371a36d219164974abdf848841528152" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } health_report { overall_health: HEALTHY } } }
I20260130 02:58:25.206164 18875 catalog_manager.cc:5746] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021) does not have the latest schema for tablet 838e49e5e8874db393bdc15a1c92f87e (table default.Table_da3d581b15a94840898777f74611994e [id=b7e241ced53e4160ac6fa1f65ef3674e]). Expected version 1 got 0
I20260130 02:58:25.206235 19379 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "306452e208ca4e719609f67345114a13" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74" is_pre_election: true
I20260130 02:58:25.209853 19807 raft_consensus.cc:385] T cd30d7adcc4545848e3501a205a10293 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:25.211515 19313 leader_election.cc:304] T 66a7487a7ab343a4abca5312afe78862 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78c2c46c7520492db35d51c244fb4f74; no voters: 371a36d219164974abdf848841528152, d870164263bb44b6b780a557daea3656
I20260130 02:58:25.212563 19807 raft_consensus.cc:740] T cd30d7adcc4545848e3501a205a10293 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 371a36d219164974abdf848841528152, State: Initialized, Role: FOLLOWER
I20260130 02:58:25.217427 19462 leader_election.cc:304] T 2b8034a92b2043e5b57cf02f1ee6296d P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 78c2c46c7520492db35d51c244fb4f74, d870164263bb44b6b780a557daea3656; no voters: 371a36d219164974abdf848841528152
I20260130 02:58:25.220136 19523 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "306452e208ca4e719609f67345114a13" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
I20260130 02:58:25.220919 19523 raft_consensus.cc:2393] T 306452e208ca4e719609f67345114a13 P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 371a36d219164974abdf848841528152 in current term 1: Already voted for candidate d870164263bb44b6b780a557daea3656 in this term.
I20260130 02:58:25.214879 19807 consensus_queue.cc:260] T cd30d7adcc4545848e3501a205a10293 P 371a36d219164974abdf848841528152 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:25.222779 21234 tablet.cc:1722] T 345eaaa5625c48d099280b7fbb2e514b P d870164263bb44b6b780a557daea3656: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260130 02:58:25.213562 20899 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60110 (request call id 342) took 1739 ms. Trace:
I20260130 02:58:25.225690 20899 rpcz_store.cc:276] 0130 02:58:23.473913 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:58:23.474015 (+ 102us) service_pool.cc:225] Handling call
0130 02:58:24.715975 (+1241960us) ts_tablet_manager.cc:1151] Deleting tablet ba91a508b1cb4430a4d8ca26946aecc8
0130 02:58:24.716050 (+ 75us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0130 02:58:24.871129 (+155079us) tablet_metadata.cc:641] Metadata flushed
0130 02:58:25.169683 (+298554us) tablet_metadata.cc:641] Metadata flushed
0130 02:58:25.213507 (+ 43824us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1240910}
I20260130 02:58:25.226769 20998 raft_consensus.cc:2804] T 2b8034a92b2043e5b57cf02f1ee6296d P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Leader election won for term 1
I20260130 02:58:25.227208 20998 raft_consensus.cc:697] T 2b8034a92b2043e5b57cf02f1ee6296d P d870164263bb44b6b780a557daea3656 [term 1 LEADER]: Becoming Leader. State: Replica: d870164263bb44b6b780a557daea3656, State: Running, Role: LEADER
I20260130 02:58:25.227941 20998 consensus_queue.cc:237] T 2b8034a92b2043e5b57cf02f1ee6296d P d870164263bb44b6b780a557daea3656 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:25.229231 21141 raft_consensus.cc:2749] T 66a7487a7ab343a4abca5312afe78862 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260130 02:58:25.231964 19807 ts_tablet_manager.cc:1434] T cd30d7adcc4545848e3501a205a10293 P 371a36d219164974abdf848841528152: Time spent starting tablet: real 0.042s user 0.003s sys 0.003s
I20260130 02:58:25.232599 18860 catalog_manager.cc:4985] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767): tablet ba91a508b1cb4430a4d8ca26946aecc8 (table default.Table_aa496bf2019748f29823a9a253b1f3ca [id=780188e2335f43fbb11de8212eb38d56]) successfully deleted
I20260130 02:58:25.238745 19225 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dd377fab8c544bf69121dc9e9a060b29" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152"
I20260130 02:58:25.239984 19225 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9896ff9364744857968e87e80e776dd2" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152" is_pre_election: true
I20260130 02:58:25.252032 19625 ts_tablet_manager.cc:1434] T 1129f72158ec4bae89da3c94cdb70839 P d870164263bb44b6b780a557daea3656: Time spent starting tablet: real 0.055s user 0.006s sys 0.001s
I20260130 02:58:25.252976 19625 tablet_bootstrap.cc:492] T 6d1f121f49cd47c2b8247daaf6874e01 P d870164263bb44b6b780a557daea3656: Bootstrap starting.
I20260130 02:58:25.255486 19807 tablet_bootstrap.cc:492] T cfd12a72726846e587cff9d6a4d61bd6 P 371a36d219164974abdf848841528152: Bootstrap starting.
I20260130 02:58:25.204128 21173 raft_consensus.cc:515] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "658299197a424f53a37f14fadf089ae4" member_type: VOTER last_known_addr { host: "127.18.100.126" port: 38191 } } peers { permanent_uuid: "107f274cdda543038832f06a56ab9130" member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39711 } } peers { permanent_uuid: "3749baa287ed4f249329a74071781edd" member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34629 } }
I20260130 02:58:25.259402 19523 raft_consensus.cc:1275] T ca06f07bb98f45a7bf63aebc28348e08 P d870164263bb44b6b780a557daea3656 [term 2 FOLLOWER]: Refusing update from remote peer 371a36d219164974abdf848841528152: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 1. (term mismatch)
I20260130 02:58:25.259891 19523 pending_rounds.cc:85] T ca06f07bb98f45a7bf63aebc28348e08 P d870164263bb44b6b780a557daea3656: Aborting all ops after (but not including) 0
I20260130 02:58:25.260281 19523 pending_rounds.cc:107] T ca06f07bb98f45a7bf63aebc28348e08 P d870164263bb44b6b780a557daea3656: Aborting uncommitted NO_OP operation due to leader change: 1.1
I20260130 02:58:25.260640 19523 raft_consensus.cc:2889] T ca06f07bb98f45a7bf63aebc28348e08 P d870164263bb44b6b780a557daea3656 [term 2 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260130 02:58:25.262120 19840 tablet_bootstrap.cc:492] T 155ff0c35fc44cb292a28407e6ed937d P 78c2c46c7520492db35d51c244fb4f74: No bootstrap required, opened a new log
I20260130 02:58:25.262667 19840 ts_tablet_manager.cc:1403] T 155ff0c35fc44cb292a28407e6ed937d P 78c2c46c7520492db35d51c244fb4f74: Time spent bootstrapping tablet: real 0.138s user 0.010s sys 0.000s
I20260130 02:58:25.264879 19840 raft_consensus.cc:359] T 155ff0c35fc44cb292a28407e6ed937d P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:25.265499 19840 raft_consensus.cc:385] T 155ff0c35fc44cb292a28407e6ed937d P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:25.265751 19840 raft_consensus.cc:740] T 155ff0c35fc44cb292a28407e6ed937d P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78c2c46c7520492db35d51c244fb4f74, State: Initialized, Role: FOLLOWER
I20260130 02:58:25.271497 19364 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dd377fab8c544bf69121dc9e9a060b29" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74"
I20260130 02:58:25.272145 19364 raft_consensus.cc:3060] T dd377fab8c544bf69121dc9e9a060b29 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:25.274873 18960 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "658299197a424f53a37f14fadf089ae4" candidate_term: 4 candidate_status { last_received { term: 3 index: 372 } } ignore_live_leader: false dest_uuid: "107f274cdda543038832f06a56ab9130"
I20260130 02:58:25.279381 19364 raft_consensus.cc:2468] T dd377fab8c544bf69121dc9e9a060b29 P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d870164263bb44b6b780a557daea3656 in term 1.
I20260130 02:58:25.251669 18875 catalog_manager.cc:5654] T 12c3f3ec523040e7b7cd356e3da57e0d P 371a36d219164974abdf848841528152 reported cstate change: term changed from 0 to 1, leader changed from <none> to 371a36d219164974abdf848841528152 (127.18.100.65). New cstate: current_term: 1 leader_uuid: "371a36d219164974abdf848841528152" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } health_report { overall_health: HEALTHY } } }
I20260130 02:58:25.254348 21237 tablet.cc:1722] T 345eaaa5625c48d099280b7fbb2e514b P 371a36d219164974abdf848841528152: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260130 02:58:25.284106 20810 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "658299197a424f53a37f14fadf089ae4" candidate_term: 4 candidate_status { last_received { term: 3 index: 372 } } ignore_live_leader: false dest_uuid: "3749baa287ed4f249329a74071781edd"
I20260130 02:58:25.284629 20810 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 3749baa287ed4f249329a74071781edd [term 3 FOLLOWER]: Advancing to term 4
I20260130 02:58:25.234503 19462 leader_election.cc:304] T cfd12a72726846e587cff9d6a4d61bd6 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 78c2c46c7520492db35d51c244fb4f74, d870164263bb44b6b780a557daea3656; no voters: 371a36d219164974abdf848841528152
W20260130 02:58:25.287390 19462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.100.66:40767, user_credentials={real_user=slave}} blocked reactor thread for 52981.6us
I20260130 02:58:25.288007 19462 leader_election.cc:304] T 00ddbfe6218c44d788fbf65245a31dc6 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 78c2c46c7520492db35d51c244fb4f74, d870164263bb44b6b780a557daea3656; no voters: 371a36d219164974abdf848841528152
I20260130 02:58:25.288796 20998 raft_consensus.cc:2804] T cfd12a72726846e587cff9d6a4d61bd6 P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Leader election won for term 1
I20260130 02:58:25.289312 21167 raft_consensus.cc:2804] T 00ddbfe6218c44d788fbf65245a31dc6 P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Leader election won for term 1
I20260130 02:58:25.289700 21167 raft_consensus.cc:697] T 00ddbfe6218c44d788fbf65245a31dc6 P d870164263bb44b6b780a557daea3656 [term 1 LEADER]: Becoming Leader. State: Replica: d870164263bb44b6b780a557daea3656, State: Running, Role: LEADER
I20260130 02:58:25.290377 21167 consensus_queue.cc:237] T 00ddbfe6218c44d788fbf65245a31dc6 P d870164263bb44b6b780a557daea3656 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:25.292495 20810 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 3749baa287ed4f249329a74071781edd [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 658299197a424f53a37f14fadf089ae4 in term 4.
I20260130 02:58:25.292502 19370 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9896ff9364744857968e87e80e776dd2" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74" is_pre_election: true
I20260130 02:58:25.293052 19370 raft_consensus.cc:2468] T 9896ff9364744857968e87e80e776dd2 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d870164263bb44b6b780a557daea3656 in term 0.
I20260130 02:58:25.293547 18860 leader_election.cc:304] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3749baa287ed4f249329a74071781edd, 658299197a424f53a37f14fadf089ae4; no voters: 107f274cdda543038832f06a56ab9130
I20260130 02:58:25.293677 20998 raft_consensus.cc:697] T cfd12a72726846e587cff9d6a4d61bd6 P d870164263bb44b6b780a557daea3656 [term 1 LEADER]: Becoming Leader. State: Replica: d870164263bb44b6b780a557daea3656, State: Running, Role: LEADER
I20260130 02:58:25.294510 20998 consensus_queue.cc:237] T cfd12a72726846e587cff9d6a4d61bd6 P d870164263bb44b6b780a557daea3656 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:25.303669 21173 leader_election.cc:290] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4 [CANDIDATE]: Term 4 election: Requested vote from peers 107f274cdda543038832f06a56ab9130 (127.18.100.125:39711), 3749baa287ed4f249329a74071781edd (127.18.100.124:34629)
I20260130 02:58:25.307838 21094 consensus_queue.cc:1048] T 0855d88806914a36b4bf6b49036f0ad0 P d870164263bb44b6b780a557daea3656 [LEADER]: Connected to new peer: Peer: permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260130 02:58:25.310225 19463 leader_election.cc:343] T dd377fab8c544bf69121dc9e9a060b29 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Illegal state: must be running to vote when last-logged opid is not known
W20260130 02:58:25.311378 19463 leader_election.cc:343] T 9896ff9364744857968e87e80e776dd2 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:25.311506 19462 leader_election.cc:304] T 39bcc41f10ae4241b370529364454586 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 78c2c46c7520492db35d51c244fb4f74, d870164263bb44b6b780a557daea3656; no voters: 371a36d219164974abdf848841528152
I20260130 02:58:25.313540 21173 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4 [term 4 FOLLOWER]: Leader election won for term 4
I20260130 02:58:25.313669 21094 raft_consensus.cc:2804] T 39bcc41f10ae4241b370529364454586 P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Leader election won for term 1
I20260130 02:58:25.314072 21094 raft_consensus.cc:697] T 39bcc41f10ae4241b370529364454586 P d870164263bb44b6b780a557daea3656 [term 1 LEADER]: Becoming Leader. State: Replica: d870164263bb44b6b780a557daea3656, State: Running, Role: LEADER
I20260130 02:58:25.314790 21094 consensus_queue.cc:237] T 39bcc41f10ae4241b370529364454586 P d870164263bb44b6b780a557daea3656 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:25.306676 19840 consensus_queue.cc:260] T 155ff0c35fc44cb292a28407e6ed937d P 78c2c46c7520492db35d51c244fb4f74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:25.318804 21173 raft_consensus.cc:697] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4 [term 4 LEADER]: Becoming Leader. State: Replica: 658299197a424f53a37f14fadf089ae4, State: Running, Role: LEADER
I20260130 02:58:25.318825 21200 raft_consensus.cc:493] T cd30d7adcc4545848e3501a205a10293 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:25.319238 21200 raft_consensus.cc:515] T cd30d7adcc4545848e3501a205a10293 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:25.319651 21173 consensus_queue.cc:237] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 372, Committed index: 372, Last appended: 3.372, Last appended by leader: 372, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "658299197a424f53a37f14fadf089ae4" member_type: VOTER last_known_addr { host: "127.18.100.126" port: 38191 } } peers { permanent_uuid: "107f274cdda543038832f06a56ab9130" member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39711 } } peers { permanent_uuid: "3749baa287ed4f249329a74071781edd" member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34629 } }
I20260130 02:58:25.320966 21200 leader_election.cc:290] T cd30d7adcc4545848e3501a205a10293 P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d870164263bb44b6b780a557daea3656 (127.18.100.67:39571), 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767)
I20260130 02:58:25.322556 21121 raft_consensus.cc:493] T e691c238199b4914bcc3fd248f31420c P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:25.323004 21121 raft_consensus.cc:515] T e691c238199b4914bcc3fd248f31420c P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:25.324751 21121 leader_election.cc:290] T e691c238199b4914bcc3fd248f31420c P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d870164263bb44b6b780a557daea3656 (127.18.100.67:39571), 371a36d219164974abdf848841528152 (127.18.100.65:42021)
I20260130 02:58:25.324653 19462 leader_election.cc:304] T dd377fab8c544bf69121dc9e9a060b29 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 78c2c46c7520492db35d51c244fb4f74, d870164263bb44b6b780a557daea3656; no voters: 371a36d219164974abdf848841528152
I20260130 02:58:25.326323 19524 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e691c238199b4914bcc3fd248f31420c" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
I20260130 02:58:25.327587 19840 ts_tablet_manager.cc:1434] T 155ff0c35fc44cb292a28407e6ed937d P 78c2c46c7520492db35d51c244fb4f74: Time spent starting tablet: real 0.065s user 0.005s sys 0.000s
I20260130 02:58:25.330333 19225 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e691c238199b4914bcc3fd248f31420c" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152" is_pre_election: true
W20260130 02:58:25.331018 19313 leader_election.cc:343] T e691c238199b4914bcc3fd248f31420c P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): Illegal state: must be running to vote when last-logged opid is not known
W20260130 02:58:25.331534 19314 leader_election.cc:343] T e691c238199b4914bcc3fd248f31420c P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:25.331861 19314 leader_election.cc:304] T e691c238199b4914bcc3fd248f31420c P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78c2c46c7520492db35d51c244fb4f74; no voters: 371a36d219164974abdf848841528152, d870164263bb44b6b780a557daea3656
I20260130 02:58:25.332527 21141 raft_consensus.cc:2749] T e691c238199b4914bcc3fd248f31420c P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260130 02:58:25.336020 21200 consensus_queue.cc:1048] T ca06f07bb98f45a7bf63aebc28348e08 P 371a36d219164974abdf848841528152 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260130 02:58:25.343158 21211 raft_consensus.cc:2804] T dd377fab8c544bf69121dc9e9a060b29 P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Leader election won for term 1
I20260130 02:58:25.343705 21211 raft_consensus.cc:697] T dd377fab8c544bf69121dc9e9a060b29 P d870164263bb44b6b780a557daea3656 [term 1 LEADER]: Becoming Leader. State: Replica: d870164263bb44b6b780a557daea3656, State: Running, Role: LEADER
I20260130 02:58:25.344465 21211 consensus_queue.cc:237] T dd377fab8c544bf69121dc9e9a060b29 P d870164263bb44b6b780a557daea3656 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:25.349905 21218 raft_consensus.cc:493] T 0d14fb95ace9447f810ccbed4d5befdf P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:25.350340 21218 raft_consensus.cc:515] T 0d14fb95ace9447f810ccbed4d5befdf P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:25.352310 19807 tablet_bootstrap.cc:654] T cfd12a72726846e587cff9d6a4d61bd6 P 371a36d219164974abdf848841528152: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:25.352363 21218 leader_election.cc:290] T 0d14fb95ace9447f810ccbed4d5befdf P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767)
I20260130 02:58:25.364841 19225 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0d14fb95ace9447f810ccbed4d5befdf" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152" is_pre_election: true
I20260130 02:58:25.366456 19462 leader_election.cc:304] T 9896ff9364744857968e87e80e776dd2 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 78c2c46c7520492db35d51c244fb4f74, d870164263bb44b6b780a557daea3656; no voters: 371a36d219164974abdf848841528152
I20260130 02:58:25.367256 19840 tablet_bootstrap.cc:492] T 30cc52136bb3422895d95c0fed0c6798 P 78c2c46c7520492db35d51c244fb4f74: Bootstrap starting.
I20260130 02:58:25.367406 21094 raft_consensus.cc:2804] T 9896ff9364744857968e87e80e776dd2 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260130 02:58:25.367864 21094 raft_consensus.cc:493] T 9896ff9364744857968e87e80e776dd2 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260130 02:58:25.368234 21094 raft_consensus.cc:3060] T 9896ff9364744857968e87e80e776dd2 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:25.379855 21162 sys_catalog.cc:455] T 00000000000000000000000000000000 P 107f274cdda543038832f06a56ab9130 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "107f274cdda543038832f06a56ab9130" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "658299197a424f53a37f14fadf089ae4" member_type: VOTER last_known_addr { host: "127.18.100.126" port: 38191 } } peers { permanent_uuid: "107f274cdda543038832f06a56ab9130" member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39711 } } peers { permanent_uuid: "3749baa287ed4f249329a74071781edd" member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34629 } } }
I20260130 02:58:25.381546 20810 raft_consensus.cc:1240] T 00000000000000000000000000000000 P 3749baa287ed4f249329a74071781edd [term 4 FOLLOWER]: Rejecting Update request from peer 107f274cdda543038832f06a56ab9130 for earlier term 3. Current term is 4. Ops: []
I20260130 02:58:25.382738 21162 sys_catalog.cc:458] T 00000000000000000000000000000000 P 107f274cdda543038832f06a56ab9130 [sys.catalog]: This master's current role is: LEADER
I20260130 02:58:25.383594 21162 consensus_queue.cc:1059] T 00000000000000000000000000000000 P 107f274cdda543038832f06a56ab9130 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3749baa287ed4f249329a74071781edd" member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34629 }, Status: INVALID_TERM, Last received: 3.372, Next index: 373, Last known committed idx: 371, Time since last communication: 0.000s
I20260130 02:58:25.384480 21162 raft_consensus.cc:3055] T 00000000000000000000000000000000 P 107f274cdda543038832f06a56ab9130 [term 3 LEADER]: Stepping down as leader of term 3
I20260130 02:58:25.384824 21162 raft_consensus.cc:740] T 00000000000000000000000000000000 P 107f274cdda543038832f06a56ab9130 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 107f274cdda543038832f06a56ab9130, State: Running, Role: LEADER
I20260130 02:58:25.385890 21162 consensus_queue.cc:260] T 00000000000000000000000000000000 P 107f274cdda543038832f06a56ab9130 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 372, Committed index: 372, Last appended: 3.372, Last appended by leader: 372, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "658299197a424f53a37f14fadf089ae4" member_type: VOTER last_known_addr { host: "127.18.100.126" port: 38191 } } peers { permanent_uuid: "107f274cdda543038832f06a56ab9130" member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39711 } } peers { permanent_uuid: "3749baa287ed4f249329a74071781edd" member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34629 } }
I20260130 02:58:25.386914 21162 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 107f274cdda543038832f06a56ab9130 [term 3 FOLLOWER]: Advancing to term 4
I20260130 02:58:25.388172 21167 consensus_queue.cc:1048] T eb7a755ab62b4552bb8588264c5f7b75 P d870164263bb44b6b780a557daea3656 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260130 02:58:25.400279 19625 tablet_bootstrap.cc:654] T 6d1f121f49cd47c2b8247daaf6874e01 P d870164263bb44b6b780a557daea3656: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:25.407644 19378 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cd30d7adcc4545848e3501a205a10293" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74" is_pre_election: true
I20260130 02:58:25.412330 21164 catalog_manager.cc:1269] Loaded cluster ID: db1c0e7709154c10a8c9a0ea68ffe38b
I20260130 02:58:25.412272 19370 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0d14fb95ace9447f810ccbed4d5befdf" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74" is_pre_election: true
I20260130 02:58:25.413149 19370 raft_consensus.cc:2468] T 0d14fb95ace9447f810ccbed4d5befdf P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d870164263bb44b6b780a557daea3656 in term 0.
I20260130 02:58:25.413635 21164 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260130 02:58:25.433429 19515 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cd30d7adcc4545848e3501a205a10293" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
I20260130 02:58:25.434281 19515 raft_consensus.cc:2393] T cd30d7adcc4545848e3501a205a10293 P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 371a36d219164974abdf848841528152 in current term 1: Already voted for candidate 78c2c46c7520492db35d51c244fb4f74 in this term.
I20260130 02:58:25.442250 21240 sys_catalog.cc:455] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 658299197a424f53a37f14fadf089ae4. Latest consensus state: current_term: 4 leader_uuid: "658299197a424f53a37f14fadf089ae4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "658299197a424f53a37f14fadf089ae4" member_type: VOTER last_known_addr { host: "127.18.100.126" port: 38191 } } peers { permanent_uuid: "107f274cdda543038832f06a56ab9130" member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39711 } } peers { permanent_uuid: "3749baa287ed4f249329a74071781edd" member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34629 } } }
I20260130 02:58:25.443905 21240 sys_catalog.cc:458] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4 [sys.catalog]: This master's current role is: LEADER
I20260130 02:58:25.446789 18885 raft_consensus.cc:1240] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4 [term 4 LEADER]: Rejecting Update request from peer 107f274cdda543038832f06a56ab9130 for earlier term 3. Current term is 4. Ops: []
W20260130 02:58:25.460619 19159 outbound_call.cc:321] RPC callback for RPC call kudu.master.MasterService.TSHeartbeat -> {remote=127.18.100.125:39711, user_credentials={real_user=slave}} blocked reactor thread for 51116.2us
I20260130 02:58:25.461364 19159 leader_election.cc:304] T 306452e208ca4e719609f67345114a13 P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 371a36d219164974abdf848841528152; no voters: 78c2c46c7520492db35d51c244fb4f74, d870164263bb44b6b780a557daea3656
I20260130 02:58:25.462021 21095 raft_consensus.cc:2749] T 306452e208ca4e719609f67345114a13 P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260130 02:58:25.468042 19463 leader_election.cc:343] T 0d14fb95ace9447f810ccbed4d5befdf P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:25.469524 21248 tablet.cc:1722] T 345eaaa5625c48d099280b7fbb2e514b P 78c2c46c7520492db35d51c244fb4f74: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260130 02:58:25.471002 19840 tablet_bootstrap.cc:654] T 30cc52136bb3422895d95c0fed0c6798 P 78c2c46c7520492db35d51c244fb4f74: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:25.475289 21094 raft_consensus.cc:515] T 9896ff9364744857968e87e80e776dd2 P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:25.480343 19462 leader_election.cc:304] T 0d14fb95ace9447f810ccbed4d5befdf P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 78c2c46c7520492db35d51c244fb4f74, d870164263bb44b6b780a557daea3656; no voters: 371a36d219164974abdf848841528152
I20260130 02:58:25.480687 21141 raft_consensus.cc:493] T 155ff0c35fc44cb292a28407e6ed937d P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:25.481226 21141 raft_consensus.cc:515] T 155ff0c35fc44cb292a28407e6ed937d P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:25.481390 20998 consensus_queue.cc:1048] T 0855d88806914a36b4bf6b49036f0ad0 P d870164263bb44b6b780a557daea3656 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260130 02:58:25.482435 20998 raft_consensus.cc:2804] T 0d14fb95ace9447f810ccbed4d5befdf P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260130 02:58:25.482746 20998 raft_consensus.cc:493] T 0d14fb95ace9447f810ccbed4d5befdf P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260130 02:58:25.483017 20998 raft_consensus.cc:3060] T 0d14fb95ace9447f810ccbed4d5befdf P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:25.485785 21141 leader_election.cc:290] T 155ff0c35fc44cb292a28407e6ed937d P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d870164263bb44b6b780a557daea3656 (127.18.100.67:39571), 371a36d219164974abdf848841528152 (127.18.100.65:42021)
I20260130 02:58:25.487675 19519 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "155ff0c35fc44cb292a28407e6ed937d" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
W20260130 02:58:25.489977 19313 leader_election.cc:343] T 155ff0c35fc44cb292a28407e6ed937d P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:25.495707 19807 tablet_bootstrap.cc:492] T cfd12a72726846e587cff9d6a4d61bd6 P 371a36d219164974abdf848841528152: No bootstrap required, opened a new log
I20260130 02:58:25.496136 19807 ts_tablet_manager.cc:1403] T cfd12a72726846e587cff9d6a4d61bd6 P 371a36d219164974abdf848841528152: Time spent bootstrapping tablet: real 0.241s user 0.010s sys 0.000s
I20260130 02:58:25.504043 19625 tablet_bootstrap.cc:492] T 6d1f121f49cd47c2b8247daaf6874e01 P d870164263bb44b6b780a557daea3656: No bootstrap required, opened a new log
I20260130 02:58:25.504457 19625 ts_tablet_manager.cc:1403] T 6d1f121f49cd47c2b8247daaf6874e01 P d870164263bb44b6b780a557daea3656: Time spent bootstrapping tablet: real 0.252s user 0.005s sys 0.007s
I20260130 02:58:25.510908 19224 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "155ff0c35fc44cb292a28407e6ed937d" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152" is_pre_election: true
W20260130 02:58:25.516220 19314 leader_election.cc:343] T 155ff0c35fc44cb292a28407e6ed937d P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:25.516682 19314 leader_election.cc:304] T 155ff0c35fc44cb292a28407e6ed937d P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78c2c46c7520492db35d51c244fb4f74; no voters: 371a36d219164974abdf848841528152, d870164263bb44b6b780a557daea3656
I20260130 02:58:25.517753 21133 raft_consensus.cc:2749] T 155ff0c35fc44cb292a28407e6ed937d P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260130 02:58:25.521871 21121 consensus_queue.cc:1048] T 1fc8f09d6329407c8078542b4123ef4d P 78c2c46c7520492db35d51c244fb4f74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260130 02:58:25.536456 21094 leader_election.cc:290] T 9896ff9364744857968e87e80e776dd2 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 election: Requested vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767)
I20260130 02:58:25.541128 20998 raft_consensus.cc:515] T 0d14fb95ace9447f810ccbed4d5befdf P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:25.543396 19378 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9896ff9364744857968e87e80e776dd2" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74"
I20260130 02:58:25.544742 19378 raft_consensus.cc:3060] T 9896ff9364744857968e87e80e776dd2 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:25.549458 19625 raft_consensus.cc:359] T 6d1f121f49cd47c2b8247daaf6874e01 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:25.551353 19625 raft_consensus.cc:385] T 6d1f121f49cd47c2b8247daaf6874e01 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:25.551708 19625 raft_consensus.cc:740] T 6d1f121f49cd47c2b8247daaf6874e01 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d870164263bb44b6b780a557daea3656, State: Initialized, Role: FOLLOWER
I20260130 02:58:25.552459 19625 consensus_queue.cc:260] T 6d1f121f49cd47c2b8247daaf6874e01 P d870164263bb44b6b780a557daea3656 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:25.535471 19807 raft_consensus.cc:359] T cfd12a72726846e587cff9d6a4d61bd6 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:25.553853 19807 raft_consensus.cc:385] T cfd12a72726846e587cff9d6a4d61bd6 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:25.554113 19807 raft_consensus.cc:740] T cfd12a72726846e587cff9d6a4d61bd6 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 371a36d219164974abdf848841528152, State: Initialized, Role: FOLLOWER
I20260130 02:58:25.554672 19807 consensus_queue.cc:260] T cfd12a72726846e587cff9d6a4d61bd6 P 371a36d219164974abdf848841528152 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:25.556079 19218 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9896ff9364744857968e87e80e776dd2" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152"
I20260130 02:58:25.564471 19378 raft_consensus.cc:2468] T 9896ff9364744857968e87e80e776dd2 P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d870164263bb44b6b780a557daea3656 in term 1.
I20260130 02:58:25.567894 19462 leader_election.cc:304] T 9896ff9364744857968e87e80e776dd2 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78c2c46c7520492db35d51c244fb4f74, d870164263bb44b6b780a557daea3656; no voters:
I20260130 02:58:25.568538 21211 raft_consensus.cc:493] T 1129f72158ec4bae89da3c94cdb70839 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:25.568962 21211 raft_consensus.cc:515] T 1129f72158ec4bae89da3c94cdb70839 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:25.571363 21211 leader_election.cc:290] T 1129f72158ec4bae89da3c94cdb70839 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767)
I20260130 02:58:25.572978 21211 raft_consensus.cc:2804] T 9896ff9364744857968e87e80e776dd2 P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Leader election won for term 1
I20260130 02:58:25.573410 21211 raft_consensus.cc:697] T 9896ff9364744857968e87e80e776dd2 P d870164263bb44b6b780a557daea3656 [term 1 LEADER]: Becoming Leader. State: Replica: d870164263bb44b6b780a557daea3656, State: Running, Role: LEADER
I20260130 02:58:25.574304 21211 consensus_queue.cc:237] T 9896ff9364744857968e87e80e776dd2 P d870164263bb44b6b780a557daea3656 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:25.577772 19625 ts_tablet_manager.cc:1434] T 6d1f121f49cd47c2b8247daaf6874e01 P d870164263bb44b6b780a557daea3656: Time spent starting tablet: real 0.073s user 0.004s sys 0.002s
I20260130 02:58:25.578698 19625 tablet_bootstrap.cc:492] T 532e867fd5944d468d7672d2a5c92f3c P d870164263bb44b6b780a557daea3656: Bootstrap starting.
I20260130 02:58:25.579406 21237 tablet.cc:1722] T b1e8b78eff0d4a0b92059adb2de4a3ce P 371a36d219164974abdf848841528152: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260130 02:58:25.590231 20998 leader_election.cc:290] T 0d14fb95ace9447f810ccbed4d5befdf P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 election: Requested vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767)
I20260130 02:58:25.593004 19376 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0d14fb95ace9447f810ccbed4d5befdf" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74"
I20260130 02:58:25.593590 19376 raft_consensus.cc:3060] T 0d14fb95ace9447f810ccbed4d5befdf P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:25.603960 19625 tablet_bootstrap.cc:654] T 532e867fd5944d468d7672d2a5c92f3c P d870164263bb44b6b780a557daea3656: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:25.592963 19378 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1129f72158ec4bae89da3c94cdb70839" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74" is_pre_election: true
I20260130 02:58:25.604527 19378 raft_consensus.cc:2468] T 1129f72158ec4bae89da3c94cdb70839 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d870164263bb44b6b780a557daea3656 in term 0.
I20260130 02:58:25.605859 19462 leader_election.cc:304] T 1129f72158ec4bae89da3c94cdb70839 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78c2c46c7520492db35d51c244fb4f74, d870164263bb44b6b780a557daea3656; no voters:
I20260130 02:58:25.606607 21211 raft_consensus.cc:2804] T 1129f72158ec4bae89da3c94cdb70839 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260130 02:58:25.606932 21211 raft_consensus.cc:493] T 1129f72158ec4bae89da3c94cdb70839 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260130 02:58:25.607384 21211 raft_consensus.cc:3060] T 1129f72158ec4bae89da3c94cdb70839 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Advancing to term 1
E20260130 02:58:25.315606 18875 catalog_manager.cc:5809] Error updating tablets from 371a36d219164974abdf848841528152: Illegal state: replica 658299197a424f53a37f14fadf089ae4 is not leader of this config: current role FOLLOWER. Tablet report was: is_incremental: true updated_tablets { tablet_id: "49efcfced0b74c48962393149ac2a500" state: RUNNING consensus_state { current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "d480bf18140245e3870a62582d5a8318" state: RUNNING consensus_state { current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "133e9bfe175f44aab1a53114abe67e4d" state: RUNNING consensus_state { current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "a3258852aed24ed582a79f77860e6f0d" state: RUNNING consensus_state { current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "12c3f3ec523040e7b7cd356e3da57e0d" state: RUNNING consensus_state { current_term: 1 leader_uuid: "371a36d219164974abdf848841528152" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } health_report { overall_health: HEALTHY } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY stats { } } updated_tablets { tablet_id: "320dbe98a32a43b9899bbeed80b20f55" state: RUNNING consensus_state { current_term: 1 leader_uuid: "371a36d219164974abdf848841528152" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } health_report { overall_health: HEALTHY } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY stats { } } updated_tablets { tablet_id: "13f0ace4d9de43abab08eca32cdb30c1" state: RUNNING consensus_state { current_term: 1 leader_uuid: "371a36d219164974abdf848841528152" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } health_report { overall_health: HEALTHY } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY stats { on_disk_size: 27281 live_row_count: 0 } } updated_tablets { tablet_id: "3aa76993beb94b12bc706a76db26a03b" state: RUNNING consensus_state { current_term: 1 leader_uuid: "d870164263bb44b6b780a557daea3656" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "838e49e5e8874db393bdc15a1c92f87e" state: RUNNING consensus_state { current_term: 1 leader_uuid: "371a36d219164974abdf848841528152" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } health_report { overall_health: HEALTHY } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY stats { on_disk_size: 27281 live_row_count: 0 } } updated_tablets { tablet_id: "f9e5f01e69254d658c369f3e21cef523" state: RUNNING consensus_state { current_term: 1 leader_uuid: "d870164263bb44b6b780a557daea3656" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "a90712651a604b83ba187f893ab17711" state: RUNNING consensus_state { current_term: 1 leader_uuid: "371a36d219164974abdf848841528152" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } health_report { overall_health: HEALTHY } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY stats { on_disk_size: 27356 live_row_count: 0 } } updated_tablets { tablet_id: "78e9bc3e55fe4e4a98e8eb01da59d365" state: STOPPING consensus_state { current_term: 1 leader_uuid: "d870164263bb44b6b780a557daea3656" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "aae68390dbbe4dc58ce3d3bd01725ae6" state: RUNNING consensus_state { current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "2c79a3aecab844698e70834999e1caac" state: RUNNING consensus_state { current_term: 1 leader_uuid: "d870164263bb44b6b780a557daea3656" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "f41f4cff7dd4489f948700a45e9a7057" state: RUNNING consensus_state { current_term: 1 leader_uuid: "d870164263bb44b6b780a557daea3656" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "341041cb0be2456e88403367c0814704" state: RUNNING consensus_state { current_term: 1 leader_uuid: "d870164263bb44b6b780a557daea3656" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } } } schema_version: 1 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "5fbe92a2227d44d88b4531a7f8aa0e08" state: RUNNING consensus_state { current_term: 1 leader_uuid: "371a36d219164974abdf848841528152" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } health_report { overall_health: HEALTHY } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY stats { on_disk_size: 27356 live_row_count: 0 } } updated_tablets { tablet_id: "657e9b0d8c05413f904129dce745710b" state: RUNNING consensus_state { current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "7caf7cbad5054396b5617094e4d536a0" state: RUNNING consensus_state { current_term: 1 leader_uuid: "d870164263bb44b6b780a557daea3656" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } } } schema_version: 1 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "ba91a508b1cb4430a4d8ca26946aecc8" state: RUNNING consensus_state { current_term: 1 leader_uuid: "371a36d219164974abdf848841528152" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } health_report { overall_health: HEALTHY } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY stats { on_disk_size: 27356 live_row_count: 0 } } updated_tablets { tablet_id: "4907294442c145c6b03eb1de057c66a8" state: RUNNING consensus_state { current_term: 1 leader_uuid: "78c2c46c7520492db35d51c244fb4f74" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "03025d13f0e545cfa6288677a2715343" state: RUNNING consensus_state { current_term: 1 leader_uuid: "d870164263bb44b6b780a557daea3656" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "9fe0448bcc4a4346b3c67fc3f626ed3c" state: RUNNING consensus_state { current_term: 1 leader_uuid: "78c2c46c7520492db35d51c244fb4f74" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } } } schema_version: 1 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "4c139aa8cd184f488cec4e05ef7dcada" state: RUNNING consensus_state { current_term: 1 leader_uuid: "78c2c46c7520492db35d51c244fb4f74" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } } } schema_version: 1 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "f2d01bfdc9c4494386bd02bed655fe63" state: RUNNING consensus_state { current_term: 1 leader_uuid: "d870164263bb44b6b780a557daea3656" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } } } schema_version: 3 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "16691c9315e94253bb9b3fb5058e5636" state: RUNNING consensus_state { current_term: 1 leader_uuid: "78c2c46c7520492db35d51c244fb4f74" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } } } schema_version: 1 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "753344737b5941d9a93b6779e72f63d4" state: RUNNING consensus_state { current_term: 1 leader_uuid: "78c2c46c7520492db35d51c244fb4f74" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "f7607df5efd543438fb2dff3667cc8c8" state: RUNNING consensus_state { current_term: 1 leader_uuid: "371a36d219164974abdf848841528152" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } health_report { overall_health: HEALTHY } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY stats { } } updated_tablets { tablet_id: "43cec19c51ec4f05b63800c07f9534ef" state: RUNNING consensus_state { current_term: 1 leader_uuid: "78c2c46c7520492db35d51c244fb4f74" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "25fd48e693424537b91ba7c3fcc96878" state: RUNNING consensus_state { current_term: 1 leader_uuid: "371a36d219164974abdf848841528152" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } health_report { overall_health: HEALTHY } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY stats { on_disk_size: 27325 live_row_count: 0 } } updated_tablets { tablet_id: "3ae5472cba1e4613abe4cda42ccb2496" state: RUNNING consensus_state { current_term: 1 leader_uuid: "371a36d219164974abdf848841528152" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } health_report { overall_health: HEALTHY } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY stats { on_disk_size: 27356 live_row_count: 0 } } updated_tablets { tablet_id: "6218f2f7d73a42428b57a45858f53afd" state: RUNNING consensus_state { current_term: 1 leader_uuid: "371a36d219164974abdf848841528152" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } health_report { overall_health: HEALTHY } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY stats { on_disk_size: 27325 live_row_count: 0 } } updated_tablets { tablet_id: "274530359b174ff79f046c2d4a6afa76" state: RUNNING consensus_state { current_term: 1 leader_uuid: "371a36d219164974abdf848841528152" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } health_report { overall_health: HEALTHY } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY stats { on_disk_size: 27356 live_row_count: 0 } } updated_tablets { tablet_id: "3593f8b89da24ff9b6cf549d5ac58bc6" state: RUNNING consensus_state { current_term: 1 leader_uuid: "78c2c46c7520492db35d51c244fb4f74" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "ee7c8ff92ca9465fa18be6eae63d78ca" state: RUNNING consensus_state { current_term: 1 leader_uuid: "78c2c46c7520492db35d51c244fb4f74" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } } } schema_version: 1 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "7783db3568f4434e91e9e456799cc312" state: RUNNING consensus_state { current_term: 1 leader_uuid: "371a36d219164974abdf848841528152" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } health_report { overall_health: HEALTHY } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY stats { on_disk_size: 27356 live_row_count: 0 } } updated_tablets { tablet_id: "54be22352e404ff9b798a075693f7f21" state: RUNNING consensus_state { current_term: 1 leader_uuid: "371a36d219164974abdf848841528152" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } health_report { overall_health: HEALTHY } } } } schema_version: 1 tablet_data_state: TABLET_DATA_READY stats { on_disk_size: 27578 live_row_count: 0 } } updated_tablets { tablet_id: "e34d5da0ad864b2eaea0d575c982e447" state: RUNNING consensus_state { current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "b8c579ff5fa84651ba9125457d927162" state: RUNNING consensus_state { current_term: 1 leader_uuid: "78c2c46c7520492db35d51c244fb4f74" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "91f954f3ef4244c9a575a7421a7e75eb" state: RUNNING consensus_state { current_term: 1 leader_uuid: "d870164263bb44b6b780a557daea3656" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } } } schema_version: 1 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "111287b5557d4abda1c5cd560fb0d019" state: RUNNING consensus_state { current_term: 1 leader_uuid: "371a36d219164974abdf848841528152" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } health_report { ov
I20260130 02:58:25.612133 19219 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1129f72158ec4bae89da3c94cdb70839" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152" is_pre_election: true
I20260130 02:58:25.613461 19219 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0d14fb95ace9447f810ccbed4d5befdf" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152"
W20260130 02:58:25.614557 19463 leader_election.cc:343] T 9896ff9364744857968e87e80e776dd2 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:25.615617 21211 raft_consensus.cc:515] T 1129f72158ec4bae89da3c94cdb70839 P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:25.615877 20998 consensus_queue.cc:1048] T eb7a755ab62b4552bb8588264c5f7b75 P d870164263bb44b6b780a557daea3656 [LEADER]: Connected to new peer: Peer: permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260130 02:58:25.618034 19807 ts_tablet_manager.cc:1434] T cfd12a72726846e587cff9d6a4d61bd6 P 371a36d219164974abdf848841528152: Time spent starting tablet: real 0.122s user 0.005s sys 0.001s
I20260130 02:58:25.619242 19807 tablet_bootstrap.cc:492] T c76f0dc7762c49c389fa6fef41f02141 P 371a36d219164974abdf848841528152: Bootstrap starting.
I20260130 02:58:25.625355 19376 raft_consensus.cc:2468] T 0d14fb95ace9447f810ccbed4d5befdf P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d870164263bb44b6b780a557daea3656 in term 1.
W20260130 02:58:25.626839 18875 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.18.100.65:51021 (request call id 43) took 2478 ms (client timeout 1000 ms). Trace:
W20260130 02:58:25.627183 18875 rpcz_store.cc:269] 0130 02:58:23.148494 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:58:23.208147 (+ 59653us) service_pool.cc:225] Handling call
0130 02:58:25.315369 (+2107222us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:58:25.315401 (+ 32us) write_op.cc:454] Released schema lock
0130 02:58:25.626767 (+311366us) inbound_call.cc:173] Queueing failure response
Related trace 'op':
0130 02:58:25.315173 (+ 0us) op_driver.cc:389] HandleFailure(Illegal state: replica 658299197a424f53a37f14fadf089ae4 is not leader of this config: current role FOLLOWER)
0130 02:58:25.315207 (+ 34us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:58:25.315229 (+ 22us) write_op.cc:454] Released schema lock
0130 02:58:25.315249 (+ 20us) write_op.cc:335] FINISH: Op aborted
Metrics: {"mutex_wait_us":2051,"reported_tablets":65,"child_traces":[["op",{}]]}
I20260130 02:58:25.644655 19625 tablet_bootstrap.cc:492] T 532e867fd5944d468d7672d2a5c92f3c P d870164263bb44b6b780a557daea3656: No bootstrap required, opened a new log
I20260130 02:58:25.645040 19625 ts_tablet_manager.cc:1403] T 532e867fd5944d468d7672d2a5c92f3c P d870164263bb44b6b780a557daea3656: Time spent bootstrapping tablet: real 0.067s user 0.004s sys 0.004s
I20260130 02:58:25.648476 19625 raft_consensus.cc:359] T 532e867fd5944d468d7672d2a5c92f3c P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:25.649080 19625 raft_consensus.cc:385] T 532e867fd5944d468d7672d2a5c92f3c P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:25.649333 19625 raft_consensus.cc:740] T 532e867fd5944d468d7672d2a5c92f3c P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d870164263bb44b6b780a557daea3656, State: Initialized, Role: FOLLOWER
I20260130 02:58:25.663117 19625 consensus_queue.cc:260] T 532e867fd5944d468d7672d2a5c92f3c P d870164263bb44b6b780a557daea3656 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:25.665504 19625 ts_tablet_manager.cc:1434] T 532e867fd5944d468d7672d2a5c92f3c P d870164263bb44b6b780a557daea3656: Time spent starting tablet: real 0.020s user 0.003s sys 0.001s
I20260130 02:58:25.667255 19625 tablet_bootstrap.cc:492] T 3738385130c044e0a5a5803da74c567d P d870164263bb44b6b780a557daea3656: Bootstrap starting.
I20260130 02:58:25.672508 21211 leader_election.cc:290] T 1129f72158ec4bae89da3c94cdb70839 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 election: Requested vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767)
W20260130 02:58:25.680142 19159 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.100.67:39571, user_credentials={real_user=slave}} blocked reactor thread for 51583.3us
I20260130 02:58:25.681886 19159 leader_election.cc:304] T cd30d7adcc4545848e3501a205a10293 P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 371a36d219164974abdf848841528152; no voters: 78c2c46c7520492db35d51c244fb4f74, d870164263bb44b6b780a557daea3656
I20260130 02:58:25.684377 18870 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:50284 (request call id 270) took 3004 ms. Trace:
W20260130 02:58:25.684509 19463 leader_election.cc:343] T 1129f72158ec4bae89da3c94cdb70839 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:25.684898 18870 rpcz_store.cc:276] 0130 02:58:22.679777 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:58:25.683857 (+3004080us) service_pool.cc:225] Handling call
0130 02:58:25.684318 (+ 461us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260130 02:58:25.686374 18870 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:50284 (request call id 271) took 3006 ms. Trace:
I20260130 02:58:25.687829 19462 leader_election.cc:304] T 0d14fb95ace9447f810ccbed4d5befdf P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78c2c46c7520492db35d51c244fb4f74, d870164263bb44b6b780a557daea3656; no voters:
I20260130 02:58:25.688621 20998 raft_consensus.cc:2804] T 0d14fb95ace9447f810ccbed4d5befdf P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Leader election won for term 1
I20260130 02:58:25.689011 20998 raft_consensus.cc:697] T 0d14fb95ace9447f810ccbed4d5befdf P d870164263bb44b6b780a557daea3656 [term 1 LEADER]: Becoming Leader. State: Replica: d870164263bb44b6b780a557daea3656, State: Running, Role: LEADER
I20260130 02:58:25.689675 20998 consensus_queue.cc:237] T 0d14fb95ace9447f810ccbed4d5befdf P d870164263bb44b6b780a557daea3656 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:25.673810 19366 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1129f72158ec4bae89da3c94cdb70839" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74"
I20260130 02:58:25.699333 19625 tablet_bootstrap.cc:654] T 3738385130c044e0a5a5803da74c567d P d870164263bb44b6b780a557daea3656: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:25.699483 19366 raft_consensus.cc:3060] T 1129f72158ec4bae89da3c94cdb70839 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:25.679534 21121 consensus_queue.cc:1048] T 1fc8f09d6329407c8078542b4123ef4d P 78c2c46c7520492db35d51c244fb4f74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260130 02:58:25.686889 18873 catalog_manager.cc:5654] T dfbbe9045e7c4aa5b520218687e7e4c1 P d870164263bb44b6b780a557daea3656 reported cstate change: term changed from 0 to 1, leader changed from <none> to d870164263bb44b6b780a557daea3656 (127.18.100.67). New cstate: current_term: 1 leader_uuid: "d870164263bb44b6b780a557daea3656" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } health_report { overall_health: UNKNOWN } } }
I20260130 02:58:25.703924 18873 catalog_manager.cc:5746] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571) does not have the latest schema for tablet 345eaaa5625c48d099280b7fbb2e514b (table default.Table_c1886dd9a7934e7797e0e7e0b065d692 [id=7a2d5367224b492b844199088b01b149]). Expected version 1 got 0
W20260130 02:58:25.714604 19463 leader_election.cc:343] T 0d14fb95ace9447f810ccbed4d5befdf P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:25.716151 18873 catalog_manager.cc:5746] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571) does not have the latest schema for tablet b1e8b78eff0d4a0b92059adb2de4a3ce (table default.Table_d0b4516d24384e5386c1debaf2543adb [id=f1342f9497b74e7a9ecad1de8082578d]). Expected version 1 got 0
I20260130 02:58:25.718147 18875 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:50284 (request call id 272) took 3037 ms. Trace:
I20260130 02:58:25.734144 18875 rpcz_store.cc:276] 0130 02:58:22.680236 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:58:25.717749 (+3037513us) service_pool.cc:225] Handling call
0130 02:58:25.718096 (+ 347us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260130 02:58:25.735255 21200 raft_consensus.cc:3060] T cd30d7adcc4545848e3501a205a10293 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:25.686821 18870 rpcz_store.cc:276] 0130 02:58:22.680017 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:58:25.685992 (+3005975us) service_pool.cc:225] Handling call
0130 02:58:25.686324 (+ 332us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260130 02:58:25.736407 18875 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:50284 (request call id 273) took 3055 ms. Trace:
I20260130 02:58:25.736732 18875 rpcz_store.cc:276] 0130 02:58:22.680476 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:58:25.735932 (+3055456us) service_pool.cc:225] Handling call
0130 02:58:25.736356 (+ 424us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260130 02:58:25.737538 18870 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:50284 (request call id 274) took 2685 ms. Trace:
I20260130 02:58:25.740099 19840 tablet_bootstrap.cc:492] T 30cc52136bb3422895d95c0fed0c6798 P 78c2c46c7520492db35d51c244fb4f74: No bootstrap required, opened a new log
I20260130 02:58:25.742339 19807 tablet_bootstrap.cc:654] T c76f0dc7762c49c389fa6fef41f02141 P 371a36d219164974abdf848841528152: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:25.750847 19840 ts_tablet_manager.cc:1403] T 30cc52136bb3422895d95c0fed0c6798 P 78c2c46c7520492db35d51c244fb4f74: Time spent bootstrapping tablet: real 0.384s user 0.016s sys 0.000s
I20260130 02:58:25.753676 19840 raft_consensus.cc:359] T 30cc52136bb3422895d95c0fed0c6798 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:25.754120 21211 raft_consensus.cc:493] T 6d1f121f49cd47c2b8247daaf6874e01 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:25.754513 18870 rpcz_store.cc:276] 0130 02:58:23.052063 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:58:25.737056 (+2684993us) service_pool.cc:225] Handling call
0130 02:58:25.737488 (+ 432us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260130 02:58:25.754540 21211 raft_consensus.cc:515] T 6d1f121f49cd47c2b8247daaf6874e01 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:25.722266 18873 catalog_manager.cc:5654] T e34d5da0ad864b2eaea0d575c982e447 P d870164263bb44b6b780a557daea3656 reported cstate change: term changed from 0 to 1, leader changed from <none> to d870164263bb44b6b780a557daea3656 (127.18.100.67). New cstate: current_term: 1 leader_uuid: "d870164263bb44b6b780a557daea3656" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } health_report { overall_health: UNKNOWN } } }
I20260130 02:58:25.756107 18870 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:50284 (request call id 276) took 2582 ms. Trace:
I20260130 02:58:25.756477 18870 rpcz_store.cc:276] 0130 02:58:23.173508 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:58:25.755630 (+2582122us) service_pool.cc:225] Handling call
0130 02:58:25.756056 (+ 426us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260130 02:58:25.741448 18875 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:50284 (request call id 275) took 2661 ms. Trace:
I20260130 02:58:25.756847 18873 catalog_manager.cc:5654] T 2b0edaad6877460cb7982efc62b00b66 P d870164263bb44b6b780a557daea3656 reported cstate change: term changed from 0 to 1, leader changed from <none> to d870164263bb44b6b780a557daea3656 (127.18.100.67). New cstate: current_term: 1 leader_uuid: "d870164263bb44b6b780a557daea3656" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } health_report { overall_health: UNKNOWN } } }
I20260130 02:58:25.758251 19840 raft_consensus.cc:385] T 30cc52136bb3422895d95c0fed0c6798 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:25.758772 19840 raft_consensus.cc:740] T 30cc52136bb3422895d95c0fed0c6798 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78c2c46c7520492db35d51c244fb4f74, State: Initialized, Role: FOLLOWER
I20260130 02:58:25.759402 19840 consensus_queue.cc:260] T 30cc52136bb3422895d95c0fed0c6798 P 78c2c46c7520492db35d51c244fb4f74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:25.761528 19840 ts_tablet_manager.cc:1434] T 30cc52136bb3422895d95c0fed0c6798 P 78c2c46c7520492db35d51c244fb4f74: Time spent starting tablet: real 0.010s user 0.005s sys 0.000s
I20260130 02:58:25.762405 19840 tablet_bootstrap.cc:492] T 475b435b2967418f927fa172d2ff04ea P 78c2c46c7520492db35d51c244fb4f74: Bootstrap starting.
I20260130 02:58:25.757304 18875 rpcz_store.cc:276] 0130 02:58:23.080048 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:58:25.741039 (+2660991us) service_pool.cc:225] Handling call
0130 02:58:25.741398 (+ 359us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260130 02:58:25.765731 18873 catalog_manager.cc:5746] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571) does not have the latest schema for tablet 2b0edaad6877460cb7982efc62b00b66 (table default.Table_d95381c590c94802bdc83acef0f0df96 [id=64a8e0e079f34806aa68795f3d548c5e]). Expected version 1 got 0
I20260130 02:58:25.757957 18870 master_service.cc:954] ReplaceTablet: received request to replace tablet 345eaaa5625c48d099280b7fbb2e514b from {username='slave'} at 127.0.0.1:50284
I20260130 02:58:25.771009 18870 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:50284 (request call id 277) took 1761 ms. Trace:
I20260130 02:58:25.771411 18870 rpcz_store.cc:276] 0130 02:58:24.009344 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:58:25.757847 (+1748503us) service_pool.cc:225] Handling call
0130 02:58:25.770955 (+ 13108us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260130 02:58:25.772332 18873 catalog_manager.cc:5746] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571) does not have the latest schema for tablet 274530359b174ff79f046c2d4a6afa76 (table default.Table_c4f4904b6fac4b9880979cf0314e71c1 [id=c456c6418c93499a9d08f652720854ca]). Expected version 1 got 0
I20260130 02:58:25.772549 18870 master_service.cc:954] ReplaceTablet: received request to replace tablet 111287b5557d4abda1c5cd560fb0d019 from {username='slave'} at 127.0.0.1:50284
I20260130 02:58:25.773224 18870 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:50284 (request call id 278) took 1431 ms. Trace:
W20260130 02:58:25.734541 19159 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.100.67:39571, user_credentials={real_user=slave}} blocked reactor thread for 52736.6us
I20260130 02:58:25.773710 18873 catalog_manager.cc:5654] T 133e9bfe175f44aab1a53114abe67e4d P d870164263bb44b6b780a557daea3656 reported cstate change: term changed from 0 to 1, leader changed from <none> to d870164263bb44b6b780a557daea3656 (127.18.100.67). New cstate: current_term: 1 leader_uuid: "d870164263bb44b6b780a557daea3656" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } health_report { overall_health: UNKNOWN } } }
I20260130 02:58:25.779950 21133 consensus_queue.cc:1048] T c76f0dc7762c49c389fa6fef41f02141 P 78c2c46c7520492db35d51c244fb4f74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260130 02:58:25.785493 19378 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6d1f121f49cd47c2b8247daaf6874e01" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74" is_pre_election: true
I20260130 02:58:25.786103 19378 raft_consensus.cc:2468] T 6d1f121f49cd47c2b8247daaf6874e01 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d870164263bb44b6b780a557daea3656 in term 0.
I20260130 02:58:25.787041 19462 leader_election.cc:304] T 6d1f121f49cd47c2b8247daaf6874e01 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78c2c46c7520492db35d51c244fb4f74, d870164263bb44b6b780a557daea3656; no voters:
I20260130 02:58:25.787884 21215 raft_consensus.cc:2804] T 6d1f121f49cd47c2b8247daaf6874e01 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260130 02:58:25.788199 21215 raft_consensus.cc:493] T 6d1f121f49cd47c2b8247daaf6874e01 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260130 02:58:25.788529 21215 raft_consensus.cc:3060] T 6d1f121f49cd47c2b8247daaf6874e01 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:25.793412 19224 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1129f72158ec4bae89da3c94cdb70839" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152"
I20260130 02:58:25.801512 21192 raft_consensus.cc:493] T 00000000000000000000000000000000 P 107f274cdda543038832f06a56ab9130 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:25.801945 21192 raft_consensus.cc:515] T 00000000000000000000000000000000 P 107f274cdda543038832f06a56ab9130 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "658299197a424f53a37f14fadf089ae4" member_type: VOTER last_known_addr { host: "127.18.100.126" port: 38191 } } peers { permanent_uuid: "107f274cdda543038832f06a56ab9130" member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39711 } } peers { permanent_uuid: "3749baa287ed4f249329a74071781edd" member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34629 } }
I20260130 02:58:25.804003 21192 leader_election.cc:290] T 00000000000000000000000000000000 P 107f274cdda543038832f06a56ab9130 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 658299197a424f53a37f14fadf089ae4 (127.18.100.126:38191), 3749baa287ed4f249329a74071781edd (127.18.100.124:34629)
I20260130 02:58:25.805630 20810 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "107f274cdda543038832f06a56ab9130" candidate_term: 5 candidate_status { last_received { term: 3 index: 372 } } ignore_live_leader: false dest_uuid: "3749baa287ed4f249329a74071781edd" is_pre_election: true
I20260130 02:58:25.806169 20810 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 3749baa287ed4f249329a74071781edd [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 107f274cdda543038832f06a56ab9130 in term 4.
I20260130 02:58:25.807090 18935 leader_election.cc:304] T 00000000000000000000000000000000 P 107f274cdda543038832f06a56ab9130 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 107f274cdda543038832f06a56ab9130, 3749baa287ed4f249329a74071781edd; no voters:
I20260130 02:58:25.807945 21192 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 107f274cdda543038832f06a56ab9130 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260130 02:58:25.808228 21192 raft_consensus.cc:493] T 00000000000000000000000000000000 P 107f274cdda543038832f06a56ab9130 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260130 02:58:25.808117 19366 raft_consensus.cc:2468] T 1129f72158ec4bae89da3c94cdb70839 P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d870164263bb44b6b780a557daea3656 in term 1.
I20260130 02:58:25.808503 21192 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 107f274cdda543038832f06a56ab9130 [term 4 FOLLOWER]: Advancing to term 5
I20260130 02:58:25.809253 19462 leader_election.cc:304] T 1129f72158ec4bae89da3c94cdb70839 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78c2c46c7520492db35d51c244fb4f74, d870164263bb44b6b780a557daea3656; no voters:
I20260130 02:58:25.809949 20998 raft_consensus.cc:2804] T 1129f72158ec4bae89da3c94cdb70839 P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Leader election won for term 1
I20260130 02:58:25.810039 19224 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6d1f121f49cd47c2b8247daaf6874e01" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152" is_pre_election: true
I20260130 02:58:25.810341 20998 raft_consensus.cc:697] T 1129f72158ec4bae89da3c94cdb70839 P d870164263bb44b6b780a557daea3656 [term 1 LEADER]: Becoming Leader. State: Replica: d870164263bb44b6b780a557daea3656, State: Running, Role: LEADER
I20260130 02:58:25.773553 18870 rpcz_store.cc:276] 0130 02:58:24.341789 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:58:25.772428 (+1430639us) service_pool.cc:225] Handling call
0130 02:58:25.773178 (+ 750us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260130 02:58:25.820030 19376 raft_consensus.cc:1275] T ca06f07bb98f45a7bf63aebc28348e08 P 78c2c46c7520492db35d51c244fb4f74 [term 2 FOLLOWER]: Refusing update from remote peer 371a36d219164974abdf848841528152: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20260130 02:58:25.806354 18885 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "107f274cdda543038832f06a56ab9130" candidate_term: 5 candidate_status { last_received { term: 3 index: 372 } } ignore_live_leader: false dest_uuid: "658299197a424f53a37f14fadf089ae4" is_pre_election: true
I20260130 02:58:25.824050 21192 raft_consensus.cc:515] T 00000000000000000000000000000000 P 107f274cdda543038832f06a56ab9130 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "658299197a424f53a37f14fadf089ae4" member_type: VOTER last_known_addr { host: "127.18.100.126" port: 38191 } } peers { permanent_uuid: "107f274cdda543038832f06a56ab9130" member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39711 } } peers { permanent_uuid: "3749baa287ed4f249329a74071781edd" member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34629 } }
I20260130 02:58:25.826640 18885 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "107f274cdda543038832f06a56ab9130" candidate_term: 5 candidate_status { last_received { term: 3 index: 372 } } ignore_live_leader: false dest_uuid: "658299197a424f53a37f14fadf089ae4"
I20260130 02:58:25.829694 20810 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "107f274cdda543038832f06a56ab9130" candidate_term: 5 candidate_status { last_received { term: 3 index: 372 } } ignore_live_leader: false dest_uuid: "3749baa287ed4f249329a74071781edd"
I20260130 02:58:25.830173 20810 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 3749baa287ed4f249329a74071781edd [term 4 FOLLOWER]: Advancing to term 5
I20260130 02:58:25.835750 20810 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 3749baa287ed4f249329a74071781edd [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 107f274cdda543038832f06a56ab9130 in term 5.
I20260130 02:58:25.836889 18935 leader_election.cc:304] T 00000000000000000000000000000000 P 107f274cdda543038832f06a56ab9130 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 107f274cdda543038832f06a56ab9130, 3749baa287ed4f249329a74071781edd; no voters: 658299197a424f53a37f14fadf089ae4
I20260130 02:58:25.837903 21246 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 107f274cdda543038832f06a56ab9130 [term 5 FOLLOWER]: Leader election won for term 5
I20260130 02:58:25.838328 21246 raft_consensus.cc:697] T 00000000000000000000000000000000 P 107f274cdda543038832f06a56ab9130 [term 5 LEADER]: Becoming Leader. State: Replica: 107f274cdda543038832f06a56ab9130, State: Running, Role: LEADER
I20260130 02:58:25.839074 21246 consensus_queue.cc:237] T 00000000000000000000000000000000 P 107f274cdda543038832f06a56ab9130 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 372, Committed index: 372, Last appended: 3.372, Last appended by leader: 372, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "658299197a424f53a37f14fadf089ae4" member_type: VOTER last_known_addr { host: "127.18.100.126" port: 38191 } } peers { permanent_uuid: "107f274cdda543038832f06a56ab9130" member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39711 } } peers { permanent_uuid: "3749baa287ed4f249329a74071781edd" member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34629 } }
I20260130 02:58:25.844264 21164 catalog_manager.cc:1514] Loading token signing keys...
I20260130 02:58:25.845901 21121 consensus_queue.cc:1048] T 306452e208ca4e719609f67345114a13 P 78c2c46c7520492db35d51c244fb4f74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260130 02:58:25.851083 19807 tablet_bootstrap.cc:492] T c76f0dc7762c49c389fa6fef41f02141 P 371a36d219164974abdf848841528152: No bootstrap required, opened a new log
I20260130 02:58:25.851604 19807 ts_tablet_manager.cc:1403] T c76f0dc7762c49c389fa6fef41f02141 P 371a36d219164974abdf848841528152: Time spent bootstrapping tablet: real 0.233s user 0.009s sys 0.002s
I20260130 02:58:25.852080 20807 raft_consensus.cc:1240] T 00000000000000000000000000000000 P 3749baa287ed4f249329a74071781edd [term 5 FOLLOWER]: Rejecting Update request from peer 658299197a424f53a37f14fadf089ae4 for earlier term 4. Current term is 5. Ops: []
I20260130 02:58:25.853746 21240 consensus_queue.cc:1059] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3749baa287ed4f249329a74071781edd" member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34629 }, Status: INVALID_TERM, Last received: 0.0, Next index: 373, Last known committed idx: 371, Time since last communication: 0.000s
I20260130 02:58:25.854478 21240 raft_consensus.cc:3055] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4 [term 4 LEADER]: Stepping down as leader of term 4
I20260130 02:58:25.855320 21240 raft_consensus.cc:740] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 658299197a424f53a37f14fadf089ae4, State: Running, Role: LEADER
I20260130 02:58:25.853858 19807 raft_consensus.cc:359] T c76f0dc7762c49c389fa6fef41f02141 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:25.856134 21240 consensus_queue.cc:260] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 372, Committed index: 372, Last appended: 4.373, Last appended by leader: 373, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "658299197a424f53a37f14fadf089ae4" member_type: VOTER last_known_addr { host: "127.18.100.126" port: 38191 } } peers { permanent_uuid: "107f274cdda543038832f06a56ab9130" member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39711 } } peers { permanent_uuid: "3749baa287ed4f249329a74071781edd" member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34629 } }
I20260130 02:58:25.856741 19807 raft_consensus.cc:385] T c76f0dc7762c49c389fa6fef41f02141 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:25.857009 19807 raft_consensus.cc:740] T c76f0dc7762c49c389fa6fef41f02141 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 371a36d219164974abdf848841528152, State: Initialized, Role: FOLLOWER
I20260130 02:58:25.857127 21240 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4 [term 4 FOLLOWER]: Advancing to term 5
I20260130 02:58:25.858706 19625 tablet_bootstrap.cc:492] T 3738385130c044e0a5a5803da74c567d P d870164263bb44b6b780a557daea3656: No bootstrap required, opened a new log
I20260130 02:58:25.859155 19625 ts_tablet_manager.cc:1403] T 3738385130c044e0a5a5803da74c567d P d870164263bb44b6b780a557daea3656: Time spent bootstrapping tablet: real 0.192s user 0.012s sys 0.011s
I20260130 02:58:25.860862 18873 catalog_manager.cc:5746] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571) does not have the latest schema for tablet f2d01bfdc9c4494386bd02bed655fe63 (table default.Table_aaf139c1a55d4a1783298ca12ae0d5c0 [id=d128767e303145a885eec3c5f714a27f]). Expected version 4 got 3
I20260130 02:58:25.867100 19807 consensus_queue.cc:260] T c76f0dc7762c49c389fa6fef41f02141 P 371a36d219164974abdf848841528152 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:25.867982 21215 raft_consensus.cc:515] T 6d1f121f49cd47c2b8247daaf6874e01 P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:25.869848 21215 leader_election.cc:290] T 6d1f121f49cd47c2b8247daaf6874e01 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 election: Requested vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767)
I20260130 02:58:25.810986 20998 consensus_queue.cc:237] T 1129f72158ec4bae89da3c94cdb70839 P d870164263bb44b6b780a557daea3656 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:25.870954 19366 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6d1f121f49cd47c2b8247daaf6874e01" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74"
I20260130 02:58:25.871487 19366 raft_consensus.cc:3060] T 6d1f121f49cd47c2b8247daaf6874e01 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:25.872246 19807 ts_tablet_manager.cc:1434] T c76f0dc7762c49c389fa6fef41f02141 P 371a36d219164974abdf848841528152: Time spent starting tablet: real 0.020s user 0.001s sys 0.003s
I20260130 02:58:25.872530 18873 catalog_manager.cc:5654] T 320dbe98a32a43b9899bbeed80b20f55 P d870164263bb44b6b780a557daea3656 reported cstate change: term changed from 0 to 1, leader changed from <none> to 371a36d219164974abdf848841528152 (127.18.100.65). New cstate: current_term: 1 leader_uuid: "371a36d219164974abdf848841528152" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } }
I20260130 02:58:25.873399 19807 tablet_bootstrap.cc:492] T 94355fc1338f479e8798070aff04fcba P 371a36d219164974abdf848841528152: Bootstrap starting.
W20260130 02:58:25.873668 19463 leader_election.cc:343] T 1129f72158ec4bae89da3c94cdb70839 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:25.875651 21192 leader_election.cc:290] T 00000000000000000000000000000000 P 107f274cdda543038832f06a56ab9130 [CANDIDATE]: Term 5 election: Requested vote from peers 658299197a424f53a37f14fadf089ae4 (127.18.100.126:38191), 3749baa287ed4f249329a74071781edd (127.18.100.124:34629)
I20260130 02:58:25.803443 21211 leader_election.cc:290] T 6d1f121f49cd47c2b8247daaf6874e01 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767)
W20260130 02:58:25.874553 19463 leader_election.cc:343] T 6d1f121f49cd47c2b8247daaf6874e01 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:25.879936 19625 raft_consensus.cc:359] T 3738385130c044e0a5a5803da74c567d P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:25.882071 19625 raft_consensus.cc:385] T 3738385130c044e0a5a5803da74c567d P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:25.882395 19625 raft_consensus.cc:740] T 3738385130c044e0a5a5803da74c567d P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d870164263bb44b6b780a557daea3656, State: Initialized, Role: FOLLOWER
I20260130 02:58:25.883062 19625 consensus_queue.cc:260] T 3738385130c044e0a5a5803da74c567d P d870164263bb44b6b780a557daea3656 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:25.884887 19625 ts_tablet_manager.cc:1434] T 3738385130c044e0a5a5803da74c567d P d870164263bb44b6b780a557daea3656: Time spent starting tablet: real 0.026s user 0.006s sys 0.000s
I20260130 02:58:25.874106 18873 catalog_manager.cc:5654] T 657e9b0d8c05413f904129dce745710b P d870164263bb44b6b780a557daea3656 reported cstate change: term changed from 0 to 1, leader changed from <none> to d870164263bb44b6b780a557daea3656 (127.18.100.67). New cstate: current_term: 1 leader_uuid: "d870164263bb44b6b780a557daea3656" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } health_report { overall_health: HEALTHY } } }
I20260130 02:58:25.886502 19625 tablet_bootstrap.cc:492] T e691c238199b4914bcc3fd248f31420c P d870164263bb44b6b780a557daea3656: Bootstrap starting.
I20260130 02:58:25.888195 19366 raft_consensus.cc:2468] T 6d1f121f49cd47c2b8247daaf6874e01 P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d870164263bb44b6b780a557daea3656 in term 1.
I20260130 02:58:25.888939 18873 catalog_manager.cc:5746] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571) does not have the latest schema for tablet 657e9b0d8c05413f904129dce745710b (table default.Table_526677c8ef4144bd9de7a54a5e78e2be [id=b16908b75587459597a978c71756451c]). Expected version 1 got 0
I20260130 02:58:25.889580 18873 catalog_manager.cc:5746] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571) does not have the latest schema for tablet 49efcfced0b74c48962393149ac2a500 (table default.Table_2e81f0a0cbc449a397141b717db17025 [id=a9c919f57c0c437cb35a6921103e45a6]). Expected version 1 got 0
I20260130 02:58:25.890080 18873 catalog_manager.cc:5746] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571) does not have the latest schema for tablet 25fd48e693424537b91ba7c3fcc96878 (table default.Table_ebc8b8379c274c18b0a06a919dbe0ee0 [id=28156e0843674fce88f5914a3e4195ae]). Expected version 1 got 0
I20260130 02:58:25.890100 19462 leader_election.cc:304] T 6d1f121f49cd47c2b8247daaf6874e01 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78c2c46c7520492db35d51c244fb4f74, d870164263bb44b6b780a557daea3656; no voters:
I20260130 02:58:25.891000 21094 raft_consensus.cc:2804] T 6d1f121f49cd47c2b8247daaf6874e01 P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Leader election won for term 1
I20260130 02:58:25.891465 21094 raft_consensus.cc:697] T 6d1f121f49cd47c2b8247daaf6874e01 P d870164263bb44b6b780a557daea3656 [term 1 LEADER]: Becoming Leader. State: Replica: d870164263bb44b6b780a557daea3656, State: Running, Role: LEADER
I20260130 02:58:25.892189 21094 consensus_queue.cc:237] T 6d1f121f49cd47c2b8247daaf6874e01 P d870164263bb44b6b780a557daea3656 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:25.907863 18873 catalog_manager.cc:5654] T ca06f07bb98f45a7bf63aebc28348e08 P d870164263bb44b6b780a557daea3656 reported cstate change: term changed from 0 to 1, leader changed from <none> to d870164263bb44b6b780a557daea3656 (127.18.100.67). New cstate: current_term: 1 leader_uuid: "d870164263bb44b6b780a557daea3656" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } health_report { overall_health: UNKNOWN } } }
I20260130 02:58:25.909394 18873 catalog_manager.cc:5654] T 13f0ace4d9de43abab08eca32cdb30c1 P d870164263bb44b6b780a557daea3656 reported cstate change: term changed from 0 to 1, leader changed from <none> to 371a36d219164974abdf848841528152 (127.18.100.65). New cstate: current_term: 1 leader_uuid: "371a36d219164974abdf848841528152" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } }
I20260130 02:58:25.915450 21162 sys_catalog.cc:455] T 00000000000000000000000000000000 P 107f274cdda543038832f06a56ab9130 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 107f274cdda543038832f06a56ab9130. Latest consensus state: current_term: 5 leader_uuid: "107f274cdda543038832f06a56ab9130" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "658299197a424f53a37f14fadf089ae4" member_type: VOTER last_known_addr { host: "127.18.100.126" port: 38191 } } peers { permanent_uuid: "107f274cdda543038832f06a56ab9130" member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39711 } } peers { permanent_uuid: "3749baa287ed4f249329a74071781edd" member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34629 } } }
I20260130 02:58:25.916208 21162 sys_catalog.cc:458] T 00000000000000000000000000000000 P 107f274cdda543038832f06a56ab9130 [sys.catalog]: This master's current role is: LEADER
I20260130 02:58:25.920516 21121 consensus_queue.cc:1048] T cd30d7adcc4545848e3501a205a10293 P 78c2c46c7520492db35d51c244fb4f74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260130 02:58:25.921725 19840 tablet_bootstrap.cc:654] T 475b435b2967418f927fa172d2ff04ea P 78c2c46c7520492db35d51c244fb4f74: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:25.921096 18873 catalog_manager.cc:5654] T a3258852aed24ed582a79f77860e6f0d P d870164263bb44b6b780a557daea3656 reported cstate change: term changed from 0 to 1, leader changed from <none> to d870164263bb44b6b780a557daea3656 (127.18.100.67). New cstate: current_term: 1 leader_uuid: "d870164263bb44b6b780a557daea3656" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } health_report { overall_health: UNKNOWN } } }
W20260130 02:58:25.923259 21267 log.cc:927] Time spent T ca06f07bb98f45a7bf63aebc28348e08 P 371a36d219164974abdf848841528152: Append to log took a long time: real 0.094s user 0.000s sys 0.000s
I20260130 02:58:25.923287 18873 catalog_manager.cc:5654] T 838e49e5e8874db393bdc15a1c92f87e P d870164263bb44b6b780a557daea3656 reported cstate change: term changed from 0 to 1, leader changed from <none> to 371a36d219164974abdf848841528152 (127.18.100.65). New cstate: current_term: 1 leader_uuid: "371a36d219164974abdf848841528152" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } }
I20260130 02:58:25.931072 18873 catalog_manager.cc:5746] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571) does not have the latest schema for tablet 838e49e5e8874db393bdc15a1c92f87e (table default.Table_da3d581b15a94840898777f74611994e [id=b7e241ced53e4160ac6fa1f65ef3674e]). Expected version 1 got 0
I20260130 02:58:25.933025 18871 catalog_manager.cc:5941] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4: Sending DeleteTablet for 3 replicas of tablet e47b22aafe5247c4b62c4193c24f427c
I20260130 02:58:25.934329 18871 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:50284 (request call id 263) took 5983 ms. Trace:
I20260130 02:58:25.934651 18871 rpcz_store.cc:276] 0130 02:58:19.950642 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:58:22.256112 (+2305470us) service_pool.cc:225] Handling call
0130 02:58:22.263606 (+ 7494us) catalog_manager.cc:2393] Looking up, locking, and authorizing table
0130 02:58:22.266473 (+ 2867us) catalog_manager.cc:2815] Modifying in-memory table state
0130 02:58:22.266704 (+ 231us) catalog_manager.cc:2824] Locking tablets
0130 02:58:22.267316 (+ 612us) catalog_manager.cc:2838] Removing table and tablets from system table
0130 02:58:22.355590 (+ 88274us) catalog_manager.cc:2856] Removing table from by-name map
0130 02:58:22.362488 (+ 6898us) catalog_manager.cc:2872] Committing in-memory state
0130 02:58:25.932936 (+3570448us) catalog_manager.cc:2876] Aborting table tasks
0130 02:58:25.934272 (+ 1336us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0130 02:58:22.272156 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0130 02:58:22.272642 (+ 486us) write_op.cc:432] Acquiring schema lock in shared mode
0130 02:58:22.272670 (+ 28us) write_op.cc:435] Acquired schema lock
0130 02:58:22.272686 (+ 16us) tablet.cc:660] Decoding operations
0130 02:58:22.273058 (+ 372us) write_op.cc:620] Acquiring the partition lock for write op
0130 02:58:22.273103 (+ 45us) write_op.cc:641] Partition lock acquired for write op
0130 02:58:22.273126 (+ 23us) tablet.cc:683] Acquiring locks for 2 operations
0130 02:58:22.273418 (+ 292us) tablet.cc:699] Row locks acquired
0130 02:58:22.273436 (+ 18us) write_op.cc:260] PREPARE: finished
0130 02:58:22.273557 (+ 121us) write_op.cc:270] Start()
0130 02:58:22.273646 (+ 89us) write_op.cc:276] Timestamp: P: 1769741902273468 usec, L: 0
0130 02:58:22.273665 (+ 19us) op_driver.cc:348] REPLICATION: starting
0130 02:58:22.274187 (+ 522us) log.cc:844] Serialized 765 byte log entry
0130 02:58:22.332900 (+ 58713us) op_driver.cc:464] REPLICATION: finished
0130 02:58:22.333195 (+ 295us) write_op.cc:301] APPLY: starting
0130 02:58:22.333251 (+ 56us) tablet.cc:1365] starting BulkCheckPresence
0130 02:58:22.333357 (+ 106us) tablet.cc:1368] finished BulkCheckPresence
0130 02:58:22.333458 (+ 101us) tablet.cc:1370] starting ApplyRowOperation cycle
0130 02:58:22.333747 (+ 289us) tablet.cc:1381] finished ApplyRowOperation cycle
0130 02:58:22.333887 (+ 140us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0130 02:58:22.333929 (+ 42us) write_op.cc:312] APPLY: finished
0130 02:58:22.334096 (+ 167us) log.cc:844] Serialized 29 byte log entry
0130 02:58:22.334240 (+ 144us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:58:22.334319 (+ 79us) write_op.cc:454] Released schema lock
0130 02:58:22.334348 (+ 29us) write_op.cc:341] FINISH: Updating metrics
0130 02:58:22.334547 (+ 199us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:58:22.334607 (+ 60us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":167,"apply.run_cpu_time_us":1322,"apply.run_wall_time_us":1321,"num_ops":2,"prepare.queue_time_us":90,"prepare.run_cpu_time_us":2334,"prepare.run_wall_time_us":2335,"raft.queue_time_us":925,"raft.run_cpu_time_us":636,"raft.run_wall_time_us":636,"replication_time_us":59148}]]}
I20260130 02:58:25.941111 19358 tablet_service.cc:1552] Processing DeleteTablet for tablet e47b22aafe5247c4b62c4193c24f427c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:58:22 UTC) from {username='slave'} at 127.0.0.1:60110
I20260130 02:58:25.946835 19505 tablet_service.cc:1552] Processing DeleteTablet for tablet e47b22aafe5247c4b62c4193c24f427c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:58:22 UTC) from {username='slave'} at 127.0.0.1:59068
I20260130 02:58:25.948205 21276 tablet_replica.cc:333] T e47b22aafe5247c4b62c4193c24f427c P d870164263bb44b6b780a557daea3656: stopping tablet replica
W20260130 02:58:25.950222 18866 catalog_manager.cc:6192] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4: An error occurred while updating sys-tables: Illegal state: replica 658299197a424f53a37f14fadf089ae4 is not leader of this config: current role FOLLOWER
W20260130 02:58:25.953576 18868 catalog_manager.cc:6192] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4: An error occurred while updating sys-tables: Illegal state: replica 658299197a424f53a37f14fadf089ae4 is not leader of this config: current role FOLLOWER
I20260130 02:58:25.957605 21248 tablet.cc:1722] T b1e8b78eff0d4a0b92059adb2de4a3ce P 78c2c46c7520492db35d51c244fb4f74: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20260130 02:58:25.965741 18867 catalog_manager.cc:6192] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4: An error occurred while updating sys-tables: Illegal state: replica 658299197a424f53a37f14fadf089ae4 is not leader of this config: current role FOLLOWER
I20260130 02:58:25.975099 21133 consensus_queue.cc:1048] T 306452e208ca4e719609f67345114a13 P 78c2c46c7520492db35d51c244fb4f74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260130 02:58:25.980029 19807 tablet_bootstrap.cc:654] T 94355fc1338f479e8798070aff04fcba P 371a36d219164974abdf848841528152: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:25.996978 21121 raft_consensus.cc:493] T 532e867fd5944d468d7672d2a5c92f3c P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:25.997522 21121 raft_consensus.cc:515] T 532e867fd5944d468d7672d2a5c92f3c P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:26.000806 19524 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "532e867fd5944d468d7672d2a5c92f3c" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
I20260130 02:58:26.001601 19524 raft_consensus.cc:2468] T 532e867fd5944d468d7672d2a5c92f3c P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78c2c46c7520492db35d51c244fb4f74 in term 0.
I20260130 02:58:26.003511 19313 leader_election.cc:304] T 532e867fd5944d468d7672d2a5c92f3c P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78c2c46c7520492db35d51c244fb4f74, d870164263bb44b6b780a557daea3656; no voters:
I20260130 02:58:26.004233 21141 raft_consensus.cc:2804] T 532e867fd5944d468d7672d2a5c92f3c P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260130 02:58:26.004559 21141 raft_consensus.cc:493] T 532e867fd5944d468d7672d2a5c92f3c P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260130 02:58:26.004851 21141 raft_consensus.cc:3060] T 532e867fd5944d468d7672d2a5c92f3c P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:26.011879 21276 raft_consensus.cc:2243] T e47b22aafe5247c4b62c4193c24f427c P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260130 02:58:26.012437 21276 raft_consensus.cc:2272] T e47b22aafe5247c4b62c4193c24f427c P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260130 02:58:26.016261 21276 ts_tablet_manager.cc:1916] T e47b22aafe5247c4b62c4193c24f427c P d870164263bb44b6b780a557daea3656: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260130 02:58:26.028429 21141 raft_consensus.cc:515] T 532e867fd5944d468d7672d2a5c92f3c P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:26.030956 21121 leader_election.cc:290] T 532e867fd5944d468d7672d2a5c92f3c P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d870164263bb44b6b780a557daea3656 (127.18.100.67:39571), 371a36d219164974abdf848841528152 (127.18.100.65:42021)
I20260130 02:58:26.032032 19807 tablet_bootstrap.cc:492] T 94355fc1338f479e8798070aff04fcba P 371a36d219164974abdf848841528152: No bootstrap required, opened a new log
I20260130 02:58:26.032438 19807 ts_tablet_manager.cc:1403] T 94355fc1338f479e8798070aff04fcba P 371a36d219164974abdf848841528152: Time spent bootstrapping tablet: real 0.159s user 0.005s sys 0.005s
I20260130 02:58:26.034245 21211 consensus_queue.cc:1048] T 2b8034a92b2043e5b57cf02f1ee6296d P d870164263bb44b6b780a557daea3656 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260130 02:58:26.035198 19523 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "532e867fd5944d468d7672d2a5c92f3c" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656"
I20260130 02:58:26.035709 19523 raft_consensus.cc:3060] T 532e867fd5944d468d7672d2a5c92f3c P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:26.047753 19625 tablet_bootstrap.cc:654] T e691c238199b4914bcc3fd248f31420c P d870164263bb44b6b780a557daea3656: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:26.051887 21276 ts_tablet_manager.cc:1929] T e47b22aafe5247c4b62c4193c24f427c P d870164263bb44b6b780a557daea3656: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260130 02:58:26.052632 19807 raft_consensus.cc:359] T 94355fc1338f479e8798070aff04fcba P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:26.053269 19807 raft_consensus.cc:385] T 94355fc1338f479e8798070aff04fcba P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:26.053505 19807 raft_consensus.cc:740] T 94355fc1338f479e8798070aff04fcba P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 371a36d219164974abdf848841528152, State: Initialized, Role: FOLLOWER
I20260130 02:58:26.054098 19807 consensus_queue.cc:260] T 94355fc1338f479e8798070aff04fcba P 371a36d219164974abdf848841528152 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:26.055871 21276 log.cc:1199] T e47b22aafe5247c4b62c4193c24f427c P d870164263bb44b6b780a557daea3656: Deleting WAL directory at /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-2/wal/wals/e47b22aafe5247c4b62c4193c24f427c
I20260130 02:58:26.056663 19807 ts_tablet_manager.cc:1434] T 94355fc1338f479e8798070aff04fcba P 371a36d219164974abdf848841528152: Time spent starting tablet: real 0.024s user 0.006s sys 0.001s
I20260130 02:58:26.057538 19807 tablet_bootstrap.cc:492] T 2b8034a92b2043e5b57cf02f1ee6296d P 371a36d219164974abdf848841528152: Bootstrap starting.
I20260130 02:58:26.058561 19197 tablet_service.cc:1552] Processing DeleteTablet for tablet e47b22aafe5247c4b62c4193c24f427c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:58:22 UTC) from {username='slave'} at 127.0.0.1:42870
I20260130 02:58:26.059154 21276 ts_tablet_manager.cc:1950] T e47b22aafe5247c4b62c4193c24f427c P d870164263bb44b6b780a557daea3656: Deleting consensus metadata
I20260130 02:58:26.066522 18860 catalog_manager.cc:4985] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): tablet e47b22aafe5247c4b62c4193c24f427c (table default.Table_d11b53ac041740d8818edf7d2861dc6d [id=edeb74c78d234ee6a5a031c8347d3884]) successfully deleted
I20260130 02:58:26.085230 21095 consensus_queue.cc:1048] T ca06f07bb98f45a7bf63aebc28348e08 P 371a36d219164974abdf848841528152 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260130 02:58:26.093353 19505 tablet_service.cc:1552] Processing DeleteTablet for tablet ec28ff12927d450ead16d26cf016138d with delete_type TABLET_DATA_DELETED (replaced by tablet 6d1f121f49cd47c2b8247daaf6874e01) from {username='slave'} at 127.0.0.1:59068
I20260130 02:58:26.100441 19496 tablet_service.cc:1552] Processing DeleteTablet for tablet a8a5038e9def44eb8b3b1e4aeae25652 with delete_type TABLET_DATA_DELETED (replaced by tablet 475b435b2967418f927fa172d2ff04ea) from {username='slave'} at 127.0.0.1:59068
I20260130 02:58:26.101369 19496 tablet_service.cc:1552] Processing DeleteTablet for tablet 78e9bc3e55fe4e4a98e8eb01da59d365 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:58:08 UTC) from {username='slave'} at 127.0.0.1:59068
W20260130 02:58:26.101922 18860 catalog_manager.cc:4952] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): delete failed for tablet ec28ff12927d450ead16d26cf016138d because the tablet was not found. No further retry: Not found: Tablet not found: ec28ff12927d450ead16d26cf016138d
W20260130 02:58:26.103335 18860 catalog_manager.cc:4952] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): delete failed for tablet a8a5038e9def44eb8b3b1e4aeae25652 because the tablet was not found. No further retry: Not found: Tablet not found: a8a5038e9def44eb8b3b1e4aeae25652
W20260130 02:58:26.105273 18860 catalog_manager.cc:4952] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): delete failed for tablet 78e9bc3e55fe4e4a98e8eb01da59d365 because the tablet was not found. No further retry: Not found: Tablet not found: 78e9bc3e55fe4e4a98e8eb01da59d365
I20260130 02:58:26.107610 19501 tablet_service.cc:1552] Processing DeleteTablet for tablet e2aaaf00f53d418b87d5c7af286e4d84 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:58:08 UTC) from {username='slave'} at 127.0.0.1:59068
I20260130 02:58:26.108341 19496 tablet_service.cc:1552] Processing DeleteTablet for tablet c7c621b99deb457998c44ab1644d3939 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:58:16 UTC) from {username='slave'} at 127.0.0.1:59068
I20260130 02:58:26.109186 21276 tablet_replica.cc:333] T e2aaaf00f53d418b87d5c7af286e4d84 P d870164263bb44b6b780a557daea3656: stopping tablet replica
I20260130 02:58:26.110028 21276 raft_consensus.cc:2243] T e2aaaf00f53d418b87d5c7af286e4d84 P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260130 02:58:26.111063 21141 leader_election.cc:290] T 532e867fd5944d468d7672d2a5c92f3c P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 election: Requested vote from peers d870164263bb44b6b780a557daea3656 (127.18.100.67:39571), 371a36d219164974abdf848841528152 (127.18.100.65:42021)
I20260130 02:58:26.112057 19224 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6d1f121f49cd47c2b8247daaf6874e01" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152"
I20260130 02:58:26.114544 21276 raft_consensus.cc:2272] T e2aaaf00f53d418b87d5c7af286e4d84 P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260130 02:58:26.120030 19807 tablet_bootstrap.cc:654] T 2b8034a92b2043e5b57cf02f1ee6296d P 371a36d219164974abdf848841528152: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:26.120674 18870 rpcz_store.cc:275] Call kudu.master.MasterService.GetMasterRegistration from 127.0.0.1:35042 (request call id 1) took 3148 ms. Trace:
I20260130 02:58:26.120999 18870 rpcz_store.cc:276] 0130 02:58:22.972367 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:58:25.822340 (+2849973us) service_pool.cc:225] Handling call
0130 02:58:26.120410 (+298070us) spinlock_profiling.cc:243] Waited 260 ms on lock 0x7b5400030554. stack: 00007fb0bd3f7373 00007fb0bd3f7270 00007fb0bcf6c574 0000000000508744 00000000005086d9 00007fb0c44485a1 00007fb0c44e7939 00007fb0c61bd86c 00007fb0c62e4207 00007fb0c3f34327 00007fb0c3f342b2 00007fb0c3f341e1 00007fb0c3f3415c 00007fb0c3f33412 00007fb0c21dd34c 00007fb0c21dc786
0130 02:58:26.120598 (+ 188us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":598420352}
I20260130 02:58:25.814435 21200 raft_consensus.cc:2749] T cd30d7adcc4545848e3501a205a10293 P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260130 02:58:26.123731 21276 ts_tablet_manager.cc:1916] T e2aaaf00f53d418b87d5c7af286e4d84 P d870164263bb44b6b780a557daea3656: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260130 02:58:26.114809 19505 tablet_service.cc:1552] Processing DeleteTablet for tablet 8e57ae31ab3941f789ac69627384e3b0 with delete_type TABLET_DATA_DELETED (replaced by tablet 8b47653c7c3945a68e9bf4680bfc0496) from {username='slave'} at 127.0.0.1:59068
I20260130 02:58:26.132756 21080 catalog_manager.cc:1612] T 00000000000000000000000000000000 P 3749baa287ed4f249329a74071781edd: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260130 02:58:26.137984 21211 consensus_queue.cc:1048] T cfd12a72726846e587cff9d6a4d61bd6 P d870164263bb44b6b780a557daea3656 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
E20260130 02:58:25.970927 18873 catalog_manager.cc:5809] Error updating tablets from d870164263bb44b6b780a557daea3656: Illegal state: replica 658299197a424f53a37f14fadf089ae4 is not leader of this config: current role FOLLOWER. Tablet report was: is_incremental: true updated_tablets { tablet_id: "1fc8f09d6329407c8078542b4123ef4d" state: RUNNING consensus_state { current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "0855d88806914a36b4bf6b49036f0ad0" state: RUNNING consensus_state { current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "ab6afa40408a4f458495abf20290e55b" state: RUNNING consensus_state { current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "eb7a755ab62b4552bb8588264c5f7b75" state: RUNNING consensus_state { current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "2b0edaad6877460cb7982efc62b00b66" state: RUNNING consensus_state { current_term: 1 leader_uuid: "d870164263bb44b6b780a557daea3656" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } health_report { overall_health: UNKNOWN } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY stats { } } updated_tablets { tablet_id: "d480bf18140245e3870a62582d5a8318" state: RUNNING consensus_state { current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "49efcfced0b74c48962393149ac2a500" state: RUNNING consensus_state { current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "a3258852aed24ed582a79f77860e6f0d" state: RUNNING consensus_state { current_term: 1 leader_uuid: "d870164263bb44b6b780a557daea3656" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } health_report { overall_health: UNKNOWN } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY stats { } } updated_tablets { tablet_id: "133e9bfe175f44aab1a53114abe67e4d" state: RUNNING consensus_state { current_term: 1 leader_uuid: "d870164263bb44b6b780a557daea3656" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } health_report { overall_health: UNKNOWN } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY stats { } } updated_tablets { tablet_id: "320dbe98a32a43b9899bbeed80b20f55" state: RUNNING consensus_state { current_term: 1 leader_uuid: "371a36d219164974abdf848841528152" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "12c3f3ec523040e7b7cd356e3da57e0d" state: RUNNING consensus_state { current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "13f0ace4d9de43abab08eca32cdb30c1" state: RUNNING consensus_state { current_term: 1 leader_uuid: "371a36d219164974abdf848841528152" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "e12d130bcdf34965888753f12c1ff848" state: STOPPED consensus_state { current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_DELETED } updated_tablets { tablet_id: "3aa76993beb94b12bc706a76db26a03b" state: RUNNING consensus_state { current_term: 1 leader_uuid: "d870164263bb44b6b780a557daea3656" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } health_report { overall_health: HEALTHY } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY stats { } } updated_tablets { tablet_id: "838e49e5e8874db393bdc15a1c92f87e" state: RUNNING consensus_state { current_term: 1 leader_uuid: "371a36d219164974abdf848841528152" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "f9e5f01e69254d658c369f3e21cef523" state: RUNNING consensus_state { current_term: 1 leader_uuid: "d870164263bb44b6b780a557daea3656" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } health_report { overall_health: HEALTHY } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY stats { on_disk_size: 27325 live_row_count: 0 } } updated_tablets { tablet_id: "a90712651a604b83ba187f893ab17711" state: RUNNING consensus_state { current_term: 1 leader_uuid: "371a36d219164974abdf848841528152" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "aae68390dbbe4dc58ce3d3bd01725ae6" state: RUNNING consensus_state { current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "2c79a3aecab844698e70834999e1caac" state: RUNNING consensus_state { current_term: 1 leader_uuid: "d870164263bb44b6b780a557daea3656" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } health_report { overall_health: UNKNOWN } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY stats { } } updated_tablets { tablet_id: "f41f4cff7dd4489f948700a45e9a7057" state: RUNNING consensus_state { current_term: 1 leader_uuid: "d870164263bb44b6b780a557daea3656" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } health_report { overall_health: HEALTHY } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY stats { on_disk_size: 27356 live_row_count: 0 } } updated_tablets { tablet_id: "341041cb0be2456e88403367c0814704" state: RUNNING consensus_state { current_term: 1 leader_uuid: "d870164263bb44b6b780a557daea3656" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } health_report { overall_health: HEALTHY } } } } schema_version: 1 tablet_data_state: TABLET_DATA_READY stats { on_disk_size: 27578 live_row_count: 0 } } updated_tablets { tablet_id: "5fbe92a2227d44d88b4531a7f8aa0e08" state: RUNNING consensus_state { current_term: 1 leader_uuid: "371a36d219164974abdf848841528152" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "03025d13f0e545cfa6288677a2715343" state: RUNNING consensus_state { current_term: 1 leader_uuid: "d870164263bb44b6b780a557daea3656" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } health_report { overall_health: HEALTHY } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY stats { on_disk_size: 27356 live_row_count: 0 } } updated_tablets { tablet_id: "9fe0448bcc4a4346b3c67fc3f626ed3c" state: RUNNING consensus_state { current_term: 1 leader_uuid: "78c2c46c7520492db35d51c244fb4f74" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } } } schema_version: 1 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "4907294442c145c6b03eb1de057c66a8" state: RUNNING consensus_state { current_term: 1 leader_uuid: "78c2c46c7520492db35d51c244fb4f74" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "ba91a508b1cb4430a4d8ca26946aecc8" state: RUNNING consensus_state { current_term: 1 leader_uuid: "371a36d219164974abdf848841528152" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "f7607df5efd543438fb2dff3667cc8c8" state: RUNNING consensus_state { current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "4c139aa8cd184f488cec4e05ef7dcada" state: RUNNING consensus_state { current_term: 1 leader_uuid: "78c2c46c7520492db35d51c244fb4f74" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } } } schema_version: 1 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "43cec19c51ec4f05b63800c07f9534ef" state: RUNNING consensus_state { current_term: 1 leader_uuid: "78c2c46c7520492db35d51c244fb4f74" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "657e9b0d8c05413f904129dce745710b" state: RUNNING consensus_state { current_term: 1 leader_uuid: "d870164263bb44b6b780a557daea3656" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } health_report { overall_health: HEALTHY } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY stats { } } updated_tablets { tablet_id: "7caf7cbad5054396b5617094e4d536a0" state: RUNNING consensus_state { current_term: 1 leader_uuid: "d870164263bb44b6b780a557daea3656" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } health_report { overall_health: HEALTHY } } } } schema_version: 1 tablet_data_state: TABLET_DATA_READY stats { on_disk_size: 27578 live_row_count: 0 } } updated_tablets { tablet_id: "f2d01bfdc9c4494386bd02bed655fe63" state: RUNNING consensus_state { current_term: 1 leader_uuid: "d870164263bb44b6b780a557daea3656" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } health_report { overall_health: HEALTHY } } } } schema_version: 3 tablet_data_state: TABLET_DATA_READY stats { on_disk_size: 28022 live_row_count: 0 } } updated_tablets { tablet_id: "16691c9315e94253bb9b3fb5058e5636" state: RUNNING consensus_state { current_term: 1 leader_uuid: "78c2c46c7520492db35d51c244fb4f74" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } } } schema_version: 1 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "753344737b5941d9a93b6779e72f63d4" state: RUNNING consensus_state { current_term: 1 leader_uuid: "78c2c46c7520492db35d51c244fb4f74" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "25fd48e693424537b91ba7c3fcc96878" state: RUNNING consensus_state { current_term: 1 leader_uuid: "371a36d219164974abdf848841528152" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "3ae5472cba1e4613abe4cda42ccb2496" state: RUNNING consensus_state { current_term: 1 leader_uuid: "371a36d219164974abdf848841528152" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "6218f2f7d73a42428b57a45858f53afd" state: RUNNING consensus_state { current_term: 1 leader_uuid: "371a36d219164974abdf848841528152" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "ee7c8ff92ca9465fa18be6eae63d78ca" state: RUNNING consensus_state { current_term: 1 leader_uuid: "78c2c46c7520492db35d51c244fb4f74" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } } } schema_version: 1 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "e2aaaf00f53d418b87d5c7af286e4d84" state: RUNNING consensus_state { current_term: 1 leader_uuid: "371a36d219164974abdf848841528152" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "274530359b174ff79f046c2d4a6afa76" state: RUNNING consensus_state { current_term: 1 leader_uuid: "371a36d219164974abdf848841528152" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "3593f8b89da24ff9b6cf549d5ac58bc6" state: RUNNING consensus_state { current_term: 1 leader_uuid: "78c2c46c7520492db35d51c244fb4f74" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "7783db3568f4434e91e9e456799cc312" state: RUNNING consensus_state { current_term: 1 leader_uuid: "371a36d219164974abdf848841528152" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66"
I20260130 02:58:26.173461 21275 tablet_replica.cc:333] T e47b22aafe5247c4b62c4193c24f427c P 78c2c46c7520492db35d51c244fb4f74: stopping tablet replica
W20260130 02:58:26.181145 18860 catalog_manager.cc:5058] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767): alter failed for tablet 8e57ae31ab3941f789ac69627384e3b0 (table default.Table_1ae68ae3f0404843aca3c72f02678ac6 [id=8a9182f49e18489fafec8dca190c1b2f]),no further retry: Not found: Tablet not found: 8e57ae31ab3941f789ac69627384e3b0
I20260130 02:58:26.191371 19505 tablet_service.cc:1552] Processing DeleteTablet for tablet 04f4ac27800c4ac3a79ae12d35479757 with delete_type TABLET_DATA_DELETED (replaced by tablet 3738385130c044e0a5a5803da74c567d) from {username='slave'} at 127.0.0.1:59068
I20260130 02:58:26.216641 19523 raft_consensus.cc:2468] T 532e867fd5944d468d7672d2a5c92f3c P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78c2c46c7520492db35d51c244fb4f74 in term 1.
I20260130 02:58:26.224692 19625 tablet_bootstrap.cc:492] T e691c238199b4914bcc3fd248f31420c P d870164263bb44b6b780a557daea3656: No bootstrap required, opened a new log
I20260130 02:58:26.225131 19625 ts_tablet_manager.cc:1403] T e691c238199b4914bcc3fd248f31420c P d870164263bb44b6b780a557daea3656: Time spent bootstrapping tablet: real 0.339s user 0.008s sys 0.000s
I20260130 02:58:26.230124 19505 tablet_service.cc:1552] Processing DeleteTablet for tablet a8a5038e9def44eb8b3b1e4aeae25652 with delete_type TABLET_DATA_DELETED (replaced by tablet 475b435b2967418f927fa172d2ff04ea) from {username='slave'} at 127.0.0.1:59068
I20260130 02:58:26.230155 19221 raft_consensus.cc:3060] T c76f0dc7762c49c389fa6fef41f02141 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:26.232178 19313 leader_election.cc:304] T 532e867fd5944d468d7672d2a5c92f3c P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78c2c46c7520492db35d51c244fb4f74, d870164263bb44b6b780a557daea3656; no voters:
I20260130 02:58:26.230471 19625 raft_consensus.cc:359] T e691c238199b4914bcc3fd248f31420c P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:26.232846 19625 raft_consensus.cc:385] T e691c238199b4914bcc3fd248f31420c P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:26.233096 19625 raft_consensus.cc:740] T e691c238199b4914bcc3fd248f31420c P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d870164263bb44b6b780a557daea3656, State: Initialized, Role: FOLLOWER
I20260130 02:58:26.234186 19217 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "532e867fd5944d468d7672d2a5c92f3c" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152" is_pre_election: true
W20260130 02:58:26.237834 18872 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.18.100.65:51021 (request call id 39) took 8054 ms (client timeout 1000 ms). Trace:
I20260130 02:58:26.237522 19625 consensus_queue.cc:260] T e691c238199b4914bcc3fd248f31420c P d870164263bb44b6b780a557daea3656 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:26.241472 19505 tablet_service.cc:1552] Processing DeleteTablet for tablet 78e9bc3e55fe4e4a98e8eb01da59d365 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:58:08 UTC) from {username='slave'} at 127.0.0.1:59068
W20260130 02:58:26.243170 18874 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.18.100.67:47213 (request call id 32) took 8739 ms (client timeout 1000 ms). Trace:
W20260130 02:58:26.243497 18874 rpcz_store.cc:269] 0130 02:58:17.503191 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:58:18.372379 (+869188us) service_pool.cc:225] Handling call
0130 02:58:26.242516 (+7870137us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":232472,"reported_tablets":51}
I20260130 02:58:26.248135 19625 ts_tablet_manager.cc:1434] T e691c238199b4914bcc3fd248f31420c P d870164263bb44b6b780a557daea3656: Time spent starting tablet: real 0.023s user 0.006s sys 0.000s
I20260130 02:58:26.255424 21282 tablet_replica.cc:333] T e47b22aafe5247c4b62c4193c24f427c P 371a36d219164974abdf848841528152: stopping tablet replica
I20260130 02:58:26.256410 21282 raft_consensus.cc:2243] T e47b22aafe5247c4b62c4193c24f427c P 371a36d219164974abdf848841528152 [term 1 LEADER]: Raft consensus shutting down.
I20260130 02:58:26.257309 19807 tablet_bootstrap.cc:492] T 2b8034a92b2043e5b57cf02f1ee6296d P 371a36d219164974abdf848841528152: No bootstrap required, opened a new log
I20260130 02:58:26.257787 19807 ts_tablet_manager.cc:1403] T 2b8034a92b2043e5b57cf02f1ee6296d P 371a36d219164974abdf848841528152: Time spent bootstrapping tablet: real 0.200s user 0.011s sys 0.016s
I20260130 02:58:26.261874 21164 catalog_manager.cc:6038] T 00000000000000000000000000000000 P 107f274cdda543038832f06a56ab9130: Loaded TSK: 0
I20260130 02:58:26.263053 21285 tablet.cc:1722] T b1e8b78eff0d4a0b92059adb2de4a3ce P d870164263bb44b6b780a557daea3656: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20260130 02:58:26.264521 18873 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.18.100.67:47213 (request call id 37) took 3027 ms (client timeout 1000 ms). Trace:
W20260130 02:58:26.264883 18873 rpcz_store.cc:269] 0130 02:58:23.237113 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:58:23.534665 (+297552us) service_pool.cc:225] Handling call
0130 02:58:25.970645 (+2435980us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:58:25.970673 (+ 28us) write_op.cc:454] Released schema lock
0130 02:58:26.264445 (+293772us) inbound_call.cc:173] Queueing failure response
Related trace 'op':
0130 02:58:25.957943 (+ 0us) op_driver.cc:389] HandleFailure(Illegal state: replica 658299197a424f53a37f14fadf089ae4 is not leader of this config: current role FOLLOWER)
0130 02:58:25.957975 (+ 32us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:58:25.957995 (+ 20us) write_op.cc:454] Released schema lock
0130 02:58:25.958014 (+ 19us) write_op.cc:335] FINISH: Op aborted
Metrics: {"reported_tablets":71,"child_traces":[["op",{}]]}
I20260130 02:58:26.268103 21288 raft_consensus.cc:2804] T 532e867fd5944d468d7672d2a5c92f3c P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Leader election won for term 1
I20260130 02:58:26.269102 21288 raft_consensus.cc:697] T 532e867fd5944d468d7672d2a5c92f3c P 78c2c46c7520492db35d51c244fb4f74 [term 1 LEADER]: Becoming Leader. State: Replica: 78c2c46c7520492db35d51c244fb4f74, State: Running, Role: LEADER
I20260130 02:58:26.269254 21240 raft_consensus.cc:493] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:26.269680 21240 raft_consensus.cc:515] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "658299197a424f53a37f14fadf089ae4" member_type: VOTER last_known_addr { host: "127.18.100.126" port: 38191 } } peers { permanent_uuid: "107f274cdda543038832f06a56ab9130" member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39711 } } peers { permanent_uuid: "3749baa287ed4f249329a74071781edd" member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34629 } }
I20260130 02:58:26.269876 21288 consensus_queue.cc:237] T 532e867fd5944d468d7672d2a5c92f3c P 78c2c46c7520492db35d51c244fb4f74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:26.272156 21240 leader_election.cc:290] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 107f274cdda543038832f06a56ab9130 (127.18.100.125:39711), 3749baa287ed4f249329a74071781edd (127.18.100.124:34629)
I20260130 02:58:26.272235 18960 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "658299197a424f53a37f14fadf089ae4" candidate_term: 6 candidate_status { last_received { term: 4 index: 373 } } ignore_live_leader: false dest_uuid: "107f274cdda543038832f06a56ab9130" is_pre_election: true
I20260130 02:58:26.273890 20807 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "658299197a424f53a37f14fadf089ae4" candidate_term: 6 candidate_status { last_received { term: 4 index: 373 } } ignore_live_leader: false dest_uuid: "3749baa287ed4f249329a74071781edd" is_pre_election: true
I20260130 02:58:26.274050 19224 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "532e867fd5944d468d7672d2a5c92f3c" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152"
I20260130 02:58:26.274468 20807 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 3749baa287ed4f249329a74071781edd [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 658299197a424f53a37f14fadf089ae4 in term 5.
W20260130 02:58:26.275465 18869 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.18.100.67:47213 (request call id 33) took 7469 ms (client timeout 1000 ms). Trace:
I20260130 02:58:26.275725 18860 leader_election.cc:304] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3749baa287ed4f249329a74071781edd, 658299197a424f53a37f14fadf089ae4; no voters:
W20260130 02:58:26.276962 19313 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.100.67:39571, user_credentials={real_user=slave}} blocked reactor thread for 44862.9us
I20260130 02:58:26.278759 21240 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260130 02:58:26.279162 21240 raft_consensus.cc:493] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260130 02:58:26.279624 21240 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4 [term 5 FOLLOWER]: Advancing to term 6
I20260130 02:58:26.295423 19625 tablet_bootstrap.cc:492] T 8b47653c7c3945a68e9bf4680bfc0496 P d870164263bb44b6b780a557daea3656: Bootstrap starting.
I20260130 02:58:26.303449 19505 tablet_service.cc:1552] Processing DeleteTablet for tablet e12d130bcdf34965888753f12c1ff848 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:58:18 UTC) from {username='slave'} at 127.0.0.1:59068
I20260130 02:58:26.304198 21282 raft_consensus.cc:2272] T e47b22aafe5247c4b62c4193c24f427c P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260130 02:58:26.308215 19840 tablet_bootstrap.cc:492] T 475b435b2967418f927fa172d2ff04ea P 78c2c46c7520492db35d51c244fb4f74: No bootstrap required, opened a new log
I20260130 02:58:26.314967 21215 consensus_queue.cc:1048] T 39bcc41f10ae4241b370529364454586 P d870164263bb44b6b780a557daea3656 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260130 02:58:26.305918 19159 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.100.67:39571, user_credentials={real_user=slave}} blocked reactor thread for 147852us
W20260130 02:58:26.275784 18869 rpcz_store.cc:269] 0130 02:58:18.805978 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:58:18.933900 (+127922us) service_pool.cc:225] Handling call
0130 02:58:26.275377 (+7341477us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0130 02:58:21.358673 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0130 02:58:21.359133 (+ 460us) write_op.cc:432] Acquiring schema lock in shared mode
0130 02:58:21.359161 (+ 28us) write_op.cc:435] Acquired schema lock
0130 02:58:21.359178 (+ 17us) tablet.cc:660] Decoding operations
0130 02:58:21.359466 (+ 288us) write_op.cc:620] Acquiring the partition lock for write op
0130 02:58:21.359524 (+ 58us) write_op.cc:641] Partition lock acquired for write op
0130 02:58:21.359552 (+ 28us) tablet.cc:683] Acquiring locks for 1 operations
0130 02:58:21.359712 (+ 160us) tablet.cc:699] Row locks acquired
0130 02:58:21.359727 (+ 15us) write_op.cc:260] PREPARE: finished
0130 02:58:21.359850 (+ 123us) write_op.cc:270] Start()
0130 02:58:21.359935 (+ 85us) write_op.cc:276] Timestamp: P: 1769741901359813 usec, L: 0
0130 02:58:21.359952 (+ 17us) op_driver.cc:348] REPLICATION: starting
0130 02:58:21.360502 (+ 550us) log.cc:844] Serialized 533 byte log entry
0130 02:58:21.447803 (+ 87301us) op_driver.cc:464] REPLICATION: finished
0130 02:58:21.492105 (+ 44302us) write_op.cc:301] APPLY: starting
0130 02:58:21.492160 (+ 55us) tablet.cc:1365] starting BulkCheckPresence
0130 02:58:21.492241 (+ 81us) tablet.cc:1368] finished BulkCheckPresence
0130 02:58:21.492348 (+ 107us) tablet.cc:1370] starting ApplyRowOperation cycle
0130 02:58:21.492537 (+ 189us) tablet.cc:1381] finished ApplyRowOperation cycle
0130 02:58:21.492685 (+ 148us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0130 02:58:21.492730 (+ 45us) write_op.cc:312] APPLY: finished
0130 02:58:21.492931 (+ 201us) log.cc:844] Serialized 23 byte log entry
0130 02:58:21.493080 (+ 149us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:58:21.493137 (+ 57us) write_op.cc:454] Released schema lock
0130 02:58:21.493171 (+ 34us) write_op.cc:341] FINISH: Updating metrics
0130 02:58:21.544670 (+ 51499us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:58:21.544698 (+ 28us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":65206,"reported_tablets":54,"child_traces":[["op",{"apply.queue_time_us":44118,"apply.run_cpu_time_us":1333,"apply.run_wall_time_us":52435,"num_ops":1,"prepare.queue_time_us":2560,"prepare.run_cpu_time_us":2198,"prepare.run_wall_time_us":16643,"raft.queue_time_us":85,"raft.run_cpu_time_us":663,"raft.run_wall_time_us":662,"replication_time_us":87723,"spinlock_wait_cycles":67840,"thread_start_us":91026,"threads_started":2}]]}
I20260130 02:58:26.331064 21240 raft_consensus.cc:515] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "658299197a424f53a37f14fadf089ae4" member_type: VOTER last_known_addr { host: "127.18.100.126" port: 38191 } } peers { permanent_uuid: "107f274cdda543038832f06a56ab9130" member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39711 } } peers { permanent_uuid: "3749baa287ed4f249329a74071781edd" member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34629 } }
W20260130 02:58:26.331784 18867 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.18.100.67:47213 (request call id 34) took 6425 ms (client timeout 1000 ms). Trace:
W20260130 02:58:26.336171 18867 rpcz_store.cc:269] 0130 02:58:19.906800 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:58:19.933047 (+ 26247us) service_pool.cc:225] Handling call
0130 02:58:25.965538 (+6032491us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:58:25.965558 (+ 20us) write_op.cc:454] Released schema lock
0130 02:58:26.331718 (+366160us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0130 02:58:21.842471 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0130 02:58:21.842986 (+ 515us) write_op.cc:432] Acquiring schema lock in shared mode
0130 02:58:21.843013 (+ 27us) write_op.cc:435] Acquired schema lock
0130 02:58:21.843029 (+ 16us) tablet.cc:660] Decoding operations
0130 02:58:21.843390 (+ 361us) write_op.cc:620] Acquiring the partition lock for write op
0130 02:58:21.843439 (+ 49us) write_op.cc:641] Partition lock acquired for write op
0130 02:58:21.843464 (+ 25us) tablet.cc:683] Acquiring locks for 2 operations
0130 02:58:21.843745 (+ 281us) tablet.cc:699] Row locks acquired
0130 02:58:21.843761 (+ 16us) write_op.cc:260] PREPARE: finished
0130 02:58:21.843888 (+ 127us) write_op.cc:270] Start()
0130 02:58:21.843971 (+ 83us) write_op.cc:276] Timestamp: P: 1769741901843838 usec, L: 0
0130 02:58:21.843988 (+ 17us) op_driver.cc:348] REPLICATION: starting
0130 02:58:21.844499 (+ 511us) log.cc:844] Serialized 913 byte log entry
0130 02:58:21.853960 (+ 9461us) op_driver.cc:464] REPLICATION: finished
0130 02:58:21.854293 (+ 333us) write_op.cc:301] APPLY: starting
0130 02:58:21.854343 (+ 50us) tablet.cc:1365] starting BulkCheckPresence
0130 02:58:21.854448 (+ 105us) tablet.cc:1368] finished BulkCheckPresence
0130 02:58:21.854540 (+ 92us) tablet.cc:1370] starting ApplyRowOperation cycle
0130 02:58:21.854875 (+ 335us) tablet.cc:1381] finished ApplyRowOperation cycle
0130 02:58:21.855197 (+ 322us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0130 02:58:21.855244 (+ 47us) write_op.cc:312] APPLY: finished
0130 02:58:21.855426 (+ 182us) log.cc:844] Serialized 29 byte log entry
0130 02:58:21.855588 (+ 162us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:58:21.855652 (+ 64us) write_op.cc:454] Released schema lock
0130 02:58:21.855686 (+ 34us) write_op.cc:341] FINISH: Updating metrics
0130 02:58:21.855887 (+ 201us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:58:21.855910 (+ 23us) write_op.cc:454] Released schema lock
Related trace 'op':
0130 02:58:25.965353 (+ 0us) op_driver.cc:389] HandleFailure(Illegal state: replica 658299197a424f53a37f14fadf089ae4 is not leader of this config: current role FOLLOWER)
0130 02:58:25.965383 (+ 30us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:58:25.965413 (+ 30us) write_op.cc:454] Released schema lock
0130 02:58:25.965432 (+ 19us) write_op.cc:335] FINISH: Op aborted
Metrics: {"mutex_wait_us":29499,"reported_tablets":55,"child_traces":[["op",{"apply.queue_time_us":196,"apply.run_cpu_time_us":1403,"apply.run_wall_time_us":1562,"num_ops":2,"prepare.queue_time_us":102,"prepare.run_cpu_time_us":2315,"prepare.run_wall_time_us":2310,"raft.queue_time_us":81,"raft.run_cpu_time_us":646,"raft.run_wall_time_us":646,"replication_time_us":9876,"spinlock_wait_cycles":64896}],["op",{}]]}
I20260130 02:58:26.340906 18885 raft_consensus.cc:1240] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4 [term 6 FOLLOWER]: Rejecting Update request from peer 107f274cdda543038832f06a56ab9130 for earlier term 5. Current term is 6. Ops: []
W20260130 02:58:26.238152 18872 rpcz_store.cc:269] 0130 02:58:18.183605 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:58:18.472311 (+288706us) service_pool.cc:225] Handling call
0130 02:58:25.939499 (+7467188us) spinlock_profiling.cc:243] Waited 2.21 ms on lock 0x7b6800034d60. stack: 00007fb0bd3f7373 00007fb0bd3f7270 00007fb0bcf6c574 0000000000508744 00000000005086d9 00007fb0c6203c91 00007fb0c61e102a 00007fb0c61df768 00007fb0c61ddda0 00007fb0c62e19ce 00007fb0c3efe4a4 00007fb0c3efe432 00007fb0c3efe361 00007fb0c3efe2dc 00007fb0c3efd592 00007fb0c21dd34c
0130 02:58:26.237680 (+298181us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0130 02:58:19.642047 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0130 02:58:19.642421 (+ 374us) write_op.cc:432] Acquiring schema lock in shared mode
0130 02:58:19.642445 (+ 24us) write_op.cc:435] Acquired schema lock
0130 02:58:19.642456 (+ 11us) tablet.cc:660] Decoding operations
0130 02:58:19.643788 (+ 1332us) write_op.cc:620] Acquiring the partition lock for write op
0130 02:58:19.643822 (+ 34us) write_op.cc:641] Partition lock acquired for write op
0130 02:58:19.643843 (+ 21us) tablet.cc:683] Acquiring locks for 2 operations
0130 02:58:19.644094 (+ 251us) tablet.cc:699] Row locks acquired
0130 02:58:19.644108 (+ 14us) write_op.cc:260] PREPARE: finished
0130 02:58:19.644226 (+ 118us) write_op.cc:270] Start()
0130 02:58:19.644302 (+ 76us) write_op.cc:276] Timestamp: P: 1769741899644125 usec, L: 0
0130 02:58:19.644317 (+ 15us) op_driver.cc:348] REPLICATION: starting
0130 02:58:19.644767 (+ 450us) log.cc:844] Serialized 913 byte log entry
0130 02:58:19.660106 (+ 15339us) op_driver.cc:464] REPLICATION: finished
0130 02:58:19.663961 (+ 3855us) write_op.cc:301] APPLY: starting
0130 02:58:19.664011 (+ 50us) tablet.cc:1365] starting BulkCheckPresence
0130 02:58:19.664110 (+ 99us) tablet.cc:1368] finished BulkCheckPresence
0130 02:58:19.664191 (+ 81us) tablet.cc:1370] starting ApplyRowOperation cycle
0130 02:58:19.664420 (+ 229us) tablet.cc:1381] finished ApplyRowOperation cycle
0130 02:58:19.664567 (+ 147us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0130 02:58:19.664612 (+ 45us) write_op.cc:312] APPLY: finished
0130 02:58:19.664778 (+ 166us) log.cc:844] Serialized 29 byte log entry
0130 02:58:19.673193 (+ 8415us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:58:19.673261 (+ 68us) write_op.cc:454] Released schema lock
0130 02:58:19.673291 (+ 30us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"mutex_wait_us":298215,"reported_tablets":55,"spinlock_wait_cycles":5080832,"child_traces":[["op",{"apply.queue_time_us":3738,"apply.run_cpu_time_us":1269,"apply.run_wall_time_us":9520,"mutex_wait_us":31924,"num_ops":2,"prepare.queue_time_us":34928,"prepare.run_cpu_time_us":1966,"prepare.run_wall_time_us":4181,"replication_time_us":15718,"thread_start_us":60739,"threads_started":1}]]}
I20260130 02:58:26.344509 21246 consensus_queue.cc:1059] T 00000000000000000000000000000000 P 107f274cdda543038832f06a56ab9130 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "658299197a424f53a37f14fadf089ae4" member_type: VOTER last_known_addr { host: "127.18.100.126" port: 38191 }, Status: INVALID_TERM, Last received: 0.0, Next index: 373, Last known committed idx: 372, Time since last communication: 0.000s
I20260130 02:58:26.345244 21246 consensus_queue.cc:1243] T 00000000000000000000000000000000 P 107f274cdda543038832f06a56ab9130 [LEADER]: Peer 658299197a424f53a37f14fadf089ae4 log is divergent from this leader: its last log entry 4.373 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 372
I20260130 02:58:26.345654 21246 raft_consensus.cc:3055] T 00000000000000000000000000000000 P 107f274cdda543038832f06a56ab9130 [term 5 LEADER]: Stepping down as leader of term 5
I20260130 02:58:26.345934 21246 raft_consensus.cc:740] T 00000000000000000000000000000000 P 107f274cdda543038832f06a56ab9130 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 107f274cdda543038832f06a56ab9130, State: Running, Role: LEADER
I20260130 02:58:26.346657 21246 consensus_queue.cc:260] T 00000000000000000000000000000000 P 107f274cdda543038832f06a56ab9130 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 372, Committed index: 372, Last appended: 5.373, Last appended by leader: 373, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "658299197a424f53a37f14fadf089ae4" member_type: VOTER last_known_addr { host: "127.18.100.126" port: 38191 } } peers { permanent_uuid: "107f274cdda543038832f06a56ab9130" member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39711 } } peers { permanent_uuid: "3749baa287ed4f249329a74071781edd" member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34629 } }
I20260130 02:58:26.347669 21246 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 107f274cdda543038832f06a56ab9130 [term 5 FOLLOWER]: Advancing to term 6
I20260130 02:58:26.351689 21167 raft_consensus.cc:493] T 3738385130c044e0a5a5803da74c567d P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:26.352136 21167 raft_consensus.cc:515] T 3738385130c044e0a5a5803da74c567d P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:26.352944 21275 raft_consensus.cc:2243] T e47b22aafe5247c4b62c4193c24f427c P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260130 02:58:26.353941 21167 leader_election.cc:290] T 3738385130c044e0a5a5803da74c567d P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767)
I20260130 02:58:26.354080 19840 ts_tablet_manager.cc:1403] T 475b435b2967418f927fa172d2ff04ea P 78c2c46c7520492db35d51c244fb4f74: Time spent bootstrapping tablet: real 0.592s user 0.003s sys 0.008s
I20260130 02:58:26.354148 21275 raft_consensus.cc:2272] T e47b22aafe5247c4b62c4193c24f427c P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260130 02:58:26.356380 21275 ts_tablet_manager.cc:1916] T e47b22aafe5247c4b62c4193c24f427c P 78c2c46c7520492db35d51c244fb4f74: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260130 02:58:26.325181 19807 raft_consensus.cc:359] T 2b8034a92b2043e5b57cf02f1ee6296d P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:26.356945 19807 raft_consensus.cc:385] T 2b8034a92b2043e5b57cf02f1ee6296d P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:26.356748 19840 raft_consensus.cc:359] T 475b435b2967418f927fa172d2ff04ea P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:26.357182 19807 raft_consensus.cc:740] T 2b8034a92b2043e5b57cf02f1ee6296d P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 371a36d219164974abdf848841528152, State: Initialized, Role: FOLLOWER
I20260130 02:58:26.357342 19840 raft_consensus.cc:385] T 475b435b2967418f927fa172d2ff04ea P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:26.357591 19840 raft_consensus.cc:740] T 475b435b2967418f927fa172d2ff04ea P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78c2c46c7520492db35d51c244fb4f74, State: Initialized, Role: FOLLOWER
I20260130 02:58:26.358150 19840 consensus_queue.cc:260] T 475b435b2967418f927fa172d2ff04ea P 78c2c46c7520492db35d51c244fb4f74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:26.341183 21197 raft_consensus.cc:493] T c76f0dc7762c49c389fa6fef41f02141 P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:26.360129 19840 ts_tablet_manager.cc:1434] T 475b435b2967418f927fa172d2ff04ea P 78c2c46c7520492db35d51c244fb4f74: Time spent starting tablet: real 0.006s user 0.001s sys 0.003s
I20260130 02:58:26.360980 19840 tablet_bootstrap.cc:492] T 3738385130c044e0a5a5803da74c567d P 78c2c46c7520492db35d51c244fb4f74: Bootstrap starting.
I20260130 02:58:26.359881 21197 raft_consensus.cc:515] T c76f0dc7762c49c389fa6fef41f02141 P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:26.362739 21197 leader_election.cc:290] T c76f0dc7762c49c389fa6fef41f02141 P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d870164263bb44b6b780a557daea3656 (127.18.100.67:39571), 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767)
I20260130 02:58:26.363157 21167 raft_consensus.cc:493] T e691c238199b4914bcc3fd248f31420c P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:26.363512 21164 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260130 02:58:26.363569 21167 raft_consensus.cc:515] T e691c238199b4914bcc3fd248f31420c P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:26.365636 21167 leader_election.cc:290] T e691c238199b4914bcc3fd248f31420c P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767)
I20260130 02:58:26.368162 18960 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "658299197a424f53a37f14fadf089ae4" candidate_term: 6 candidate_status { last_received { term: 4 index: 373 } } ignore_live_leader: false dest_uuid: "107f274cdda543038832f06a56ab9130"
I20260130 02:58:26.368983 18960 raft_consensus.cc:2410] T 00000000000000000000000000000000 P 107f274cdda543038832f06a56ab9130 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 658299197a424f53a37f14fadf089ae4 for term 6 because replica has last-logged OpId of term: 5 index: 373, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 373.
I20260130 02:58:26.372611 20807 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "658299197a424f53a37f14fadf089ae4" candidate_term: 6 candidate_status { last_received { term: 4 index: 373 } } ignore_live_leader: false dest_uuid: "3749baa287ed4f249329a74071781edd"
I20260130 02:58:26.373147 20807 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 3749baa287ed4f249329a74071781edd [term 5 FOLLOWER]: Advancing to term 6
W20260130 02:58:26.379596 19314 leader_election.cc:343] T 532e867fd5944d468d7672d2a5c92f3c P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Illegal state: must be running to vote when last-logged opid is not known
W20260130 02:58:26.380833 19314 leader_election.cc:343] T 532e867fd5944d468d7672d2a5c92f3c P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Illegal state: must be running to vote when last-logged opid is not known
W20260130 02:58:26.381510 19463 leader_election.cc:343] T 6d1f121f49cd47c2b8247daaf6874e01 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:26.382138 21133 consensus_queue.cc:1048] T c76f0dc7762c49c389fa6fef41f02141 P 78c2c46c7520492db35d51c244fb4f74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260130 02:58:26.394253 19370 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3738385130c044e0a5a5803da74c567d" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74" is_pre_election: true
I20260130 02:58:26.394714 19379 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e691c238199b4914bcc3fd248f31420c" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74" is_pre_election: true
I20260130 02:58:26.395323 19379 raft_consensus.cc:2468] T e691c238199b4914bcc3fd248f31420c P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d870164263bb44b6b780a557daea3656 in term 0.
I20260130 02:58:26.398212 21282 ts_tablet_manager.cc:1916] T e47b22aafe5247c4b62c4193c24f427c P 371a36d219164974abdf848841528152: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260130 02:58:26.421903 21240 leader_election.cc:290] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4 [CANDIDATE]: Term 6 election: Requested vote from peers 107f274cdda543038832f06a56ab9130 (127.18.100.125:39711), 3749baa287ed4f249329a74071781edd (127.18.100.124:34629)
I20260130 02:58:26.423636 20807 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 3749baa287ed4f249329a74071781edd [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 658299197a424f53a37f14fadf089ae4 in term 6.
I20260130 02:58:26.424850 18860 leader_election.cc:304] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3749baa287ed4f249329a74071781edd, 658299197a424f53a37f14fadf089ae4; no voters: 107f274cdda543038832f06a56ab9130
I20260130 02:58:26.425872 21240 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4 [term 6 FOLLOWER]: Leader election won for term 6
I20260130 02:58:26.357733 19807 consensus_queue.cc:260] T 2b8034a92b2043e5b57cf02f1ee6296d P 371a36d219164974abdf848841528152 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
W20260130 02:58:26.432804 19462 leader_election.cc:343] T 3738385130c044e0a5a5803da74c567d P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:26.433638 19462 leader_election.cc:304] T e691c238199b4914bcc3fd248f31420c P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78c2c46c7520492db35d51c244fb4f74, d870164263bb44b6b780a557daea3656; no voters:
I20260130 02:58:26.434873 21094 raft_consensus.cc:2804] T e691c238199b4914bcc3fd248f31420c P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260130 02:58:26.434823 21211 consensus_queue.cc:1048] T 9896ff9364744857968e87e80e776dd2 P d870164263bb44b6b780a557daea3656 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260130 02:58:26.435243 21094 raft_consensus.cc:493] T e691c238199b4914bcc3fd248f31420c P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260130 02:58:26.435571 21094 raft_consensus.cc:3060] T e691c238199b4914bcc3fd248f31420c P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:26.435837 19625 tablet_bootstrap.cc:654] T 8b47653c7c3945a68e9bf4680bfc0496 P d870164263bb44b6b780a557daea3656: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:26.452463 21276 ts_tablet_manager.cc:1929] T e2aaaf00f53d418b87d5c7af286e4d84 P d870164263bb44b6b780a557daea3656: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260130 02:58:26.454267 21275 ts_tablet_manager.cc:1929] T e47b22aafe5247c4b62c4193c24f427c P 78c2c46c7520492db35d51c244fb4f74: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260130 02:58:26.458389 19358 tablet_service.cc:1552] Processing DeleteTablet for tablet a8a5038e9def44eb8b3b1e4aeae25652 with delete_type TABLET_DATA_DELETED (replaced by tablet 475b435b2967418f927fa172d2ff04ea) from {username='slave'} at 127.0.0.1:60110
I20260130 02:58:26.462437 21275 log.cc:1199] T e47b22aafe5247c4b62c4193c24f427c P 78c2c46c7520492db35d51c244fb4f74: Deleting WAL directory at /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-1/wal/wals/e47b22aafe5247c4b62c4193c24f427c
I20260130 02:58:26.463629 21275 ts_tablet_manager.cc:1950] T e47b22aafe5247c4b62c4193c24f427c P 78c2c46c7520492db35d51c244fb4f74: Deleting consensus metadata
I20260130 02:58:26.465262 21240 raft_consensus.cc:697] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4 [term 6 LEADER]: Becoming Leader. State: Replica: 658299197a424f53a37f14fadf089ae4, State: Running, Role: LEADER
I20260130 02:58:26.468601 18860 catalog_manager.cc:4985] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767): tablet e47b22aafe5247c4b62c4193c24f427c (table default.Table_d11b53ac041740d8818edf7d2861dc6d [id=edeb74c78d234ee6a5a031c8347d3884]) successfully deleted
I20260130 02:58:26.477257 21276 log.cc:1199] T e2aaaf00f53d418b87d5c7af286e4d84 P d870164263bb44b6b780a557daea3656: Deleting WAL directory at /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-2/wal/wals/e2aaaf00f53d418b87d5c7af286e4d84
I20260130 02:58:26.478042 21292 raft_consensus.cc:493] T 30cc52136bb3422895d95c0fed0c6798 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:26.478479 21292 raft_consensus.cc:515] T 30cc52136bb3422895d95c0fed0c6798 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:26.480295 21292 leader_election.cc:290] T 30cc52136bb3422895d95c0fed0c6798 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d870164263bb44b6b780a557daea3656 (127.18.100.67:39571), 371a36d219164974abdf848841528152 (127.18.100.65:42021)
I20260130 02:58:26.479912 19807 ts_tablet_manager.cc:1434] T 2b8034a92b2043e5b57cf02f1ee6296d P 371a36d219164974abdf848841528152: Time spent starting tablet: real 0.222s user 0.006s sys 0.001s
I20260130 02:58:26.482111 19807 tablet_bootstrap.cc:492] T 39bcc41f10ae4241b370529364454586 P 371a36d219164974abdf848841528152: Bootstrap starting.
I20260130 02:58:26.466207 21240 consensus_queue.cc:237] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 372, Committed index: 372, Last appended: 4.373, Last appended by leader: 373, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "658299197a424f53a37f14fadf089ae4" member_type: VOTER last_known_addr { host: "127.18.100.126" port: 38191 } } peers { permanent_uuid: "107f274cdda543038832f06a56ab9130" member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39711 } } peers { permanent_uuid: "3749baa287ed4f249329a74071781edd" member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34629 } }
I20260130 02:58:26.485562 21276 ts_tablet_manager.cc:1950] T e2aaaf00f53d418b87d5c7af286e4d84 P d870164263bb44b6b780a557daea3656: Deleting consensus metadata
I20260130 02:58:26.499207 19197 tablet_service.cc:1552] Processing DeleteTablet for tablet e2aaaf00f53d418b87d5c7af286e4d84 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:58:08 UTC) from {username='slave'} at 127.0.0.1:42870
I20260130 02:58:26.500207 19197 tablet_service.cc:1552] Processing DeleteTablet for tablet 3389db58493448aea3cf00aca411047c with delete_type TABLET_DATA_DELETED (replaced by tablet af0f90071a334016aa17fae2e3a97fc8) from {username='slave'} at 127.0.0.1:42870
I20260130 02:58:26.503515 19197 tablet_service.cc:1552] Processing DeleteTablet for tablet c7c621b99deb457998c44ab1644d3939 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:58:16 UTC) from {username='slave'} at 127.0.0.1:42870
I20260130 02:58:26.505126 19807 tablet_bootstrap.cc:654] T 39bcc41f10ae4241b370529364454586 P 371a36d219164974abdf848841528152: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:26.507580 19523 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c76f0dc7762c49c389fa6fef41f02141" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
I20260130 02:58:26.509495 19376 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c76f0dc7762c49c389fa6fef41f02141" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74" is_pre_election: true
I20260130 02:58:26.500406 19625 tablet_bootstrap.cc:492] T 8b47653c7c3945a68e9bf4680bfc0496 P d870164263bb44b6b780a557daea3656: No bootstrap required, opened a new log
I20260130 02:58:26.511580 19625 ts_tablet_manager.cc:1403] T 8b47653c7c3945a68e9bf4680bfc0496 P d870164263bb44b6b780a557daea3656: Time spent bootstrapping tablet: real 0.216s user 0.009s sys 0.001s
I20260130 02:58:26.520257 18860 catalog_manager.cc:4985] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): tablet e2aaaf00f53d418b87d5c7af286e4d84 (table default.Table_e84c6277dfc944059f62cbe0d6b937ee [id=747af828147149efb7b7b8d33c49286d]) successfully deleted
I20260130 02:58:26.522650 19358 tablet_service.cc:1552] Processing DeleteTablet for tablet e47b22aafe5247c4b62c4193c24f427c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:58:22 UTC) from {username='slave'} at 127.0.0.1:60110
I20260130 02:58:26.528926 19358 tablet_service.cc:1552] Processing DeleteTablet for tablet 8e57ae31ab3941f789ac69627384e3b0 with delete_type TABLET_DATA_DELETED (replaced by tablet 8b47653c7c3945a68e9bf4680bfc0496) from {username='slave'} at 127.0.0.1:60110
W20260130 02:58:26.535423 18860 catalog_manager.cc:4952] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767): delete failed for tablet a8a5038e9def44eb8b3b1e4aeae25652 because the tablet was not found. No further retry: Not found: Tablet not found: a8a5038e9def44eb8b3b1e4aeae25652
W20260130 02:58:26.536657 18860 catalog_manager.cc:4952] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767): delete failed for tablet e47b22aafe5247c4b62c4193c24f427c because the tablet was not found. No further retry: Not found: Tablet not found: e47b22aafe5247c4b62c4193c24f427c
W20260130 02:58:26.537202 18860 catalog_manager.cc:4952] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767): delete failed for tablet 8e57ae31ab3941f789ac69627384e3b0 because the tablet was not found. No further retry: Not found: Tablet not found: 8e57ae31ab3941f789ac69627384e3b0
I20260130 02:58:26.543152 19358 tablet_service.cc:1552] Processing DeleteTablet for tablet 04f4ac27800c4ac3a79ae12d35479757 with delete_type TABLET_DATA_DELETED (replaced by tablet 3738385130c044e0a5a5803da74c567d) from {username='slave'} at 127.0.0.1:60110
W20260130 02:58:26.544869 18860 catalog_manager.cc:4952] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767): delete failed for tablet 04f4ac27800c4ac3a79ae12d35479757 because the tablet was not found. No further retry: Not found: Tablet not found: 04f4ac27800c4ac3a79ae12d35479757
I20260130 02:58:26.546200 21094 raft_consensus.cc:515] T e691c238199b4914bcc3fd248f31420c P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:26.548612 19197 tablet_service.cc:1552] Processing DeleteTablet for tablet 04f4ac27800c4ac3a79ae12d35479757 with delete_type TABLET_DATA_DELETED (replaced by tablet 3738385130c044e0a5a5803da74c567d) from {username='slave'} at 127.0.0.1:42870
I20260130 02:58:26.549571 19197 tablet_service.cc:1552] Processing DeleteTablet for tablet 78e9bc3e55fe4e4a98e8eb01da59d365 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:58:08 UTC) from {username='slave'} at 127.0.0.1:42870
I20260130 02:58:26.553350 19220 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3738385130c044e0a5a5803da74c567d" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152" is_pre_election: true
I20260130 02:58:26.554651 19220 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e691c238199b4914bcc3fd248f31420c" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152" is_pre_election: true
W20260130 02:58:26.568275 18860 catalog_manager.cc:4952] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): delete failed for tablet c7c621b99deb457998c44ab1644d3939 because the tablet was not found. No further retry: Not found: Tablet not found: c7c621b99deb457998c44ab1644d3939
I20260130 02:58:26.573254 19358 tablet_service.cc:1552] Processing DeleteTablet for tablet 8e57ae31ab3941f789ac69627384e3b0 with delete_type TABLET_DATA_DELETED (replaced by tablet 8b47653c7c3945a68e9bf4680bfc0496) from {username='slave'} at 127.0.0.1:60110
W20260130 02:58:26.575109 18860 catalog_manager.cc:4952] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767): delete failed for tablet 8e57ae31ab3941f789ac69627384e3b0 because the tablet was not found. No further retry: Not found: Tablet not found: 8e57ae31ab3941f789ac69627384e3b0
I20260130 02:58:26.578161 21200 raft_consensus.cc:493] T cfd12a72726846e587cff9d6a4d61bd6 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:26.578639 21200 raft_consensus.cc:515] T cfd12a72726846e587cff9d6a4d61bd6 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:26.580410 21200 leader_election.cc:290] T cfd12a72726846e587cff9d6a4d61bd6 P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d870164263bb44b6b780a557daea3656 (127.18.100.67:39571), 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767)
I20260130 02:58:26.593879 19520 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "30cc52136bb3422895d95c0fed0c6798" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
I20260130 02:58:26.598865 19358 tablet_service.cc:1552] Processing DeleteTablet for tablet e2aaaf00f53d418b87d5c7af286e4d84 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:58:08 UTC) from {username='slave'} at 127.0.0.1:60110
W20260130 02:58:26.600239 18860 catalog_manager.cc:4952] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767): delete failed for tablet e2aaaf00f53d418b87d5c7af286e4d84 because the tablet was not found. No further retry: Not found: Tablet not found: e2aaaf00f53d418b87d5c7af286e4d84
I20260130 02:58:26.605974 21307 sys_catalog.cc:455] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 658299197a424f53a37f14fadf089ae4. Latest consensus state: current_term: 6 leader_uuid: "658299197a424f53a37f14fadf089ae4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "658299197a424f53a37f14fadf089ae4" member_type: VOTER last_known_addr { host: "127.18.100.126" port: 38191 } } peers { permanent_uuid: "107f274cdda543038832f06a56ab9130" member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39711 } } peers { permanent_uuid: "3749baa287ed4f249329a74071781edd" member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34629 } } }
I20260130 02:58:26.606546 19807 tablet_bootstrap.cc:492] T 39bcc41f10ae4241b370529364454586 P 371a36d219164974abdf848841528152: No bootstrap required, opened a new log
I20260130 02:58:26.606865 21307 sys_catalog.cc:458] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4 [sys.catalog]: This master's current role is: LEADER
I20260130 02:58:26.606950 19807 ts_tablet_manager.cc:1403] T 39bcc41f10ae4241b370529364454586 P 371a36d219164974abdf848841528152: Time spent bootstrapping tablet: real 0.125s user 0.008s sys 0.001s
I20260130 02:58:26.607708 21215 consensus_queue.cc:1048] T 00ddbfe6218c44d788fbf65245a31dc6 P d870164263bb44b6b780a557daea3656 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260130 02:58:26.608654 19358 tablet_service.cc:1552] Processing DeleteTablet for tablet e12d130bcdf34965888753f12c1ff848 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:58:18 UTC) from {username='slave'} at 127.0.0.1:60110
I20260130 02:58:26.609020 19807 raft_consensus.cc:359] T 39bcc41f10ae4241b370529364454586 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:26.609586 19807 raft_consensus.cc:385] T 39bcc41f10ae4241b370529364454586 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:26.609437 21215 consensus_queue.cc:1048] T dd377fab8c544bf69121dc9e9a060b29 P d870164263bb44b6b780a557daea3656 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260130 02:58:26.609835 19807 raft_consensus.cc:740] T 39bcc41f10ae4241b370529364454586 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 371a36d219164974abdf848841528152, State: Initialized, Role: FOLLOWER
W20260130 02:58:26.610263 18860 catalog_manager.cc:4952] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767): delete failed for tablet e12d130bcdf34965888753f12c1ff848 because the tablet was not found. No further retry: Not found: Tablet not found: e12d130bcdf34965888753f12c1ff848
I20260130 02:58:26.610368 19807 consensus_queue.cc:260] T 39bcc41f10ae4241b370529364454586 P 371a36d219164974abdf848841528152 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:26.613173 19358 tablet_service.cc:1552] Processing DeleteTablet for tablet e12d130bcdf34965888753f12c1ff848 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:58:18 UTC) from {username='slave'} at 127.0.0.1:60110
W20260130 02:58:26.614609 18860 catalog_manager.cc:4952] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767): delete failed for tablet e12d130bcdf34965888753f12c1ff848 because the tablet was not found. No further retry: Not found: Tablet not found: e12d130bcdf34965888753f12c1ff848
I20260130 02:58:26.546219 19625 raft_consensus.cc:359] T 8b47653c7c3945a68e9bf4680bfc0496 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:26.617095 19625 raft_consensus.cc:385] T 8b47653c7c3945a68e9bf4680bfc0496 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:26.617348 19625 raft_consensus.cc:740] T 8b47653c7c3945a68e9bf4680bfc0496 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d870164263bb44b6b780a557daea3656, State: Initialized, Role: FOLLOWER
I20260130 02:58:26.617740 19807 ts_tablet_manager.cc:1434] T 39bcc41f10ae4241b370529364454586 P 371a36d219164974abdf848841528152: Time spent starting tablet: real 0.011s user 0.004s sys 0.000s
I20260130 02:58:26.618368 19220 raft_consensus.cc:3060] T 2b8034a92b2043e5b57cf02f1ee6296d P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:26.618602 19807 tablet_bootstrap.cc:492] T 00ddbfe6218c44d788fbf65245a31dc6 P 371a36d219164974abdf848841528152: Bootstrap starting.
I20260130 02:58:26.623795 19807 tablet_bootstrap.cc:654] T 00ddbfe6218c44d788fbf65245a31dc6 P 371a36d219164974abdf848841528152: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:26.628638 19358 tablet_service.cc:1552] Processing DeleteTablet for tablet 2c79a3aecab844698e70834999e1caac with delete_type TABLET_DATA_DELETED (replaced by tablet 94355fc1338f479e8798070aff04fcba) from {username='slave'} at 127.0.0.1:60110
W20260130 02:58:26.630620 18860 catalog_manager.cc:4952] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767): delete failed for tablet 2c79a3aecab844698e70834999e1caac because the tablet was not found. No further retry: Not found: Tablet not found: 2c79a3aecab844698e70834999e1caac
I20260130 02:58:26.634488 21094 leader_election.cc:290] T e691c238199b4914bcc3fd248f31420c P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 election: Requested vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767)
I20260130 02:58:26.635474 19224 raft_consensus.cc:3060] T cfd12a72726846e587cff9d6a4d61bd6 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:26.637835 19358 tablet_service.cc:1552] Processing DeleteTablet for tablet 2c79a3aecab844698e70834999e1caac with delete_type TABLET_DATA_DELETED (replaced by tablet 94355fc1338f479e8798070aff04fcba) from {username='slave'} at 127.0.0.1:60110
W20260130 02:58:26.638048 18860 catalog_manager.cc:4952] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): delete failed for tablet 8e57ae31ab3941f789ac69627384e3b0 because the tablet was not found. No further retry: Not found: Tablet not found: 8e57ae31ab3941f789ac69627384e3b0
W20260130 02:58:26.638651 18860 catalog_manager.cc:4952] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): delete failed for tablet 04f4ac27800c4ac3a79ae12d35479757 because the tablet was not found. No further retry: Not found: Tablet not found: 04f4ac27800c4ac3a79ae12d35479757
W20260130 02:58:26.639274 18860 catalog_manager.cc:4952] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): delete failed for tablet a8a5038e9def44eb8b3b1e4aeae25652 because the tablet was not found. No further retry: Not found: Tablet not found: a8a5038e9def44eb8b3b1e4aeae25652
I20260130 02:58:26.639773 19358 tablet_service.cc:1552] Processing DeleteTablet for tablet ab6afa40408a4f458495abf20290e55b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:58:15 UTC) from {username='slave'} at 127.0.0.1:60110
W20260130 02:58:26.639845 18860 catalog_manager.cc:4952] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767): delete failed for tablet 2c79a3aecab844698e70834999e1caac because the tablet was not found. No further retry: Not found: Tablet not found: 2c79a3aecab844698e70834999e1caac
I20260130 02:58:26.640496 21275 tablet_replica.cc:333] T ab6afa40408a4f458495abf20290e55b P 78c2c46c7520492db35d51c244fb4f74: stopping tablet replica
W20260130 02:58:26.641036 18860 catalog_manager.cc:4952] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): delete failed for tablet 78e9bc3e55fe4e4a98e8eb01da59d365 because the tablet was not found. No further retry: Not found: Tablet not found: 78e9bc3e55fe4e4a98e8eb01da59d365
W20260130 02:58:26.643534 18866 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.18.100.66:59691 (request call id 39) took 4969 ms (client timeout 1000 ms). Trace:
I20260130 02:58:26.645145 19224 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e691c238199b4914bcc3fd248f31420c" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152"
W20260130 02:58:26.659924 19462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.100.66:40767, user_credentials={real_user=slave}} blocked reactor thread for 51397.7us
I20260130 02:58:26.664659 19625 consensus_queue.cc:260] T 8b47653c7c3945a68e9bf4680bfc0496 P d870164263bb44b6b780a557daea3656 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
W20260130 02:58:26.669911 19313 leader_election.cc:343] T 30cc52136bb3422895d95c0fed0c6798 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:26.674134 19520 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cfd12a72726846e587cff9d6a4d61bd6" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
I20260130 02:58:26.674978 19366 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cfd12a72726846e587cff9d6a4d61bd6" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74" is_pre_election: true
W20260130 02:58:26.644294 18860 catalog_manager.cc:4952] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): delete failed for tablet e12d130bcdf34965888753f12c1ff848 because the tablet was not found. No further retry: Not found: Tablet not found: e12d130bcdf34965888753f12c1ff848
W20260130 02:58:26.671684 18868 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.18.100.66:59691 (request call id 38) took 6731 ms (client timeout 1000 ms). Trace:
W20260130 02:58:26.684823 18868 rpcz_store.cc:269] 0130 02:58:19.940547 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:58:20.037706 (+ 97159us) service_pool.cc:225] Handling call
0130 02:58:25.953362 (+5915656us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:58:25.953382 (+ 20us) write_op.cc:454] Released schema lock
0130 02:58:26.671608 (+718226us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0130 02:58:25.953177 (+ 0us) op_driver.cc:389] HandleFailure(Illegal state: replica 658299197a424f53a37f14fadf089ae4 is not leader of this config: current role FOLLOWER)
0130 02:58:25.953212 (+ 35us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:58:25.953233 (+ 21us) write_op.cc:454] Released schema lock
0130 02:58:25.953252 (+ 19us) write_op.cc:335] FINISH: Op aborted
Metrics: {"reported_tablets":79,"child_traces":[["op",{}]]}
I20260130 02:58:26.697381 19840 tablet_bootstrap.cc:654] T 3738385130c044e0a5a5803da74c567d P 78c2c46c7520492db35d51c244fb4f74: Neither blocks nor log segments found. Creating new log.
W20260130 02:58:26.700435 18859 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet 657e9b0d8c05413f904129dce745710b (table default.Table_526677c8ef4144bd9de7a54a5e78e2be [id=b16908b75587459597a978c71756451c], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 657e9b0d8c05413f904129dce745710b consensus state has no leader
I20260130 02:58:26.703517 21318 tablet.cc:1722] T 274530359b174ff79f046c2d4a6afa76 P 78c2c46c7520492db35d51c244fb4f74: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260130 02:58:26.708832 19159 leader_election.cc:304] T c76f0dc7762c49c389fa6fef41f02141 P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 371a36d219164974abdf848841528152; no voters: 78c2c46c7520492db35d51c244fb4f74, d870164263bb44b6b780a557daea3656
I20260130 02:58:26.711803 19625 ts_tablet_manager.cc:1434] T 8b47653c7c3945a68e9bf4680bfc0496 P d870164263bb44b6b780a557daea3656: Time spent starting tablet: real 0.200s user 0.004s sys 0.002s
I20260130 02:58:26.715904 21294 raft_consensus.cc:2749] T c76f0dc7762c49c389fa6fef41f02141 P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260130 02:58:26.742933 21275 raft_consensus.cc:2243] T ab6afa40408a4f458495abf20290e55b P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260130 02:58:26.743544 21275 raft_consensus.cc:2272] T ab6afa40408a4f458495abf20290e55b P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260130 02:58:26.745469 21275 ts_tablet_manager.cc:1916] T ab6afa40408a4f458495abf20290e55b P 78c2c46c7520492db35d51c244fb4f74: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260130 02:58:26.754271 19159 leader_election.cc:304] T cfd12a72726846e587cff9d6a4d61bd6 P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 371a36d219164974abdf848841528152; no voters: 78c2c46c7520492db35d51c244fb4f74, d870164263bb44b6b780a557daea3656
I20260130 02:58:26.758698 21169 raft_consensus.cc:2749] T cfd12a72726846e587cff9d6a4d61bd6 P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260130 02:58:26.679498 18866 rpcz_store.cc:269] 0130 02:58:21.674467 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:58:22.251098 (+576631us) service_pool.cc:225] Handling call
0130 02:58:25.950004 (+3698906us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:58:25.950025 (+ 21us) write_op.cc:454] Released schema lock
0130 02:58:26.643460 (+693435us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0130 02:58:23.144790 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0130 02:58:23.145336 (+ 546us) write_op.cc:432] Acquiring schema lock in shared mode
0130 02:58:23.145364 (+ 28us) write_op.cc:435] Acquired schema lock
0130 02:58:23.145390 (+ 26us) tablet.cc:660] Decoding operations
0130 02:58:23.145679 (+ 289us) write_op.cc:620] Acquiring the partition lock for write op
0130 02:58:23.145723 (+ 44us) write_op.cc:641] Partition lock acquired for write op
0130 02:58:23.145752 (+ 29us) tablet.cc:683] Acquiring locks for 1 operations
0130 02:58:23.145914 (+ 162us) tablet.cc:699] Row locks acquired
0130 02:58:23.145930 (+ 16us) write_op.cc:260] PREPARE: finished
0130 02:58:23.146050 (+ 120us) write_op.cc:270] Start()
0130 02:58:23.146143 (+ 93us) write_op.cc:276] Timestamp: P: 1769741903146018 usec, L: 0
0130 02:58:23.146161 (+ 18us) op_driver.cc:348] REPLICATION: starting
0130 02:58:23.146779 (+ 618us) log.cc:844] Serialized 533 byte log entry
0130 02:58:23.161036 (+ 14257us) op_driver.cc:464] REPLICATION: finished
0130 02:58:23.161410 (+ 374us) write_op.cc:301] APPLY: starting
0130 02:58:23.161465 (+ 55us) tablet.cc:1365] starting BulkCheckPresence
0130 02:58:23.161548 (+ 83us) tablet.cc:1368] finished BulkCheckPresence
0130 02:58:23.161661 (+ 113us) tablet.cc:1370] starting ApplyRowOperation cycle
0130 02:58:23.161853 (+ 192us) tablet.cc:1381] finished ApplyRowOperation cycle
0130 02:58:23.161987 (+ 134us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0130 02:58:23.162025 (+ 38us) write_op.cc:312] APPLY: finished
0130 02:58:23.162194 (+ 169us) log.cc:844] Serialized 23 byte log entry
0130 02:58:23.162326 (+ 132us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:58:23.162390 (+ 64us) write_op.cc:454] Released schema lock
0130 02:58:23.162419 (+ 29us) write_op.cc:341] FINISH: Updating metrics
0130 02:58:23.162665 (+ 246us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:58:23.162689 (+ 24us) write_op.cc:454] Released schema lock
Related trace 'op':
0130 02:58:25.949799 (+ 0us) op_driver.cc:389] HandleFailure(Illegal state: replica 658299197a424f53a37f14fadf089ae4 is not leader of this config: current role FOLLOWER)
0130 02:58:25.949835 (+ 36us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:58:25.949855 (+ 20us) write_op.cc:454] Released schema lock
0130 02:58:25.949873 (+ 18us) write_op.cc:335] FINISH: Op aborted
Metrics: {"mutex_wait_us":190579,"reported_tablets":79,"child_traces":[["op",{"apply.queue_time_us":232,"apply.run_cpu_time_us":1183,"apply.run_wall_time_us":1182,"num_ops":1,"prepare.queue_time_us":3053,"prepare.run_cpu_time_us":2303,"prepare.run_wall_time_us":2304,"raft.queue_time_us":203,"raft.run_cpu_time_us":628,"raft.run_wall_time_us":628,"replication_time_us":14784}],["op",{}]]}
I20260130 02:58:26.769444 19807 tablet_bootstrap.cc:492] T 00ddbfe6218c44d788fbf65245a31dc6 P 371a36d219164974abdf848841528152: No bootstrap required, opened a new log
I20260130 02:58:26.769250 21249 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260130 02:58:26.769853 19807 ts_tablet_manager.cc:1403] T 00ddbfe6218c44d788fbf65245a31dc6 P 371a36d219164974abdf848841528152: Time spent bootstrapping tablet: real 0.151s user 0.030s sys 0.031s
I20260130 02:58:26.778486 19807 raft_consensus.cc:359] T 00ddbfe6218c44d788fbf65245a31dc6 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:26.783126 19807 raft_consensus.cc:385] T 00ddbfe6218c44d788fbf65245a31dc6 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:26.706122 21317 tablet.cc:1722] T 274530359b174ff79f046c2d4a6afa76 P 371a36d219164974abdf848841528152: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260130 02:58:26.750907 21285 tablet.cc:1722] T 274530359b174ff79f046c2d4a6afa76 P d870164263bb44b6b780a557daea3656: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20260130 02:58:26.729148 19585 thread.cc:641] client-negotiator [worker] (thread pool) Time spent creating pthread: real 1.031s user 0.002s sys 0.006s
W20260130 02:58:26.796883 19585 thread.cc:608] client-negotiator [worker] (thread pool) Time spent starting thread: real 1.102s user 0.003s sys 0.006s
W20260130 02:58:25.688920 19584 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.18.100.126:38191)
W20260130 02:58:26.799314 19584 outbound_call.cc:321] RPC callback for RPC call kudu.master.MasterService.AlterTable -> {remote=127.18.100.126:38191, user_credentials={real_user=slave}} blocked reactor thread for 1.11304e+06us
W20260130 02:58:26.802529 19584 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.18.100.126:38191)
I20260130 02:58:26.805732 19364 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e691c238199b4914bcc3fd248f31420c" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74"
I20260130 02:58:26.806535 19364 raft_consensus.cc:3060] T e691c238199b4914bcc3fd248f31420c P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:26.815021 19364 raft_consensus.cc:2468] T e691c238199b4914bcc3fd248f31420c P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d870164263bb44b6b780a557daea3656 in term 1.
I20260130 02:58:26.817634 19625 tablet_bootstrap.cc:492] T bc04be1284414a8699d3c4b88bf62813 P d870164263bb44b6b780a557daea3656: Bootstrap starting.
I20260130 02:58:26.783420 19807 raft_consensus.cc:740] T 00ddbfe6218c44d788fbf65245a31dc6 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 371a36d219164974abdf848841528152, State: Initialized, Role: FOLLOWER
I20260130 02:58:26.828708 19807 consensus_queue.cc:260] T 00ddbfe6218c44d788fbf65245a31dc6 P 371a36d219164974abdf848841528152 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:26.830765 19807 ts_tablet_manager.cc:1434] T 00ddbfe6218c44d788fbf65245a31dc6 P 371a36d219164974abdf848841528152: Time spent starting tablet: real 0.061s user 0.005s sys 0.004s
I20260130 02:58:26.832345 19807 tablet_bootstrap.cc:492] T 9896ff9364744857968e87e80e776dd2 P 371a36d219164974abdf848841528152: Bootstrap starting.
I20260130 02:58:26.830343 21121 consensus_queue.cc:1048] T cd30d7adcc4545848e3501a205a10293 P 78c2c46c7520492db35d51c244fb4f74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260130 02:58:26.840061 21282 ts_tablet_manager.cc:1929] T e47b22aafe5247c4b62c4193c24f427c P 371a36d219164974abdf848841528152: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
W20260130 02:58:26.834103 19463 consensus_peers.cc:597] T dd377fab8c544bf69121dc9e9a060b29 P d870164263bb44b6b780a557daea3656 -> Peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Couldn't send request to peer 371a36d219164974abdf848841528152. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260130 02:58:26.846622 21314 raft_consensus.cc:493] T 8b47653c7c3945a68e9bf4680bfc0496 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:26.847054 21314 raft_consensus.cc:515] T 8b47653c7c3945a68e9bf4680bfc0496 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:26.848829 21314 leader_election.cc:290] T 8b47653c7c3945a68e9bf4680bfc0496 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767)
W20260130 02:58:26.864136 19584 master_proxy_rpc.cc:203] Re-attempting DeleteTable request to leader Master (127.18.100.126:38191)
W20260130 02:58:26.864470 19584 outbound_call.cc:321] RPC callback for RPC call kudu.master.MasterService.DeleteTable -> {remote=127.18.100.126:38191, user_credentials={real_user=slave}} blocked reactor thread for 61115.5us
W20260130 02:58:26.866283 19584 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.18.100.126:38191)
W20260130 02:58:26.891209 19463 consensus_peers.cc:597] T 39bcc41f10ae4241b370529364454586 P d870164263bb44b6b780a557daea3656 -> Peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Couldn't send request to peer 371a36d219164974abdf848841528152. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
W20260130 02:58:26.892755 19463 consensus_peers.cc:597] T 00ddbfe6218c44d788fbf65245a31dc6 P d870164263bb44b6b780a557daea3656 -> Peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Couldn't send request to peer 371a36d219164974abdf848841528152. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20260130 02:58:26.893255 19463 leader_election.cc:343] T 3738385130c044e0a5a5803da74c567d P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:26.893599 19463 leader_election.cc:304] T 3738385130c044e0a5a5803da74c567d P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d870164263bb44b6b780a557daea3656; no voters: 371a36d219164974abdf848841528152, 78c2c46c7520492db35d51c244fb4f74
I20260130 02:58:26.898788 21314 raft_consensus.cc:2749] T 3738385130c044e0a5a5803da74c567d P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260130 02:58:26.899533 19463 leader_election.cc:343] T e691c238199b4914bcc3fd248f31420c P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:26.899842 19462 leader_election.cc:304] T e691c238199b4914bcc3fd248f31420c P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78c2c46c7520492db35d51c244fb4f74, d870164263bb44b6b780a557daea3656; no voters:
I20260130 02:58:26.900427 21167 consensus_queue.cc:1048] T 6d1f121f49cd47c2b8247daaf6874e01 P d870164263bb44b6b780a557daea3656 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260130 02:58:26.911878 19584 outbound_call.cc:321] RPC callback for RPC call kudu.master.MasterService.CreateTable -> {remote=127.18.100.126:38191, user_credentials={real_user=slave}} blocked reactor thread for 44758.5us
I20260130 02:58:26.912170 21304 consensus_queue.cc:1048] T 2b8034a92b2043e5b57cf02f1ee6296d P d870164263bb44b6b780a557daea3656 [LEADER]: Connected to new peer: Peer: permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260130 02:58:26.913288 19463 leader_election.cc:343] T e691c238199b4914bcc3fd248f31420c P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:26.913820 21304 consensus_queue.cc:1048] T cfd12a72726846e587cff9d6a4d61bd6 P d870164263bb44b6b780a557daea3656 [LEADER]: Connected to new peer: Peer: permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260130 02:58:26.917670 19463 consensus_peers.cc:597] T 9896ff9364744857968e87e80e776dd2 P d870164263bb44b6b780a557daea3656 -> Peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Couldn't send request to peer 371a36d219164974abdf848841528152. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260130 02:58:26.918843 19625 tablet_bootstrap.cc:654] T bc04be1284414a8699d3c4b88bf62813 P d870164263bb44b6b780a557daea3656: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:26.921224 21167 raft_consensus.cc:2804] T e691c238199b4914bcc3fd248f31420c P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Leader election won for term 1
I20260130 02:58:26.921658 21167 raft_consensus.cc:697] T e691c238199b4914bcc3fd248f31420c P d870164263bb44b6b780a557daea3656 [term 1 LEADER]: Becoming Leader. State: Replica: d870164263bb44b6b780a557daea3656, State: Running, Role: LEADER
I20260130 02:58:26.922407 21167 consensus_queue.cc:237] T e691c238199b4914bcc3fd248f31420c P d870164263bb44b6b780a557daea3656 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
W20260130 02:58:26.932720 21164 catalog_manager.cc:1419] Failed waiting for node to catch up after master election: Illegal state: Node 107f274cdda543038832f06a56ab9130 not leader. Raft Consensus state: current_term: 6 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "658299197a424f53a37f14fadf089ae4" member_type: VOTER last_known_addr { host: "127.18.100.126" port: 38191 } } peers { permanent_uuid: "107f274cdda543038832f06a56ab9130" member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39711 } } peers { permanent_uuid: "3749baa287ed4f249329a74071781edd" member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34629 } } }
W20260130 02:58:26.934231 19584 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.18.100.126:38191)
I20260130 02:58:26.941346 20998 consensus_queue.cc:1048] T 1129f72158ec4bae89da3c94cdb70839 P d870164263bb44b6b780a557daea3656 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260130 02:58:26.941910 19840 tablet_bootstrap.cc:492] T 3738385130c044e0a5a5803da74c567d P 78c2c46c7520492db35d51c244fb4f74: No bootstrap required, opened a new log
I20260130 02:58:26.942292 19840 ts_tablet_manager.cc:1403] T 3738385130c044e0a5a5803da74c567d P 78c2c46c7520492db35d51c244fb4f74: Time spent bootstrapping tablet: real 0.581s user 0.008s sys 0.002s
I20260130 02:58:26.944866 19840 raft_consensus.cc:359] T 3738385130c044e0a5a5803da74c567d P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:26.945487 19840 raft_consensus.cc:385] T 3738385130c044e0a5a5803da74c567d P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:26.945729 19840 raft_consensus.cc:740] T 3738385130c044e0a5a5803da74c567d P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78c2c46c7520492db35d51c244fb4f74, State: Initialized, Role: FOLLOWER
W20260130 02:58:26.947048 21164 catalog_manager.cc:1419] Failed waiting for node to catch up after master election: Illegal state: Node 107f274cdda543038832f06a56ab9130 not leader. Raft Consensus state: current_term: 6 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "658299197a424f53a37f14fadf089ae4" member_type: VOTER last_known_addr { host: "127.18.100.126" port: 38191 } } peers { permanent_uuid: "107f274cdda543038832f06a56ab9130" member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39711 } } peers { permanent_uuid: "3749baa287ed4f249329a74071781edd" member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34629 } } }
I20260130 02:58:26.956228 19625 tablet_bootstrap.cc:492] T bc04be1284414a8699d3c4b88bf62813 P d870164263bb44b6b780a557daea3656: No bootstrap required, opened a new log
I20260130 02:58:26.956620 19625 ts_tablet_manager.cc:1403] T bc04be1284414a8699d3c4b88bf62813 P d870164263bb44b6b780a557daea3656: Time spent bootstrapping tablet: real 0.139s user 0.007s sys 0.001s
I20260130 02:58:26.960227 21121 raft_consensus.cc:493] T 475b435b2967418f927fa172d2ff04ea P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:26.960772 21121 raft_consensus.cc:515] T 475b435b2967418f927fa172d2ff04ea P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:26.963207 21121 leader_election.cc:290] T 475b435b2967418f927fa172d2ff04ea P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), d870164263bb44b6b780a557daea3656 (127.18.100.67:39571)
I20260130 02:58:26.963035 19625 raft_consensus.cc:359] T bc04be1284414a8699d3c4b88bf62813 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:26.963644 19625 raft_consensus.cc:385] T bc04be1284414a8699d3c4b88bf62813 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:26.963898 19625 raft_consensus.cc:740] T bc04be1284414a8699d3c4b88bf62813 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d870164263bb44b6b780a557daea3656, State: Initialized, Role: FOLLOWER
I20260130 02:58:26.964695 19625 consensus_queue.cc:260] T bc04be1284414a8699d3c4b88bf62813 P d870164263bb44b6b780a557daea3656 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:26.966993 19625 ts_tablet_manager.cc:1434] T bc04be1284414a8699d3c4b88bf62813 P d870164263bb44b6b780a557daea3656: Time spent starting tablet: real 0.010s user 0.006s sys 0.001s
W20260130 02:58:26.967092 21164 catalog_manager.cc:1419] Failed waiting for node to catch up after master election: Illegal state: Node 107f274cdda543038832f06a56ab9130 not leader. Raft Consensus state: current_term: 6 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "658299197a424f53a37f14fadf089ae4" member_type: VOTER last_known_addr { host: "127.18.100.126" port: 38191 } } peers { permanent_uuid: "107f274cdda543038832f06a56ab9130" member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39711 } } peers { permanent_uuid: "3749baa287ed4f249329a74071781edd" member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34629 } } }
I20260130 02:58:26.946285 19840 consensus_queue.cc:260] T 3738385130c044e0a5a5803da74c567d P 78c2c46c7520492db35d51c244fb4f74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:26.975744 20807 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 3749baa287ed4f249329a74071781edd [term 6 FOLLOWER]: Refusing update from remote peer 658299197a424f53a37f14fadf089ae4: Log matching property violated. Preceding OpId in replica: term: 3 index: 372. Preceding OpId from leader: term: 6 index: 374. (index mismatch)
I20260130 02:58:26.977161 21307 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3749baa287ed4f249329a74071781edd" member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 374, Last known committed idx: 371, Time since last communication: 0.000s
W20260130 02:58:26.993306 19313 outbound_call.cc:321] RPC callback for RPC call kudu.master.MasterService.TSHeartbeat -> {remote=127.18.100.125:39711, user_credentials={real_user=slave}} blocked reactor thread for 119598us
I20260130 02:58:26.843411 19807 tablet_bootstrap.cc:654] T 9896ff9364744857968e87e80e776dd2 P 371a36d219164974abdf848841528152: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:26.995075 21282 log.cc:1199] T e47b22aafe5247c4b62c4193c24f427c P 371a36d219164974abdf848841528152: Deleting WAL directory at /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-0/wal/wals/e47b22aafe5247c4b62c4193c24f427c
I20260130 02:58:26.995889 21282 ts_tablet_manager.cc:1950] T e47b22aafe5247c4b62c4193c24f427c P 371a36d219164974abdf848841528152: Deleting consensus metadata
I20260130 02:58:26.977268 21294 raft_consensus.cc:493] T 2b8034a92b2043e5b57cf02f1ee6296d P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:26.997435 21294 raft_consensus.cc:515] T 2b8034a92b2043e5b57cf02f1ee6296d P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:26.999260 21294 leader_election.cc:290] T 2b8034a92b2043e5b57cf02f1ee6296d P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d870164263bb44b6b780a557daea3656 (127.18.100.67:39571), 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767)
I20260130 02:58:27.000916 19378 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2b8034a92b2043e5b57cf02f1ee6296d" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74" is_pre_election: true
I20260130 02:58:27.003935 19840 ts_tablet_manager.cc:1434] T 3738385130c044e0a5a5803da74c567d P 78c2c46c7520492db35d51c244fb4f74: Time spent starting tablet: real 0.061s user 0.007s sys 0.000s
I20260130 02:58:27.004575 19523 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2b8034a92b2043e5b57cf02f1ee6296d" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
I20260130 02:58:27.004925 19840 tablet_bootstrap.cc:492] T bc04be1284414a8699d3c4b88bf62813 P 78c2c46c7520492db35d51c244fb4f74: Bootstrap starting.
I20260130 02:58:27.005997 19159 leader_election.cc:304] T 2b8034a92b2043e5b57cf02f1ee6296d P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 371a36d219164974abdf848841528152; no voters: 78c2c46c7520492db35d51c244fb4f74, d870164263bb44b6b780a557daea3656
I20260130 02:58:27.007656 21169 raft_consensus.cc:2749] T 2b8034a92b2043e5b57cf02f1ee6296d P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260130 02:58:27.023432 19807 tablet_bootstrap.cc:492] T 9896ff9364744857968e87e80e776dd2 P 371a36d219164974abdf848841528152: No bootstrap required, opened a new log
I20260130 02:58:27.023932 19807 ts_tablet_manager.cc:1403] T 9896ff9364744857968e87e80e776dd2 P 371a36d219164974abdf848841528152: Time spent bootstrapping tablet: real 0.192s user 0.015s sys 0.002s
I20260130 02:58:27.026117 19807 raft_consensus.cc:359] T 9896ff9364744857968e87e80e776dd2 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:27.026885 19807 raft_consensus.cc:385] T 9896ff9364744857968e87e80e776dd2 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:27.027180 19807 raft_consensus.cc:740] T 9896ff9364744857968e87e80e776dd2 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 371a36d219164974abdf848841528152, State: Initialized, Role: FOLLOWER
I20260130 02:58:27.028134 19807 consensus_queue.cc:260] T 9896ff9364744857968e87e80e776dd2 P 371a36d219164974abdf848841528152 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:27.031008 19807 ts_tablet_manager.cc:1434] T 9896ff9364744857968e87e80e776dd2 P 371a36d219164974abdf848841528152: Time spent starting tablet: real 0.007s user 0.004s sys 0.001s
I20260130 02:58:27.032909 18960 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 107f274cdda543038832f06a56ab9130 [term 6 FOLLOWER]: Refusing update from remote peer 658299197a424f53a37f14fadf089ae4: Log matching property violated. Preceding OpId in replica: term: 5 index: 373. Preceding OpId from leader: term: 6 index: 374. (index mismatch)
I20260130 02:58:27.034236 21307 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "107f274cdda543038832f06a56ab9130" member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39711 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 374, Last known committed idx: 372, Time since last communication: 0.000s
I20260130 02:58:27.034828 21307 consensus_queue.cc:1243] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4 [LEADER]: Peer 107f274cdda543038832f06a56ab9130 log is divergent from this leader: its last log entry 5.373 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 372
I20260130 02:58:27.036754 18960 pending_rounds.cc:85] T 00000000000000000000000000000000 P 107f274cdda543038832f06a56ab9130: Aborting all ops after (but not including) 372
I20260130 02:58:27.037027 18960 pending_rounds.cc:107] T 00000000000000000000000000000000 P 107f274cdda543038832f06a56ab9130: Aborting uncommitted NO_OP operation due to leader change: 5.373
I20260130 02:58:27.037267 18960 raft_consensus.cc:2889] T 00000000000000000000000000000000 P 107f274cdda543038832f06a56ab9130 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260130 02:58:27.040720 19221 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "30cc52136bb3422895d95c0fed0c6798" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152" is_pre_election: true
I20260130 02:58:27.053427 21332 raft_consensus.cc:493] T 00000000000000000000000000000000 P 107f274cdda543038832f06a56ab9130 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 658299197a424f53a37f14fadf089ae4)
I20260130 02:58:27.054080 21332 raft_consensus.cc:515] T 00000000000000000000000000000000 P 107f274cdda543038832f06a56ab9130 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "658299197a424f53a37f14fadf089ae4" member_type: VOTER last_known_addr { host: "127.18.100.126" port: 38191 } } peers { permanent_uuid: "107f274cdda543038832f06a56ab9130" member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39711 } } peers { permanent_uuid: "3749baa287ed4f249329a74071781edd" member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34629 } }
I20260130 02:58:27.055346 21326 sys_catalog.cc:455] T 00000000000000000000000000000000 P 107f274cdda543038832f06a56ab9130 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 658299197a424f53a37f14fadf089ae4. Latest consensus state: current_term: 6 leader_uuid: "658299197a424f53a37f14fadf089ae4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "658299197a424f53a37f14fadf089ae4" member_type: VOTER last_known_addr { host: "127.18.100.126" port: 38191 } } peers { permanent_uuid: "107f274cdda543038832f06a56ab9130" member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39711 } } peers { permanent_uuid: "3749baa287ed4f249329a74071781edd" member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34629 } } }
I20260130 02:58:27.057286 21326 sys_catalog.cc:458] T 00000000000000000000000000000000 P 107f274cdda543038832f06a56ab9130 [sys.catalog]: This master's current role is: FOLLOWER
I20260130 02:58:27.060740 21240 sys_catalog.cc:455] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 6 leader_uuid: "658299197a424f53a37f14fadf089ae4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "658299197a424f53a37f14fadf089ae4" member_type: VOTER last_known_addr { host: "127.18.100.126" port: 38191 } } peers { permanent_uuid: "107f274cdda543038832f06a56ab9130" member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39711 } } peers { permanent_uuid: "3749baa287ed4f249329a74071781edd" member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34629 } } }
I20260130 02:58:27.061501 21240 sys_catalog.cc:458] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4 [sys.catalog]: This master's current role is: LEADER
I20260130 02:58:27.064221 20810 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "107f274cdda543038832f06a56ab9130" candidate_term: 7 candidate_status { last_received { term: 6 index: 374 } } ignore_live_leader: false dest_uuid: "3749baa287ed4f249329a74071781edd" is_pre_election: true
I20260130 02:58:27.070739 21332 leader_election.cc:290] T 00000000000000000000000000000000 P 107f274cdda543038832f06a56ab9130 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 658299197a424f53a37f14fadf089ae4 (127.18.100.126:38191), 3749baa287ed4f249329a74071781edd (127.18.100.124:34629)
I20260130 02:58:27.074501 19625 tablet_bootstrap.cc:492] T 30cc52136bb3422895d95c0fed0c6798 P d870164263bb44b6b780a557daea3656: Bootstrap starting.
I20260130 02:58:27.084198 19625 tablet_bootstrap.cc:654] T 30cc52136bb3422895d95c0fed0c6798 P d870164263bb44b6b780a557daea3656: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:27.100330 19523 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "475b435b2967418f927fa172d2ff04ea" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
I20260130 02:58:27.110313 19807 tablet_bootstrap.cc:492] T 0d14fb95ace9447f810ccbed4d5befdf P 371a36d219164974abdf848841528152: Bootstrap starting.
I20260130 02:58:27.118054 21275 ts_tablet_manager.cc:1929] T ab6afa40408a4f458495abf20290e55b P 78c2c46c7520492db35d51c244fb4f74: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260130 02:58:27.121493 19807 tablet_bootstrap.cc:654] T 0d14fb95ace9447f810ccbed4d5befdf P 371a36d219164974abdf848841528152: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:27.145797 21327 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3749baa287ed4f249329a74071781edd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 658299197a424f53a37f14fadf089ae4. Latest consensus state: current_term: 6 leader_uuid: "658299197a424f53a37f14fadf089ae4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "658299197a424f53a37f14fadf089ae4" member_type: VOTER last_known_addr { host: "127.18.100.126" port: 38191 } } peers { permanent_uuid: "107f274cdda543038832f06a56ab9130" member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39711 } } peers { permanent_uuid: "3749baa287ed4f249329a74071781edd" member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34629 } } }
I20260130 02:58:27.147049 21327 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3749baa287ed4f249329a74071781edd [sys.catalog]: This master's current role is: FOLLOWER
I20260130 02:58:27.170353 21331 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3749baa287ed4f249329a74071781edd [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 6 leader_uuid: "658299197a424f53a37f14fadf089ae4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "658299197a424f53a37f14fadf089ae4" member_type: VOTER last_known_addr { host: "127.18.100.126" port: 38191 } } peers { permanent_uuid: "107f274cdda543038832f06a56ab9130" member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39711 } } peers { permanent_uuid: "3749baa287ed4f249329a74071781edd" member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34629 } } }
I20260130 02:58:27.171629 21331 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3749baa287ed4f249329a74071781edd [sys.catalog]: This master's current role is: FOLLOWER
I20260130 02:58:27.172828 19220 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "475b435b2967418f927fa172d2ff04ea" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152" is_pre_election: true
I20260130 02:58:27.199529 21275 log.cc:1199] T ab6afa40408a4f458495abf20290e55b P 78c2c46c7520492db35d51c244fb4f74: Deleting WAL directory at /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-1/wal/wals/ab6afa40408a4f458495abf20290e55b
I20260130 02:58:27.200448 21275 ts_tablet_manager.cc:1950] T ab6afa40408a4f458495abf20290e55b P 78c2c46c7520492db35d51c244fb4f74: Deleting consensus metadata
I20260130 02:58:27.156594 19378 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8b47653c7c3945a68e9bf4680bfc0496" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74" is_pre_election: true
I20260130 02:58:27.200961 19378 raft_consensus.cc:2468] T 8b47653c7c3945a68e9bf4680bfc0496 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d870164263bb44b6b780a557daea3656 in term 0.
I20260130 02:58:27.128969 19840 tablet_bootstrap.cc:654] T bc04be1284414a8699d3c4b88bf62813 P 78c2c46c7520492db35d51c244fb4f74: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:27.208930 19625 tablet_bootstrap.cc:492] T 30cc52136bb3422895d95c0fed0c6798 P d870164263bb44b6b780a557daea3656: No bootstrap required, opened a new log
I20260130 02:58:27.209371 19625 ts_tablet_manager.cc:1403] T 30cc52136bb3422895d95c0fed0c6798 P d870164263bb44b6b780a557daea3656: Time spent bootstrapping tablet: real 0.135s user 0.006s sys 0.009s
I20260130 02:58:27.212769 19625 raft_consensus.cc:359] T 30cc52136bb3422895d95c0fed0c6798 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:27.213364 19625 raft_consensus.cc:385] T 30cc52136bb3422895d95c0fed0c6798 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:27.213591 19625 raft_consensus.cc:740] T 30cc52136bb3422895d95c0fed0c6798 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d870164263bb44b6b780a557daea3656, State: Initialized, Role: FOLLOWER
I20260130 02:58:27.214831 21332 sys_catalog.cc:455] T 00000000000000000000000000000000 P 107f274cdda543038832f06a56ab9130 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 6 leader_uuid: "658299197a424f53a37f14fadf089ae4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "658299197a424f53a37f14fadf089ae4" member_type: VOTER last_known_addr { host: "127.18.100.126" port: 38191 } } peers { permanent_uuid: "107f274cdda543038832f06a56ab9130" member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39711 } } peers { permanent_uuid: "3749baa287ed4f249329a74071781edd" member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34629 } } }
I20260130 02:58:27.215741 21332 sys_catalog.cc:458] T 00000000000000000000000000000000 P 107f274cdda543038832f06a56ab9130 [sys.catalog]: This master's current role is: FOLLOWER
I20260130 02:58:27.216452 21332 sys_catalog.cc:455] T 00000000000000000000000000000000 P 107f274cdda543038832f06a56ab9130 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 6 leader_uuid: "658299197a424f53a37f14fadf089ae4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "658299197a424f53a37f14fadf089ae4" member_type: VOTER last_known_addr { host: "127.18.100.126" port: 38191 } } peers { permanent_uuid: "107f274cdda543038832f06a56ab9130" member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39711 } } peers { permanent_uuid: "3749baa287ed4f249329a74071781edd" member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34629 } } }
I20260130 02:58:27.217301 21332 sys_catalog.cc:458] T 00000000000000000000000000000000 P 107f274cdda543038832f06a56ab9130 [sys.catalog]: This master's current role is: FOLLOWER
I20260130 02:58:27.234278 19840 tablet_bootstrap.cc:492] T bc04be1284414a8699d3c4b88bf62813 P 78c2c46c7520492db35d51c244fb4f74: No bootstrap required, opened a new log
I20260130 02:58:27.239387 19840 ts_tablet_manager.cc:1403] T bc04be1284414a8699d3c4b88bf62813 P 78c2c46c7520492db35d51c244fb4f74: Time spent bootstrapping tablet: real 0.235s user 0.009s sys 0.002s
I20260130 02:58:27.245407 19840 raft_consensus.cc:359] T bc04be1284414a8699d3c4b88bf62813 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:27.249935 19807 tablet_bootstrap.cc:492] T 0d14fb95ace9447f810ccbed4d5befdf P 371a36d219164974abdf848841528152: No bootstrap required, opened a new log
I20260130 02:58:27.250698 19840 raft_consensus.cc:385] T bc04be1284414a8699d3c4b88bf62813 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:27.251060 19840 raft_consensus.cc:740] T bc04be1284414a8699d3c4b88bf62813 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78c2c46c7520492db35d51c244fb4f74, State: Initialized, Role: FOLLOWER
I20260130 02:58:27.250413 19807 ts_tablet_manager.cc:1403] T 0d14fb95ace9447f810ccbed4d5befdf P 371a36d219164974abdf848841528152: Time spent bootstrapping tablet: real 0.140s user 0.010s sys 0.008s
I20260130 02:58:27.251865 19840 consensus_queue.cc:260] T bc04be1284414a8699d3c4b88bf62813 P 78c2c46c7520492db35d51c244fb4f74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:27.254112 19840 ts_tablet_manager.cc:1434] T bc04be1284414a8699d3c4b88bf62813 P 78c2c46c7520492db35d51c244fb4f74: Time spent starting tablet: real 0.014s user 0.004s sys 0.002s
I20260130 02:58:27.214157 19625 consensus_queue.cc:260] T 30cc52136bb3422895d95c0fed0c6798 P d870164263bb44b6b780a557daea3656 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:27.275050 20998 raft_consensus.cc:493] T 30cc52136bb3422895d95c0fed0c6798 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:27.275465 20998 raft_consensus.cc:515] T 30cc52136bb3422895d95c0fed0c6798 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:27.278419 20998 leader_election.cc:290] T 30cc52136bb3422895d95c0fed0c6798 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767)
I20260130 02:58:27.293254 19625 ts_tablet_manager.cc:1434] T 30cc52136bb3422895d95c0fed0c6798 P d870164263bb44b6b780a557daea3656: Time spent starting tablet: real 0.084s user 0.004s sys 0.005s
I20260130 02:58:27.295857 19807 raft_consensus.cc:359] T 0d14fb95ace9447f810ccbed4d5befdf P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:27.299724 19807 raft_consensus.cc:385] T 0d14fb95ace9447f810ccbed4d5befdf P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:27.300304 19807 raft_consensus.cc:740] T 0d14fb95ace9447f810ccbed4d5befdf P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 371a36d219164974abdf848841528152, State: Initialized, Role: FOLLOWER
I20260130 02:58:27.314615 21294 raft_consensus.cc:493] T 9896ff9364744857968e87e80e776dd2 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:27.315272 21294 raft_consensus.cc:515] T 9896ff9364744857968e87e80e776dd2 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:27.318105 19807 consensus_queue.cc:260] T 0d14fb95ace9447f810ccbed4d5befdf P 371a36d219164974abdf848841528152 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:27.345896 19520 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9896ff9364744857968e87e80e776dd2" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
I20260130 02:58:27.351931 19221 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8b47653c7c3945a68e9bf4680bfc0496" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152" is_pre_election: true
I20260130 02:58:27.350302 19807 ts_tablet_manager.cc:1434] T 0d14fb95ace9447f810ccbed4d5befdf P 371a36d219164974abdf848841528152: Time spent starting tablet: real 0.098s user 0.003s sys 0.008s
I20260130 02:58:27.368498 19379 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9896ff9364744857968e87e80e776dd2" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74" is_pre_election: true
I20260130 02:58:27.369190 19807 tablet_bootstrap.cc:492] T dd377fab8c544bf69121dc9e9a060b29 P 371a36d219164974abdf848841528152: Bootstrap starting.
I20260130 02:58:27.371223 21317 tablet.cc:1722] T 25fd48e693424537b91ba7c3fcc96878 P 371a36d219164974abdf848841528152: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260130 02:58:27.410084 19159 leader_election.cc:304] T 9896ff9364744857968e87e80e776dd2 P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 371a36d219164974abdf848841528152; no voters: 78c2c46c7520492db35d51c244fb4f74, d870164263bb44b6b780a557daea3656
I20260130 02:58:27.411026 21347 raft_consensus.cc:2749] T 9896ff9364744857968e87e80e776dd2 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260130 02:58:27.428078 21294 leader_election.cc:290] T 9896ff9364744857968e87e80e776dd2 P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d870164263bb44b6b780a557daea3656 (127.18.100.67:39571), 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767)
I20260130 02:58:27.439828 19625 tablet_bootstrap.cc:492] T 475b435b2967418f927fa172d2ff04ea P d870164263bb44b6b780a557daea3656: Bootstrap starting.
I20260130 02:58:27.449762 19625 tablet_bootstrap.cc:654] T 475b435b2967418f927fa172d2ff04ea P d870164263bb44b6b780a557daea3656: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:27.451382 19379 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "30cc52136bb3422895d95c0fed0c6798" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74" is_pre_election: true
I20260130 02:58:27.451962 19379 raft_consensus.cc:2468] T 30cc52136bb3422895d95c0fed0c6798 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d870164263bb44b6b780a557daea3656 in term 0.
I20260130 02:58:27.453858 21348 tablet.cc:1722] T 25fd48e693424537b91ba7c3fcc96878 P 78c2c46c7520492db35d51c244fb4f74: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260130 02:58:27.483007 21211 consensus_queue.cc:1048] T 0d14fb95ace9447f810ccbed4d5befdf P d870164263bb44b6b780a557daea3656 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260130 02:58:27.500476 19625 tablet_bootstrap.cc:492] T 475b435b2967418f927fa172d2ff04ea P d870164263bb44b6b780a557daea3656: No bootstrap required, opened a new log
I20260130 02:58:27.500993 19625 ts_tablet_manager.cc:1403] T 475b435b2967418f927fa172d2ff04ea P d870164263bb44b6b780a557daea3656: Time spent bootstrapping tablet: real 0.061s user 0.011s sys 0.000s
I20260130 02:58:27.503306 19625 raft_consensus.cc:359] T 475b435b2967418f927fa172d2ff04ea P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:27.522140 21331 raft_consensus.cc:493] T 00000000000000000000000000000000 P 3749baa287ed4f249329a74071781edd [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 658299197a424f53a37f14fadf089ae4)
I20260130 02:58:27.522712 21331 raft_consensus.cc:515] T 00000000000000000000000000000000 P 3749baa287ed4f249329a74071781edd [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "658299197a424f53a37f14fadf089ae4" member_type: VOTER last_known_addr { host: "127.18.100.126" port: 38191 } } peers { permanent_uuid: "107f274cdda543038832f06a56ab9130" member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39711 } } peers { permanent_uuid: "3749baa287ed4f249329a74071781edd" member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34629 } }
I20260130 02:58:27.524544 21331 leader_election.cc:290] T 00000000000000000000000000000000 P 3749baa287ed4f249329a74071781edd [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 658299197a424f53a37f14fadf089ae4 (127.18.100.126:38191), 107f274cdda543038832f06a56ab9130 (127.18.100.125:39711)
I20260130 02:58:27.525395 18960 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3749baa287ed4f249329a74071781edd" candidate_term: 7 candidate_status { last_received { term: 6 index: 374 } } ignore_live_leader: false dest_uuid: "107f274cdda543038832f06a56ab9130" is_pre_election: true
I20260130 02:58:27.543642 19218 raft_consensus.cc:3060] T 00ddbfe6218c44d788fbf65245a31dc6 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:27.570434 19625 raft_consensus.cc:385] T 475b435b2967418f927fa172d2ff04ea P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:27.570797 19625 raft_consensus.cc:740] T 475b435b2967418f927fa172d2ff04ea P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d870164263bb44b6b780a557daea3656, State: Initialized, Role: FOLLOWER
I20260130 02:58:27.571435 19625 consensus_queue.cc:260] T 475b435b2967418f927fa172d2ff04ea P d870164263bb44b6b780a557daea3656 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:27.576220 19807 tablet_bootstrap.cc:654] T dd377fab8c544bf69121dc9e9a060b29 P 371a36d219164974abdf848841528152: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:27.584383 19462 leader_election.cc:304] T 8b47653c7c3945a68e9bf4680bfc0496 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78c2c46c7520492db35d51c244fb4f74, d870164263bb44b6b780a557daea3656; no voters:
I20260130 02:58:27.587388 21211 raft_consensus.cc:2804] T 8b47653c7c3945a68e9bf4680bfc0496 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260130 02:58:27.587778 21211 raft_consensus.cc:493] T 8b47653c7c3945a68e9bf4680bfc0496 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260130 02:58:27.588187 21211 raft_consensus.cc:3060] T 8b47653c7c3945a68e9bf4680bfc0496 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:27.602181 19223 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "30cc52136bb3422895d95c0fed0c6798" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152" is_pre_election: true
I20260130 02:58:27.618220 21292 raft_consensus.cc:493] T bc04be1284414a8699d3c4b88bf62813 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:27.618734 21292 raft_consensus.cc:515] T bc04be1284414a8699d3c4b88bf62813 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:27.621014 21292 leader_election.cc:290] T bc04be1284414a8699d3c4b88bf62813 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d870164263bb44b6b780a557daea3656 (127.18.100.67:39571), 371a36d219164974abdf848841528152 (127.18.100.65:42021)
I20260130 02:58:27.636783 19807 tablet_bootstrap.cc:492] T dd377fab8c544bf69121dc9e9a060b29 P 371a36d219164974abdf848841528152: No bootstrap required, opened a new log
I20260130 02:58:27.636783 19223 raft_consensus.cc:3060] T 39bcc41f10ae4241b370529364454586 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:27.637413 19807 ts_tablet_manager.cc:1403] T dd377fab8c544bf69121dc9e9a060b29 P 371a36d219164974abdf848841528152: Time spent bootstrapping tablet: real 0.270s user 0.014s sys 0.003s
W20260130 02:58:27.638238 18934 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.18.100.126:38191 timed out after 0.577s (SENT) [suppressed 6 similar messages]
I20260130 02:58:27.644836 21211 raft_consensus.cc:515] T 8b47653c7c3945a68e9bf4680bfc0496 P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:27.655633 19222 raft_consensus.cc:3060] T 9896ff9364744857968e87e80e776dd2 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:27.657322 19625 ts_tablet_manager.cc:1434] T 475b435b2967418f927fa172d2ff04ea P d870164263bb44b6b780a557daea3656: Time spent starting tablet: real 0.156s user 0.004s sys 0.000s
I20260130 02:58:27.658306 19625 tablet_bootstrap.cc:492] T 66a7487a7ab343a4abca5312afe78862 P d870164263bb44b6b780a557daea3656: Bootstrap starting.
I20260130 02:58:27.683235 21240 sys_catalog.cc:455] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 6 leader_uuid: "658299197a424f53a37f14fadf089ae4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "658299197a424f53a37f14fadf089ae4" member_type: VOTER last_known_addr { host: "127.18.100.126" port: 38191 } } peers { permanent_uuid: "107f274cdda543038832f06a56ab9130" member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39711 } } peers { permanent_uuid: "3749baa287ed4f249329a74071781edd" member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34629 } } }
I20260130 02:58:27.688860 21240 sys_catalog.cc:458] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4 [sys.catalog]: This master's current role is: LEADER
W20260130 02:58:27.704232 19314 leader_election.cc:343] T 30cc52136bb3422895d95c0fed0c6798 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:27.704632 19314 leader_election.cc:304] T 30cc52136bb3422895d95c0fed0c6798 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78c2c46c7520492db35d51c244fb4f74; no voters: 371a36d219164974abdf848841528152, d870164263bb44b6b780a557daea3656
I20260130 02:58:27.709444 21121 raft_consensus.cc:2749] T 30cc52136bb3422895d95c0fed0c6798 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260130 02:58:27.727227 18934 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.100.126:38191, user_credentials={real_user=slave}} blocked reactor thread for 89071us
W20260130 02:58:27.731047 19462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.100.66:40767, user_credentials={real_user=slave}} blocked reactor thread for 144618us
W20260130 02:58:27.734360 19462 consensus_peers.cc:597] T ab6afa40408a4f458495abf20290e55b P d870164263bb44b6b780a557daea3656 -> Peer 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767): Couldn't send request to peer 78c2c46c7520492db35d51c244fb4f74. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ab6afa40408a4f458495abf20290e55b. This is attempt 1: this message will repeat every 5th retry.
I20260130 02:58:27.727502 21211 leader_election.cc:290] T 8b47653c7c3945a68e9bf4680bfc0496 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 election: Requested vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767)
I20260130 02:58:27.741088 21331 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3749baa287ed4f249329a74071781edd [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 6 leader_uuid: "658299197a424f53a37f14fadf089ae4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "658299197a424f53a37f14fadf089ae4" member_type: VOTER last_known_addr { host: "127.18.100.126" port: 38191 } } peers { permanent_uuid: "107f274cdda543038832f06a56ab9130" member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39711 } } peers { permanent_uuid: "3749baa287ed4f249329a74071781edd" member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34629 } } }
I20260130 02:58:27.741806 21331 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3749baa287ed4f249329a74071781edd [sys.catalog]: This master's current role is: FOLLOWER
W20260130 02:58:27.749928 18861 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet 49efcfced0b74c48962393149ac2a500 (table default.Table_2e81f0a0cbc449a397141b717db17025 [id=a9c919f57c0c437cb35a6921103e45a6], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 49efcfced0b74c48962393149ac2a500 consensus state has no leader
I20260130 02:58:27.751066 21373 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3749baa287ed4f249329a74071781edd [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 6 leader_uuid: "658299197a424f53a37f14fadf089ae4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "658299197a424f53a37f14fadf089ae4" member_type: VOTER last_known_addr { host: "127.18.100.126" port: 38191 } } peers { permanent_uuid: "107f274cdda543038832f06a56ab9130" member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39711 } } peers { permanent_uuid: "3749baa287ed4f249329a74071781edd" member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34629 } } }
I20260130 02:58:27.751860 21373 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3749baa287ed4f249329a74071781edd [sys.catalog]: This master's current role is: FOLLOWER
W20260130 02:58:27.764619 19313 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.100.67:39571, user_credentials={real_user=slave}} blocked reactor thread for 60235.1us
W20260130 02:58:27.765218 19313 leader_election.cc:343] T 475b435b2967418f927fa172d2ff04ea P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:27.771055 19807 raft_consensus.cc:359] T dd377fab8c544bf69121dc9e9a060b29 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:27.775151 19807 raft_consensus.cc:385] T dd377fab8c544bf69121dc9e9a060b29 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:27.775480 19807 raft_consensus.cc:740] T dd377fab8c544bf69121dc9e9a060b29 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 371a36d219164974abdf848841528152, State: Initialized, Role: FOLLOWER
I20260130 02:58:27.778503 19625 tablet_bootstrap.cc:654] T 66a7487a7ab343a4abca5312afe78862 P d870164263bb44b6b780a557daea3656: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:27.789590 21121 consensus_queue.cc:1048] T 532e867fd5944d468d7672d2a5c92f3c P 78c2c46c7520492db35d51c244fb4f74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260130 02:58:27.776328 19807 consensus_queue.cc:260] T dd377fab8c544bf69121dc9e9a060b29 P 371a36d219164974abdf848841528152 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:27.796936 19807 ts_tablet_manager.cc:1434] T dd377fab8c544bf69121dc9e9a060b29 P 371a36d219164974abdf848841528152: Time spent starting tablet: real 0.159s user 0.009s sys 0.000s
I20260130 02:58:27.807617 19523 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bc04be1284414a8699d3c4b88bf62813" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
I20260130 02:58:27.808293 19523 raft_consensus.cc:2468] T bc04be1284414a8699d3c4b88bf62813 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78c2c46c7520492db35d51c244fb4f74 in term 0.
I20260130 02:58:27.809680 21358 raft_consensus.cc:493] T dd377fab8c544bf69121dc9e9a060b29 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:27.810113 21358 raft_consensus.cc:515] T dd377fab8c544bf69121dc9e9a060b29 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
W20260130 02:58:27.811759 19462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.100.66:40767, user_credentials={real_user=slave}} blocked reactor thread for 76640.5us
I20260130 02:58:27.812135 21358 leader_election.cc:290] T dd377fab8c544bf69121dc9e9a060b29 P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d870164263bb44b6b780a557daea3656 (127.18.100.67:39571), 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767)
I20260130 02:58:27.815289 21355 raft_consensus.cc:493] T 94355fc1338f479e8798070aff04fcba P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:27.815703 21355 raft_consensus.cc:515] T 94355fc1338f479e8798070aff04fcba P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:27.799007 19807 tablet_bootstrap.cc:492] T 6d1f121f49cd47c2b8247daaf6874e01 P 371a36d219164974abdf848841528152: Bootstrap starting.
I20260130 02:58:27.817559 21355 leader_election.cc:290] T 94355fc1338f479e8798070aff04fcba P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767), d870164263bb44b6b780a557daea3656 (127.18.100.67:39571)
I20260130 02:58:27.824957 19462 leader_election.cc:304] T 30cc52136bb3422895d95c0fed0c6798 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78c2c46c7520492db35d51c244fb4f74, d870164263bb44b6b780a557daea3656; no voters:
I20260130 02:58:27.825680 21364 raft_consensus.cc:2804] T 30cc52136bb3422895d95c0fed0c6798 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260130 02:58:27.826057 21364 raft_consensus.cc:493] T 30cc52136bb3422895d95c0fed0c6798 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260130 02:58:27.826392 21364 raft_consensus.cc:3060] T 30cc52136bb3422895d95c0fed0c6798 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Advancing to term 1
W20260130 02:58:27.827983 19314 leader_election.cc:343] T 475b435b2967418f927fa172d2ff04ea P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:27.828423 19314 leader_election.cc:304] T 475b435b2967418f927fa172d2ff04ea P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78c2c46c7520492db35d51c244fb4f74; no voters: 371a36d219164974abdf848841528152, d870164263bb44b6b780a557daea3656
I20260130 02:58:27.829111 21376 raft_consensus.cc:2749] T 475b435b2967418f927fa172d2ff04ea P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260130 02:58:27.834040 19370 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dd377fab8c544bf69121dc9e9a060b29" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74" is_pre_election: true
I20260130 02:58:27.835251 19378 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "94355fc1338f479e8798070aff04fcba" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74" is_pre_election: true
I20260130 02:58:27.835876 19378 raft_consensus.cc:2393] T 94355fc1338f479e8798070aff04fcba P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 371a36d219164974abdf848841528152 in current term 1: Already voted for candidate 78c2c46c7520492db35d51c244fb4f74 in this term.
I20260130 02:58:27.876305 21354 raft_consensus.cc:493] T 00ddbfe6218c44d788fbf65245a31dc6 P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d870164263bb44b6b780a557daea3656)
I20260130 02:58:27.876873 21354 raft_consensus.cc:515] T 00ddbfe6218c44d788fbf65245a31dc6 P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:27.887571 21354 leader_election.cc:290] T 00ddbfe6218c44d788fbf65245a31dc6 P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767), d870164263bb44b6b780a557daea3656 (127.18.100.67:39571)
I20260130 02:58:27.888499 19378 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00ddbfe6218c44d788fbf65245a31dc6" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74" is_pre_election: true
I20260130 02:58:27.890625 19218 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bc04be1284414a8699d3c4b88bf62813" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152" is_pre_election: true
W20260130 02:58:27.894201 19314 consensus_peers.cc:597] T 532e867fd5944d468d7672d2a5c92f3c P 78c2c46c7520492db35d51c244fb4f74 -> Peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Couldn't send request to peer 371a36d219164974abdf848841528152. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260130 02:58:27.901657 21376 raft_consensus.cc:493] T 3738385130c044e0a5a5803da74c567d P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260130 02:58:27.902225 19314 leader_election.cc:343] T bc04be1284414a8699d3c4b88bf62813 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:27.902134 21376 raft_consensus.cc:515] T 3738385130c044e0a5a5803da74c567d P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:27.903992 21376 leader_election.cc:290] T 3738385130c044e0a5a5803da74c567d P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), d870164263bb44b6b780a557daea3656 (127.18.100.67:39571)
I20260130 02:58:27.909803 19807 tablet_bootstrap.cc:654] T 6d1f121f49cd47c2b8247daaf6874e01 P 371a36d219164974abdf848841528152: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:27.913256 19218 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8b47653c7c3945a68e9bf4680bfc0496" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152"
I20260130 02:58:27.926501 19218 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3738385130c044e0a5a5803da74c567d" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152" is_pre_election: true
W20260130 02:58:27.927691 19314 leader_election.cc:343] T 3738385130c044e0a5a5803da74c567d P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:27.890475 19625 tablet_bootstrap.cc:492] T 66a7487a7ab343a4abca5312afe78862 P d870164263bb44b6b780a557daea3656: No bootstrap required, opened a new log
I20260130 02:58:27.939078 19625 ts_tablet_manager.cc:1403] T 66a7487a7ab343a4abca5312afe78862 P d870164263bb44b6b780a557daea3656: Time spent bootstrapping tablet: real 0.281s user 0.010s sys 0.000s
I20260130 02:58:27.941325 19625 raft_consensus.cc:359] T 66a7487a7ab343a4abca5312afe78862 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:27.941941 19625 raft_consensus.cc:385] T 66a7487a7ab343a4abca5312afe78862 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:27.942183 19625 raft_consensus.cc:740] T 66a7487a7ab343a4abca5312afe78862 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d870164263bb44b6b780a557daea3656, State: Initialized, Role: FOLLOWER
W20260130 02:58:27.958132 19462 outbound_call.cc:321] RPC callback for RPC call kudu.master.MasterService.TSHeartbeat -> {remote=127.18.100.124:34629, user_credentials={real_user=slave}} blocked reactor thread for 85100.4us
I20260130 02:58:27.965422 19515 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dd377fab8c544bf69121dc9e9a060b29" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
I20260130 02:58:27.834169 21364 raft_consensus.cc:515] T 30cc52136bb3422895d95c0fed0c6798 P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:27.987757 19523 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "94355fc1338f479e8798070aff04fcba" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
I20260130 02:58:27.993352 19523 raft_consensus.cc:2393] T 94355fc1338f479e8798070aff04fcba P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 371a36d219164974abdf848841528152 in current term 1: Already voted for candidate d870164263bb44b6b780a557daea3656 in this term.
I20260130 02:58:28.005463 21364 leader_election.cc:290] T 30cc52136bb3422895d95c0fed0c6798 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 election: Requested vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767)
I20260130 02:58:28.024493 19379 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8b47653c7c3945a68e9bf4680bfc0496" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74"
I20260130 02:58:28.025074 19379 raft_consensus.cc:3060] T 8b47653c7c3945a68e9bf4680bfc0496 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:28.032899 19379 raft_consensus.cc:2468] T 8b47653c7c3945a68e9bf4680bfc0496 P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d870164263bb44b6b780a557daea3656 in term 1.
I20260130 02:58:28.042264 21353 raft_consensus.cc:493] T 0d14fb95ace9447f810ccbed4d5befdf P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:28.029047 19519 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00ddbfe6218c44d788fbf65245a31dc6" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
W20260130 02:58:28.054836 20613 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.18.100.126:38191 timed out after 0.530s (SENT)
I20260130 02:58:28.057293 19625 consensus_queue.cc:260] T 66a7487a7ab343a4abca5312afe78862 P d870164263bb44b6b780a557daea3656 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:28.059623 19625 ts_tablet_manager.cc:1434] T 66a7487a7ab343a4abca5312afe78862 P d870164263bb44b6b780a557daea3656: Time spent starting tablet: real 0.120s user 0.005s sys 0.000s
I20260130 02:58:28.060500 19625 tablet_bootstrap.cc:492] T 155ff0c35fc44cb292a28407e6ed937d P d870164263bb44b6b780a557daea3656: Bootstrap starting.
I20260130 02:58:28.042793 21353 raft_consensus.cc:515] T 0d14fb95ace9447f810ccbed4d5befdf P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:28.032289 21382 tablet.cc:1722] T 25fd48e693424537b91ba7c3fcc96878 P d870164263bb44b6b780a557daea3656: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260130 02:58:28.082015 21353 leader_election.cc:290] T 0d14fb95ace9447f810ccbed4d5befdf P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d870164263bb44b6b780a557daea3656 (127.18.100.67:39571), 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767)
I20260130 02:58:28.083019 19807 tablet_bootstrap.cc:492] T 6d1f121f49cd47c2b8247daaf6874e01 P 371a36d219164974abdf848841528152: No bootstrap required, opened a new log
I20260130 02:58:28.083434 19807 ts_tablet_manager.cc:1403] T 6d1f121f49cd47c2b8247daaf6874e01 P 371a36d219164974abdf848841528152: Time spent bootstrapping tablet: real 0.286s user 0.012s sys 0.005s
I20260130 02:58:28.093834 19807 raft_consensus.cc:359] T 6d1f121f49cd47c2b8247daaf6874e01 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:28.095283 19807 raft_consensus.cc:385] T 6d1f121f49cd47c2b8247daaf6874e01 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:28.095718 19807 raft_consensus.cc:740] T 6d1f121f49cd47c2b8247daaf6874e01 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 371a36d219164974abdf848841528152, State: Initialized, Role: FOLLOWER
I20260130 02:58:28.096593 19807 consensus_queue.cc:260] T 6d1f121f49cd47c2b8247daaf6874e01 P 371a36d219164974abdf848841528152 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:28.098565 19378 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "30cc52136bb3422895d95c0fed0c6798" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74"
I20260130 02:58:28.099064 19378 raft_consensus.cc:3060] T 30cc52136bb3422895d95c0fed0c6798 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:28.119392 19366 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0d14fb95ace9447f810ccbed4d5befdf" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74" is_pre_election: true
I20260130 02:58:28.126413 19222 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "30cc52136bb3422895d95c0fed0c6798" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152"
I20260130 02:58:28.128500 19519 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3738385130c044e0a5a5803da74c567d" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
I20260130 02:58:28.129129 19519 raft_consensus.cc:2468] T 3738385130c044e0a5a5803da74c567d P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78c2c46c7520492db35d51c244fb4f74 in term 0.
I20260130 02:58:28.130720 19159 leader_election.cc:304] T dd377fab8c544bf69121dc9e9a060b29 P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 371a36d219164974abdf848841528152; no voters: 78c2c46c7520492db35d51c244fb4f74, d870164263bb44b6b780a557daea3656
I20260130 02:58:28.131835 21354 raft_consensus.cc:2749] T dd377fab8c544bf69121dc9e9a060b29 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260130 02:58:28.132537 19159 leader_election.cc:304] T 94355fc1338f479e8798070aff04fcba P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 371a36d219164974abdf848841528152; no voters: 78c2c46c7520492db35d51c244fb4f74, d870164263bb44b6b780a557daea3656
I20260130 02:58:28.133179 21353 raft_consensus.cc:3060] T 94355fc1338f479e8798070aff04fcba P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:28.140874 21353 raft_consensus.cc:2749] T 94355fc1338f479e8798070aff04fcba P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260130 02:58:28.141218 19807 ts_tablet_manager.cc:1434] T 6d1f121f49cd47c2b8247daaf6874e01 P 371a36d219164974abdf848841528152: Time spent starting tablet: real 0.058s user 0.008s sys 0.000s
I20260130 02:58:28.142201 19807 tablet_bootstrap.cc:492] T 1129f72158ec4bae89da3c94cdb70839 P 371a36d219164974abdf848841528152: Bootstrap starting.
I20260130 02:58:28.192349 19378 raft_consensus.cc:2468] T 30cc52136bb3422895d95c0fed0c6798 P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d870164263bb44b6b780a557daea3656 in term 1.
W20260130 02:58:28.197160 19463 consensus_peers.cc:597] T 0d14fb95ace9447f810ccbed4d5befdf P d870164263bb44b6b780a557daea3656 -> Peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Couldn't send request to peer 371a36d219164974abdf848841528152. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
W20260130 02:58:28.200045 19463 leader_election.cc:343] T 8b47653c7c3945a68e9bf4680bfc0496 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Illegal state: must be running to vote when last-logged opid is not known
W20260130 02:58:28.201681 19463 consensus_peers.cc:597] T 1129f72158ec4bae89da3c94cdb70839 P d870164263bb44b6b780a557daea3656 -> Peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Couldn't send request to peer 371a36d219164974abdf848841528152. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260130 02:58:28.210278 19807 tablet_bootstrap.cc:654] T 1129f72158ec4bae89da3c94cdb70839 P 371a36d219164974abdf848841528152: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:28.217998 19462 leader_election.cc:304] T 8b47653c7c3945a68e9bf4680bfc0496 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78c2c46c7520492db35d51c244fb4f74, d870164263bb44b6b780a557daea3656; no voters:
I20260130 02:58:28.240628 21364 raft_consensus.cc:2804] T 8b47653c7c3945a68e9bf4680bfc0496 P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Leader election won for term 1
I20260130 02:58:28.241078 21364 raft_consensus.cc:697] T 8b47653c7c3945a68e9bf4680bfc0496 P d870164263bb44b6b780a557daea3656 [term 1 LEADER]: Becoming Leader. State: Replica: d870164263bb44b6b780a557daea3656, State: Running, Role: LEADER
I20260130 02:58:28.241825 21364 consensus_queue.cc:237] T 8b47653c7c3945a68e9bf4680bfc0496 P d870164263bb44b6b780a557daea3656 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:28.256290 19625 tablet_bootstrap.cc:654] T 155ff0c35fc44cb292a28407e6ed937d P d870164263bb44b6b780a557daea3656: Neither blocks nor log segments found. Creating new log.
W20260130 02:58:28.267021 19463 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.100.65:42021, user_credentials={real_user=slave}} blocked reactor thread for 65418.2us
W20260130 02:58:28.270896 19463 consensus_peers.cc:597] T 6d1f121f49cd47c2b8247daaf6874e01 P d870164263bb44b6b780a557daea3656 -> Peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Couldn't send request to peer 371a36d219164974abdf848841528152. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20260130 02:58:28.306342 18934 leader_election.cc:336] T 00000000000000000000000000000000 P 107f274cdda543038832f06a56ab9130 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 658299197a424f53a37f14fadf089ae4 (127.18.100.126:38191): Timed out: RequestConsensusVote RPC to 127.18.100.126:38191 timed out after 0.577s (SENT)
I20260130 02:58:28.306833 18934 leader_election.cc:304] T 00000000000000000000000000000000 P 107f274cdda543038832f06a56ab9130 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 107f274cdda543038832f06a56ab9130; no voters: 3749baa287ed4f249329a74071781edd, 658299197a424f53a37f14fadf089ae4
I20260130 02:58:28.308185 19807 tablet_bootstrap.cc:492] T 1129f72158ec4bae89da3c94cdb70839 P 371a36d219164974abdf848841528152: No bootstrap required, opened a new log
I20260130 02:58:28.308588 19807 ts_tablet_manager.cc:1403] T 1129f72158ec4bae89da3c94cdb70839 P 371a36d219164974abdf848841528152: Time spent bootstrapping tablet: real 0.167s user 0.009s sys 0.002s
I20260130 02:58:28.310781 19807 raft_consensus.cc:359] T 1129f72158ec4bae89da3c94cdb70839 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:28.311379 19807 raft_consensus.cc:385] T 1129f72158ec4bae89da3c94cdb70839 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:28.311633 19807 raft_consensus.cc:740] T 1129f72158ec4bae89da3c94cdb70839 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 371a36d219164974abdf848841528152, State: Initialized, Role: FOLLOWER
I20260130 02:58:28.312196 19807 consensus_queue.cc:260] T 1129f72158ec4bae89da3c94cdb70839 P 371a36d219164974abdf848841528152 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:28.317442 19313 leader_election.cc:304] T bc04be1284414a8699d3c4b88bf62813 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 78c2c46c7520492db35d51c244fb4f74, d870164263bb44b6b780a557daea3656; no voters: 371a36d219164974abdf848841528152
I20260130 02:58:28.318261 21360 raft_consensus.cc:2804] T bc04be1284414a8699d3c4b88bf62813 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260130 02:58:28.318841 21360 raft_consensus.cc:493] T bc04be1284414a8699d3c4b88bf62813 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260130 02:58:28.319211 21360 raft_consensus.cc:3060] T bc04be1284414a8699d3c4b88bf62813 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:28.334234 21393 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 107f274cdda543038832f06a56ab9130 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260130 02:58:28.314412 19807 ts_tablet_manager.cc:1434] T 1129f72158ec4bae89da3c94cdb70839 P 371a36d219164974abdf848841528152: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20260130 02:58:28.343279 19807 tablet_bootstrap.cc:492] T bc04be1284414a8699d3c4b88bf62813 P 371a36d219164974abdf848841528152: Bootstrap starting.
W20260130 02:58:28.275182 19463 leader_election.cc:343] T 30cc52136bb3422895d95c0fed0c6798 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:28.348498 19807 tablet_bootstrap.cc:654] T bc04be1284414a8699d3c4b88bf62813 P 371a36d219164974abdf848841528152: Neither blocks nor log segments found. Creating new log.
W20260130 02:58:28.349112 19463 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.100.65:42021, user_credentials={real_user=slave}} blocked reactor thread for 73956.5us
W20260130 02:58:28.377867 19313 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.100.67:39571, user_credentials={real_user=slave}} blocked reactor thread for 43868.4us
I20260130 02:58:28.384447 19807 tablet_bootstrap.cc:492] T bc04be1284414a8699d3c4b88bf62813 P 371a36d219164974abdf848841528152: No bootstrap required, opened a new log
I20260130 02:58:28.384858 19807 ts_tablet_manager.cc:1403] T bc04be1284414a8699d3c4b88bf62813 P 371a36d219164974abdf848841528152: Time spent bootstrapping tablet: real 0.042s user 0.010s sys 0.001s
I20260130 02:58:28.387113 19807 raft_consensus.cc:359] T bc04be1284414a8699d3c4b88bf62813 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:28.387724 19807 raft_consensus.cc:385] T bc04be1284414a8699d3c4b88bf62813 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:28.387977 19807 raft_consensus.cc:740] T bc04be1284414a8699d3c4b88bf62813 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 371a36d219164974abdf848841528152, State: Initialized, Role: FOLLOWER
I20260130 02:58:28.389770 21371 raft_consensus.cc:493] T 6f5a90b75a0f4851b5db672c41253f7b P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:28.388528 19807 consensus_queue.cc:260] T bc04be1284414a8699d3c4b88bf62813 P 371a36d219164974abdf848841528152 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:28.383344 19625 tablet_bootstrap.cc:492] T 155ff0c35fc44cb292a28407e6ed937d P d870164263bb44b6b780a557daea3656: No bootstrap required, opened a new log
I20260130 02:58:28.390321 21371 raft_consensus.cc:515] T 6f5a90b75a0f4851b5db672c41253f7b P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:28.391680 19807 ts_tablet_manager.cc:1434] T bc04be1284414a8699d3c4b88bf62813 P 371a36d219164974abdf848841528152: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260130 02:58:28.393261 19222 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6f5a90b75a0f4851b5db672c41253f7b" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152" is_pre_election: true
W20260130 02:58:28.394658 19314 leader_election.cc:343] T 6f5a90b75a0f4851b5db672c41253f7b P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:28.395145 21371 leader_election.cc:290] T 6f5a90b75a0f4851b5db672c41253f7b P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d870164263bb44b6b780a557daea3656 (127.18.100.67:39571), 371a36d219164974abdf848841528152 (127.18.100.65:42021)
I20260130 02:58:28.398509 19625 ts_tablet_manager.cc:1403] T 155ff0c35fc44cb292a28407e6ed937d P d870164263bb44b6b780a557daea3656: Time spent bootstrapping tablet: real 0.337s user 0.012s sys 0.002s
I20260130 02:58:28.408969 19313 leader_election.cc:304] T 3738385130c044e0a5a5803da74c567d P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 78c2c46c7520492db35d51c244fb4f74, d870164263bb44b6b780a557daea3656; no voters: 371a36d219164974abdf848841528152
I20260130 02:58:28.410125 21372 raft_consensus.cc:2804] T 3738385130c044e0a5a5803da74c567d P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260130 02:58:28.410434 21372 raft_consensus.cc:493] T 3738385130c044e0a5a5803da74c567d P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260130 02:58:28.410830 21372 raft_consensus.cc:3060] T 3738385130c044e0a5a5803da74c567d P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:28.424607 21372 raft_consensus.cc:515] T 3738385130c044e0a5a5803da74c567d P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:28.427389 19222 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3738385130c044e0a5a5803da74c567d" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152"
W20260130 02:58:28.428704 19314 leader_election.cc:343] T 3738385130c044e0a5a5803da74c567d P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:28.439924 19625 raft_consensus.cc:359] T 155ff0c35fc44cb292a28407e6ed937d P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:28.440587 19625 raft_consensus.cc:385] T 155ff0c35fc44cb292a28407e6ed937d P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:28.440848 19625 raft_consensus.cc:740] T 155ff0c35fc44cb292a28407e6ed937d P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d870164263bb44b6b780a557daea3656, State: Initialized, Role: FOLLOWER
I20260130 02:58:28.445932 19807 tablet_bootstrap.cc:492] T 8b47653c7c3945a68e9bf4680bfc0496 P 371a36d219164974abdf848841528152: Bootstrap starting.
I20260130 02:58:28.451205 19807 tablet_bootstrap.cc:654] T 8b47653c7c3945a68e9bf4680bfc0496 P 371a36d219164974abdf848841528152: Neither blocks nor log segments found. Creating new log.
W20260130 02:58:28.454836 19463 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.100.65:42021, user_credentials={real_user=slave}} blocked reactor thread for 69176.5us
W20260130 02:58:28.463215 19463 leader_election.cc:343] T 8b47653c7c3945a68e9bf4680bfc0496 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:28.441417 19625 consensus_queue.cc:260] T 155ff0c35fc44cb292a28407e6ed937d P d870164263bb44b6b780a557daea3656 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:28.472451 19625 ts_tablet_manager.cc:1434] T 155ff0c35fc44cb292a28407e6ed937d P d870164263bb44b6b780a557daea3656: Time spent starting tablet: real 0.065s user 0.006s sys 0.000s
W20260130 02:58:28.483100 19313 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.100.67:39571, user_credentials={real_user=slave}} blocked reactor thread for 74199us
I20260130 02:58:28.329809 21360 raft_consensus.cc:515] T bc04be1284414a8699d3c4b88bf62813 P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:28.497107 19222 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bc04be1284414a8699d3c4b88bf62813" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152"
I20260130 02:58:28.497735 19222 raft_consensus.cc:3060] T bc04be1284414a8699d3c4b88bf62813 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:28.514626 19462 leader_election.cc:304] T 30cc52136bb3422895d95c0fed0c6798 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78c2c46c7520492db35d51c244fb4f74, d870164263bb44b6b780a557daea3656; no voters:
I20260130 02:58:28.515717 21361 raft_consensus.cc:2804] T 30cc52136bb3422895d95c0fed0c6798 P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Leader election won for term 1
I20260130 02:58:28.516158 21361 raft_consensus.cc:697] T 30cc52136bb3422895d95c0fed0c6798 P d870164263bb44b6b780a557daea3656 [term 1 LEADER]: Becoming Leader. State: Replica: d870164263bb44b6b780a557daea3656, State: Running, Role: LEADER
I20260130 02:58:28.516925 21361 consensus_queue.cc:237] T 30cc52136bb3422895d95c0fed0c6798 P d870164263bb44b6b780a557daea3656 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:28.523342 19625 tablet_bootstrap.cc:492] T 6f5a90b75a0f4851b5db672c41253f7b P d870164263bb44b6b780a557daea3656: Bootstrap starting.
I20260130 02:58:28.524523 21372 leader_election.cc:290] T 3738385130c044e0a5a5803da74c567d P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 election: Requested vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), d870164263bb44b6b780a557daea3656 (127.18.100.67:39571)
I20260130 02:58:28.528631 19625 tablet_bootstrap.cc:654] T 6f5a90b75a0f4851b5db672c41253f7b P d870164263bb44b6b780a557daea3656: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:28.504817 19222 raft_consensus.cc:2468] T bc04be1284414a8699d3c4b88bf62813 P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78c2c46c7520492db35d51c244fb4f74 in term 1.
I20260130 02:58:28.532562 19314 leader_election.cc:304] T bc04be1284414a8699d3c4b88bf62813 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 371a36d219164974abdf848841528152, 78c2c46c7520492db35d51c244fb4f74; no voters:
I20260130 02:58:28.534168 21371 raft_consensus.cc:2804] T bc04be1284414a8699d3c4b88bf62813 P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Leader election won for term 1
I20260130 02:58:28.534868 21371 raft_consensus.cc:697] T bc04be1284414a8699d3c4b88bf62813 P 78c2c46c7520492db35d51c244fb4f74 [term 1 LEADER]: Becoming Leader. State: Replica: 78c2c46c7520492db35d51c244fb4f74, State: Running, Role: LEADER
I20260130 02:58:28.535687 21371 consensus_queue.cc:237] T bc04be1284414a8699d3c4b88bf62813 P 78c2c46c7520492db35d51c244fb4f74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:28.542459 21360 leader_election.cc:290] T bc04be1284414a8699d3c4b88bf62813 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 election: Requested vote from peers d870164263bb44b6b780a557daea3656 (127.18.100.67:39571), 371a36d219164974abdf848841528152 (127.18.100.65:42021)
I20260130 02:58:28.547029 19625 tablet_bootstrap.cc:492] T 6f5a90b75a0f4851b5db672c41253f7b P d870164263bb44b6b780a557daea3656: No bootstrap required, opened a new log
I20260130 02:58:28.547435 19625 ts_tablet_manager.cc:1403] T 6f5a90b75a0f4851b5db672c41253f7b P d870164263bb44b6b780a557daea3656: Time spent bootstrapping tablet: real 0.024s user 0.008s sys 0.003s
I20260130 02:58:28.550395 21211 raft_consensus.cc:493] T 475b435b2967418f927fa172d2ff04ea P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:28.550844 21211 raft_consensus.cc:515] T 475b435b2967418f927fa172d2ff04ea P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:28.552557 21211 leader_election.cc:290] T 475b435b2967418f927fa172d2ff04ea P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767)
I20260130 02:58:28.549647 19625 raft_consensus.cc:359] T 6f5a90b75a0f4851b5db672c41253f7b P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
W20260130 02:58:28.557884 19463 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.100.65:42021, user_credentials={real_user=slave}} blocked reactor thread for 93849.8us
W20260130 02:58:28.559645 19463 consensus_peers.cc:597] T e691c238199b4914bcc3fd248f31420c P d870164263bb44b6b780a557daea3656 -> Peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Couldn't send request to peer 371a36d219164974abdf848841528152. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20260130 02:58:28.561811 19463 leader_election.cc:343] T 30cc52136bb3422895d95c0fed0c6798 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:28.563014 19159 leader_election.cc:304] T 00ddbfe6218c44d788fbf65245a31dc6 P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 371a36d219164974abdf848841528152; no voters: 78c2c46c7520492db35d51c244fb4f74, d870164263bb44b6b780a557daea3656
I20260130 02:58:28.563794 21169 raft_consensus.cc:2749] T 00ddbfe6218c44d788fbf65245a31dc6 P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260130 02:58:28.569947 19807 tablet_bootstrap.cc:492] T 8b47653c7c3945a68e9bf4680bfc0496 P 371a36d219164974abdf848841528152: No bootstrap required, opened a new log
I20260130 02:58:28.557775 19625 raft_consensus.cc:385] T 6f5a90b75a0f4851b5db672c41253f7b P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:28.574831 19625 raft_consensus.cc:740] T 6f5a90b75a0f4851b5db672c41253f7b P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d870164263bb44b6b780a557daea3656, State: Initialized, Role: FOLLOWER
I20260130 02:58:28.572664 19807 ts_tablet_manager.cc:1403] T 8b47653c7c3945a68e9bf4680bfc0496 P 371a36d219164974abdf848841528152: Time spent bootstrapping tablet: real 0.127s user 0.016s sys 0.000s
I20260130 02:58:28.575409 19625 consensus_queue.cc:260] T 6f5a90b75a0f4851b5db672c41253f7b P d870164263bb44b6b780a557daea3656 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:28.577490 19625 ts_tablet_manager.cc:1434] T 6f5a90b75a0f4851b5db672c41253f7b P d870164263bb44b6b780a557daea3656: Time spent starting tablet: real 0.030s user 0.003s sys 0.000s
I20260130 02:58:28.586078 19625 tablet_bootstrap.cc:492] T 920f52b975ab480fbf42656171222f09 P d870164263bb44b6b780a557daea3656: Bootstrap starting.
W20260130 02:58:28.601295 20613 leader_election.cc:336] T 00000000000000000000000000000000 P 3749baa287ed4f249329a74071781edd [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 658299197a424f53a37f14fadf089ae4 (127.18.100.126:38191): Timed out: RequestConsensusVote RPC to 127.18.100.126:38191 timed out after 0.530s (SENT)
I20260130 02:58:28.601733 20613 leader_election.cc:304] T 00000000000000000000000000000000 P 3749baa287ed4f249329a74071781edd [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3749baa287ed4f249329a74071781edd; no voters: 107f274cdda543038832f06a56ab9130, 658299197a424f53a37f14fadf089ae4
I20260130 02:58:28.605713 19625 tablet_bootstrap.cc:654] T 920f52b975ab480fbf42656171222f09 P d870164263bb44b6b780a557daea3656: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:28.612105 19519 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6f5a90b75a0f4851b5db672c41253f7b" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
I20260130 02:58:28.612721 19519 raft_consensus.cc:2468] T 6f5a90b75a0f4851b5db672c41253f7b P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78c2c46c7520492db35d51c244fb4f74 in term 0.
I20260130 02:58:28.614543 19519 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bc04be1284414a8699d3c4b88bf62813" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656"
I20260130 02:58:28.615058 19519 raft_consensus.cc:3060] T bc04be1284414a8699d3c4b88bf62813 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:28.637935 19625 tablet_bootstrap.cc:492] T 920f52b975ab480fbf42656171222f09 P d870164263bb44b6b780a557daea3656: No bootstrap required, opened a new log
I20260130 02:58:28.638365 19625 ts_tablet_manager.cc:1403] T 920f52b975ab480fbf42656171222f09 P d870164263bb44b6b780a557daea3656: Time spent bootstrapping tablet: real 0.052s user 0.006s sys 0.004s
I20260130 02:58:28.640858 19625 raft_consensus.cc:359] T 920f52b975ab480fbf42656171222f09 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:28.641475 19625 raft_consensus.cc:385] T 920f52b975ab480fbf42656171222f09 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:28.641733 19625 raft_consensus.cc:740] T 920f52b975ab480fbf42656171222f09 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d870164263bb44b6b780a557daea3656, State: Initialized, Role: FOLLOWER
I20260130 02:58:28.642292 19625 consensus_queue.cc:260] T 920f52b975ab480fbf42656171222f09 P d870164263bb44b6b780a557daea3656 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:28.644316 19625 ts_tablet_manager.cc:1434] T 920f52b975ab480fbf42656171222f09 P d870164263bb44b6b780a557daea3656: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
W20260130 02:58:28.650928 20613 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.100.126:38191, user_credentials={real_user=slave}} blocked reactor thread for 49671.6us
I20260130 02:58:28.659358 19222 raft_consensus.cc:3060] T 6d1f121f49cd47c2b8247daaf6874e01 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:28.662313 21400 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 3749baa287ed4f249329a74071781edd [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260130 02:58:28.681219 19224 raft_consensus.cc:3060] T 0d14fb95ace9447f810ccbed4d5befdf P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:28.685477 19223 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "475b435b2967418f927fa172d2ff04ea" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152" is_pre_election: true
I20260130 02:58:28.697665 21361 raft_consensus.cc:493] T 155ff0c35fc44cb292a28407e6ed937d P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:28.698125 21361 raft_consensus.cc:515] T 155ff0c35fc44cb292a28407e6ed937d P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:28.702751 21361 leader_election.cc:290] T 155ff0c35fc44cb292a28407e6ed937d P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767)
I20260130 02:58:28.704097 19519 raft_consensus.cc:2468] T bc04be1284414a8699d3c4b88bf62813 P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78c2c46c7520492db35d51c244fb4f74 in term 1.
I20260130 02:58:28.686733 19807 raft_consensus.cc:359] T 8b47653c7c3945a68e9bf4680bfc0496 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:28.710651 19807 raft_consensus.cc:385] T 8b47653c7c3945a68e9bf4680bfc0496 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:28.720155 19366 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "475b435b2967418f927fa172d2ff04ea" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74" is_pre_election: true
I20260130 02:58:28.720738 19366 raft_consensus.cc:2468] T 475b435b2967418f927fa172d2ff04ea P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d870164263bb44b6b780a557daea3656 in term 0.
I20260130 02:58:28.718091 19807 raft_consensus.cc:740] T 8b47653c7c3945a68e9bf4680bfc0496 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 371a36d219164974abdf848841528152, State: Initialized, Role: FOLLOWER
I20260130 02:58:28.730088 21375 raft_consensus.cc:493] T 66a7487a7ab343a4abca5312afe78862 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:28.730521 21375 raft_consensus.cc:515] T 66a7487a7ab343a4abca5312afe78862 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:28.732373 21375 leader_election.cc:290] T 66a7487a7ab343a4abca5312afe78862 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767)
I20260130 02:58:28.733042 19519 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0d14fb95ace9447f810ccbed4d5befdf" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
I20260130 02:58:28.745728 19366 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "155ff0c35fc44cb292a28407e6ed937d" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74" is_pre_election: true
I20260130 02:58:28.746326 19366 raft_consensus.cc:2468] T 155ff0c35fc44cb292a28407e6ed937d P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d870164263bb44b6b780a557daea3656 in term 0.
I20260130 02:58:28.744133 19807 consensus_queue.cc:260] T 8b47653c7c3945a68e9bf4680bfc0496 P 371a36d219164974abdf848841528152 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:28.783602 19807 ts_tablet_manager.cc:1434] T 8b47653c7c3945a68e9bf4680bfc0496 P 371a36d219164974abdf848841528152: Time spent starting tablet: real 0.206s user 0.004s sys 0.012s
I20260130 02:58:28.804616 19378 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "66a7487a7ab343a4abca5312afe78862" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74" is_pre_election: true
I20260130 02:58:28.805214 19378 raft_consensus.cc:2468] T 66a7487a7ab343a4abca5312afe78862 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d870164263bb44b6b780a557daea3656 in term 0.
I20260130 02:58:28.804455 19807 tablet_bootstrap.cc:492] T 532e867fd5944d468d7672d2a5c92f3c P 371a36d219164974abdf848841528152: Bootstrap starting.
I20260130 02:58:28.833601 19523 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3738385130c044e0a5a5803da74c567d" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656"
I20260130 02:58:28.834107 19523 raft_consensus.cc:3060] T 3738385130c044e0a5a5803da74c567d P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:28.846408 21372 raft_consensus.cc:493] T 920f52b975ab480fbf42656171222f09 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:28.846915 21372 raft_consensus.cc:515] T 920f52b975ab480fbf42656171222f09 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:28.849850 21372 leader_election.cc:290] T 920f52b975ab480fbf42656171222f09 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d870164263bb44b6b780a557daea3656 (127.18.100.67:39571), 371a36d219164974abdf848841528152 (127.18.100.65:42021)
I20260130 02:58:28.850041 19223 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "920f52b975ab480fbf42656171222f09" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152" is_pre_election: true
W20260130 02:58:28.851089 19314 leader_election.cc:343] T 920f52b975ab480fbf42656171222f09 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:28.855238 21375 consensus_queue.cc:1048] T e691c238199b4914bcc3fd248f31420c P d870164263bb44b6b780a557daea3656 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260130 02:58:28.875080 19523 raft_consensus.cc:2468] T 3738385130c044e0a5a5803da74c567d P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78c2c46c7520492db35d51c244fb4f74 in term 1.
I20260130 02:58:28.902910 19462 leader_election.cc:304] T 475b435b2967418f927fa172d2ff04ea P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78c2c46c7520492db35d51c244fb4f74, d870164263bb44b6b780a557daea3656; no voters:
I20260130 02:58:28.903656 21361 raft_consensus.cc:2804] T 475b435b2967418f927fa172d2ff04ea P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260130 02:58:28.903977 21361 raft_consensus.cc:493] T 475b435b2967418f927fa172d2ff04ea P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260130 02:58:28.904285 21361 raft_consensus.cc:3060] T 475b435b2967418f927fa172d2ff04ea P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:28.915504 19462 leader_election.cc:304] T 155ff0c35fc44cb292a28407e6ed937d P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78c2c46c7520492db35d51c244fb4f74, d870164263bb44b6b780a557daea3656; no voters:
I20260130 02:58:28.916193 21375 raft_consensus.cc:2804] T 155ff0c35fc44cb292a28407e6ed937d P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260130 02:58:28.916543 21375 raft_consensus.cc:493] T 155ff0c35fc44cb292a28407e6ed937d P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260130 02:58:28.916855 21375 raft_consensus.cc:3060] T 155ff0c35fc44cb292a28407e6ed937d P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:28.918748 19222 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "155ff0c35fc44cb292a28407e6ed937d" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152" is_pre_election: true
I20260130 02:58:28.922142 19223 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "66a7487a7ab343a4abca5312afe78862" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152" is_pre_election: true
I20260130 02:58:28.923394 19159 leader_election.cc:304] T 0d14fb95ace9447f810ccbed4d5befdf P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 371a36d219164974abdf848841528152; no voters: 78c2c46c7520492db35d51c244fb4f74, d870164263bb44b6b780a557daea3656
I20260130 02:58:28.924289 21353 raft_consensus.cc:2749] T 0d14fb95ace9447f810ccbed4d5befdf P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260130 02:58:28.905768 19807 tablet_bootstrap.cc:654] T 532e867fd5944d468d7672d2a5c92f3c P 371a36d219164974abdf848841528152: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:28.929349 21375 raft_consensus.cc:515] T 155ff0c35fc44cb292a28407e6ed937d P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:28.954492 21361 raft_consensus.cc:515] T 475b435b2967418f927fa172d2ff04ea P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:28.963070 19462 leader_election.cc:304] T 66a7487a7ab343a4abca5312afe78862 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78c2c46c7520492db35d51c244fb4f74, d870164263bb44b6b780a557daea3656; no voters:
I20260130 02:58:28.963752 21211 raft_consensus.cc:2804] T 66a7487a7ab343a4abca5312afe78862 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260130 02:58:28.964062 21211 raft_consensus.cc:493] T 66a7487a7ab343a4abca5312afe78862 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260130 02:58:28.964373 21211 raft_consensus.cc:3060] T 66a7487a7ab343a4abca5312afe78862 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:28.967505 21361 leader_election.cc:290] T 475b435b2967418f927fa172d2ff04ea P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 election: Requested vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767)
I20260130 02:58:28.977655 21211 raft_consensus.cc:515] T 66a7487a7ab343a4abca5312afe78862 P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:28.984329 21211 leader_election.cc:290] T 66a7487a7ab343a4abca5312afe78862 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 election: Requested vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767)
W20260130 02:58:28.994890 19463 leader_election.cc:343] T 475b435b2967418f927fa172d2ff04ea P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Illegal state: must be running to vote when last-logged opid is not known
W20260130 02:58:29.000401 19463 leader_election.cc:343] T 155ff0c35fc44cb292a28407e6ed937d P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Illegal state: must be running to vote when last-logged opid is not known
W20260130 02:58:29.009006 19463 leader_election.cc:343] T 66a7487a7ab343a4abca5312afe78862 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:29.021250 19807 tablet_bootstrap.cc:492] T 532e867fd5944d468d7672d2a5c92f3c P 371a36d219164974abdf848841528152: No bootstrap required, opened a new log
I20260130 02:58:29.021664 19807 ts_tablet_manager.cc:1403] T 532e867fd5944d468d7672d2a5c92f3c P 371a36d219164974abdf848841528152: Time spent bootstrapping tablet: real 0.224s user 0.014s sys 0.009s
I20260130 02:58:29.023919 19807 raft_consensus.cc:359] T 532e867fd5944d468d7672d2a5c92f3c P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:29.024529 19807 raft_consensus.cc:385] T 532e867fd5944d468d7672d2a5c92f3c P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:29.024768 19807 raft_consensus.cc:740] T 532e867fd5944d468d7672d2a5c92f3c P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 371a36d219164974abdf848841528152, State: Initialized, Role: FOLLOWER
I20260130 02:58:29.025318 19807 consensus_queue.cc:260] T 532e867fd5944d468d7672d2a5c92f3c P 371a36d219164974abdf848841528152 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:29.031948 21375 leader_election.cc:290] T 155ff0c35fc44cb292a28407e6ed937d P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 election: Requested vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767)
W20260130 02:58:29.055060 19159 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.100.67:39571, user_credentials={real_user=slave}} blocked reactor thread for 46382.6us
W20260130 02:58:29.062597 19313 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.100.67:39571, user_credentials={real_user=slave}} blocked reactor thread for 76059.3us
I20260130 02:58:29.065776 19807 ts_tablet_manager.cc:1434] T 532e867fd5944d468d7672d2a5c92f3c P 371a36d219164974abdf848841528152: Time spent starting tablet: real 0.044s user 0.006s sys 0.000s
I20260130 02:58:29.080194 19224 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "475b435b2967418f927fa172d2ff04ea" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152"
I20260130 02:58:29.094413 19224 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "66a7487a7ab343a4abca5312afe78862" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152"
I20260130 02:58:29.096657 19224 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "155ff0c35fc44cb292a28407e6ed937d" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152"
I20260130 02:58:29.097404 19313 leader_election.cc:304] T 6f5a90b75a0f4851b5db672c41253f7b P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 78c2c46c7520492db35d51c244fb4f74, d870164263bb44b6b780a557daea3656; no voters: 371a36d219164974abdf848841528152
I20260130 02:58:29.105166 21371 raft_consensus.cc:2804] T 6f5a90b75a0f4851b5db672c41253f7b P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260130 02:58:29.105533 21371 raft_consensus.cc:493] T 6f5a90b75a0f4851b5db672c41253f7b P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260130 02:58:29.105854 21371 raft_consensus.cc:3060] T 6f5a90b75a0f4851b5db672c41253f7b P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:29.110318 19807 tablet_bootstrap.cc:492] T 3738385130c044e0a5a5803da74c567d P 371a36d219164974abdf848841528152: Bootstrap starting.
I20260130 02:58:29.113862 21371 raft_consensus.cc:515] T 6f5a90b75a0f4851b5db672c41253f7b P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:29.115562 19807 tablet_bootstrap.cc:654] T 3738385130c044e0a5a5803da74c567d P 371a36d219164974abdf848841528152: Neither blocks nor log segments found. Creating new log.
W20260130 02:58:29.116235 19463 leader_election.cc:343] T 475b435b2967418f927fa172d2ff04ea P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Illegal state: must be running to vote when last-logged opid is not known
W20260130 02:58:29.117095 19463 leader_election.cc:343] T 66a7487a7ab343a4abca5312afe78862 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Illegal state: must be running to vote when last-logged opid is not known
W20260130 02:58:29.118048 19463 leader_election.cc:343] T 155ff0c35fc44cb292a28407e6ed937d P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:29.113399 21361 raft_consensus.cc:493] T 94355fc1338f479e8798070aff04fcba P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:29.119308 21361 raft_consensus.cc:515] T 94355fc1338f479e8798070aff04fcba P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:29.121047 21361 leader_election.cc:290] T 94355fc1338f479e8798070aff04fcba P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767)
I20260130 02:58:29.121088 21371 leader_election.cc:290] T 6f5a90b75a0f4851b5db672c41253f7b P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 election: Requested vote from peers d870164263bb44b6b780a557daea3656 (127.18.100.67:39571), 371a36d219164974abdf848841528152 (127.18.100.65:42021)
I20260130 02:58:29.137743 21353 raft_consensus.cc:493] T 8b47653c7c3945a68e9bf4680bfc0496 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:29.138386 21353 raft_consensus.cc:515] T 8b47653c7c3945a68e9bf4680bfc0496 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:29.140435 21353 leader_election.cc:290] T 8b47653c7c3945a68e9bf4680bfc0496 P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767), d870164263bb44b6b780a557daea3656 (127.18.100.67:39571)
I20260130 02:58:29.153733 19224 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6f5a90b75a0f4851b5db672c41253f7b" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152"
I20260130 02:58:29.154755 21353 raft_consensus.cc:493] T 532e867fd5944d468d7672d2a5c92f3c P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:29.155258 21353 raft_consensus.cc:515] T 532e867fd5944d468d7672d2a5c92f3c P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
W20260130 02:58:29.157063 19314 leader_election.cc:343] T 6f5a90b75a0f4851b5db672c41253f7b P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:29.159796 19366 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8b47653c7c3945a68e9bf4680bfc0496" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74" is_pre_election: true
I20260130 02:58:29.160523 19366 raft_consensus.cc:2393] T 8b47653c7c3945a68e9bf4680bfc0496 P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 371a36d219164974abdf848841528152 in current term 1: Already voted for candidate d870164263bb44b6b780a557daea3656 in this term.
I20260130 02:58:29.162786 19224 raft_consensus.cc:3060] T 532e867fd5944d468d7672d2a5c92f3c P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:29.175545 21411 raft_consensus.cc:493] T 920f52b975ab480fbf42656171222f09 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:29.175981 21411 raft_consensus.cc:515] T 920f52b975ab480fbf42656171222f09 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:29.177764 21411 leader_election.cc:290] T 920f52b975ab480fbf42656171222f09 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767)
I20260130 02:58:29.199572 21353 leader_election.cc:290] T 532e867fd5944d468d7672d2a5c92f3c P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d870164263bb44b6b780a557daea3656 (127.18.100.67:39571), 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767)
I20260130 02:58:29.201922 19313 leader_election.cc:304] T 3738385130c044e0a5a5803da74c567d P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 78c2c46c7520492db35d51c244fb4f74, d870164263bb44b6b780a557daea3656; no voters: 371a36d219164974abdf848841528152
I20260130 02:58:29.206234 21416 raft_consensus.cc:493] T 00000000000000000000000000000000 P 107f274cdda543038832f06a56ab9130 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 658299197a424f53a37f14fadf089ae4)
W20260130 02:58:29.207888 19463 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.100.65:42021, user_credentials={real_user=slave}} blocked reactor thread for 88856.4us
I20260130 02:58:29.214269 19520 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8b47653c7c3945a68e9bf4680bfc0496" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
I20260130 02:58:29.222972 21407 raft_consensus.cc:2804] T 3738385130c044e0a5a5803da74c567d P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Leader election won for term 1
I20260130 02:58:29.223453 21407 raft_consensus.cc:697] T 3738385130c044e0a5a5803da74c567d P 78c2c46c7520492db35d51c244fb4f74 [term 1 LEADER]: Becoming Leader. State: Replica: 78c2c46c7520492db35d51c244fb4f74, State: Running, Role: LEADER
I20260130 02:58:29.224220 21407 consensus_queue.cc:237] T 3738385130c044e0a5a5803da74c567d P 78c2c46c7520492db35d51c244fb4f74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:29.231006 21416 raft_consensus.cc:515] T 00000000000000000000000000000000 P 107f274cdda543038832f06a56ab9130 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "658299197a424f53a37f14fadf089ae4" member_type: VOTER last_known_addr { host: "127.18.100.126" port: 38191 } } peers { permanent_uuid: "107f274cdda543038832f06a56ab9130" member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39711 } } peers { permanent_uuid: "3749baa287ed4f249329a74071781edd" member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34629 } }
I20260130 02:58:29.201011 21169 raft_consensus.cc:493] T 1129f72158ec4bae89da3c94cdb70839 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:29.231971 21169 raft_consensus.cc:515] T 1129f72158ec4bae89da3c94cdb70839 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:29.233738 21169 leader_election.cc:290] T 1129f72158ec4bae89da3c94cdb70839 P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d870164263bb44b6b780a557daea3656 (127.18.100.67:39571), 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767)
I20260130 02:58:29.234455 19222 raft_consensus.cc:3060] T 1129f72158ec4bae89da3c94cdb70839 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:29.217118 19223 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "94355fc1338f479e8798070aff04fcba" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152" is_pre_election: true
I20260130 02:58:29.239197 19223 raft_consensus.cc:2468] T 94355fc1338f479e8798070aff04fcba P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d870164263bb44b6b780a557daea3656 in term 1.
I20260130 02:58:29.218060 19218 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "920f52b975ab480fbf42656171222f09" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152" is_pre_election: true
I20260130 02:58:29.224639 19807 tablet_bootstrap.cc:492] T 3738385130c044e0a5a5803da74c567d P 371a36d219164974abdf848841528152: No bootstrap required, opened a new log
I20260130 02:58:29.240212 19807 ts_tablet_manager.cc:1403] T 3738385130c044e0a5a5803da74c567d P 371a36d219164974abdf848841528152: Time spent bootstrapping tablet: real 0.130s user 0.012s sys 0.000s
W20260130 02:58:29.228585 19159 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.100.66:40767, user_credentials={real_user=slave}} blocked reactor thread for 53085.2us
I20260130 02:58:29.247913 19463 leader_election.cc:304] T 94355fc1338f479e8798070aff04fcba P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 371a36d219164974abdf848841528152, d870164263bb44b6b780a557daea3656; no voters:
W20260130 02:58:29.248777 19463 leader_election.cc:343] T 920f52b975ab480fbf42656171222f09 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:29.249989 21364 raft_consensus.cc:2804] T 94355fc1338f479e8798070aff04fcba P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260130 02:58:29.250324 21364 raft_consensus.cc:493] T 94355fc1338f479e8798070aff04fcba P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260130 02:58:29.250924 19520 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "532e867fd5944d468d7672d2a5c92f3c" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
I20260130 02:58:29.252796 19519 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1129f72158ec4bae89da3c94cdb70839" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
I20260130 02:58:29.253731 21364 raft_consensus.cc:3060] T 94355fc1338f479e8798070aff04fcba P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Advancing to term 2
I20260130 02:58:29.258919 19379 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1129f72158ec4bae89da3c94cdb70839" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74" is_pre_election: true
I20260130 02:58:29.260039 19365 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "532e867fd5944d468d7672d2a5c92f3c" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74" is_pre_election: true
I20260130 02:58:29.270566 19519 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "920f52b975ab480fbf42656171222f09" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
I20260130 02:58:29.272332 19519 raft_consensus.cc:2468] T 920f52b975ab480fbf42656171222f09 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78c2c46c7520492db35d51c244fb4f74 in term 0.
I20260130 02:58:29.280478 20807 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "107f274cdda543038832f06a56ab9130" candidate_term: 7 candidate_status { last_received { term: 6 index: 374 } } ignore_live_leader: false dest_uuid: "3749baa287ed4f249329a74071781edd" is_pre_election: true
I20260130 02:58:29.267196 19807 raft_consensus.cc:359] T 3738385130c044e0a5a5803da74c567d P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:29.296522 21364 raft_consensus.cc:515] T 94355fc1338f479e8798070aff04fcba P d870164263bb44b6b780a557daea3656 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:29.288944 19807 raft_consensus.cc:385] T 3738385130c044e0a5a5803da74c567d P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:29.278545 21416 leader_election.cc:290] T 00000000000000000000000000000000 P 107f274cdda543038832f06a56ab9130 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 658299197a424f53a37f14fadf089ae4 (127.18.100.126:38191), 3749baa287ed4f249329a74071781edd (127.18.100.124:34629)
I20260130 02:58:29.253252 19159 leader_election.cc:304] T 8b47653c7c3945a68e9bf4680bfc0496 P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 371a36d219164974abdf848841528152; no voters: 78c2c46c7520492db35d51c244fb4f74, d870164263bb44b6b780a557daea3656
W20260130 02:58:29.303532 19159 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.100.67:39571, user_credentials={real_user=slave}} blocked reactor thread for 50358.9us
I20260130 02:58:29.304358 21169 raft_consensus.cc:3060] T 8b47653c7c3945a68e9bf4680bfc0496 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:29.297533 19519 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6f5a90b75a0f4851b5db672c41253f7b" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656"
I20260130 02:58:29.311525 19519 raft_consensus.cc:3060] T 6f5a90b75a0f4851b5db672c41253f7b P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:29.312285 21169 raft_consensus.cc:2749] T 8b47653c7c3945a68e9bf4680bfc0496 P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260130 02:58:29.299834 19807 raft_consensus.cc:740] T 3738385130c044e0a5a5803da74c567d P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 371a36d219164974abdf848841528152, State: Initialized, Role: FOLLOWER
I20260130 02:58:29.326699 19221 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "94355fc1338f479e8798070aff04fcba" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152"
I20260130 02:58:29.327319 19221 raft_consensus.cc:3060] T 94355fc1338f479e8798070aff04fcba P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Advancing to term 2
I20260130 02:58:29.332813 19519 raft_consensus.cc:2468] T 6f5a90b75a0f4851b5db672c41253f7b P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78c2c46c7520492db35d51c244fb4f74 in term 1.
I20260130 02:58:29.334267 19221 raft_consensus.cc:2468] T 94355fc1338f479e8798070aff04fcba P 371a36d219164974abdf848841528152 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d870164263bb44b6b780a557daea3656 in term 2.
I20260130 02:58:29.335785 21364 leader_election.cc:290] T 94355fc1338f479e8798070aff04fcba P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 2 election: Requested vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767)
W20260130 02:58:29.353420 19159 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.100.67:39571, user_credentials={real_user=slave}} blocked reactor thread for 49288.8us
I20260130 02:58:29.354971 19463 leader_election.cc:304] T 94355fc1338f479e8798070aff04fcba P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 371a36d219164974abdf848841528152, d870164263bb44b6b780a557daea3656; no voters:
I20260130 02:58:29.355834 21364 raft_consensus.cc:2804] T 94355fc1338f479e8798070aff04fcba P d870164263bb44b6b780a557daea3656 [term 2 FOLLOWER]: Leader election won for term 2
I20260130 02:58:29.359885 21364 raft_consensus.cc:697] T 94355fc1338f479e8798070aff04fcba P d870164263bb44b6b780a557daea3656 [term 2 LEADER]: Becoming Leader. State: Replica: d870164263bb44b6b780a557daea3656, State: Running, Role: LEADER
I20260130 02:58:29.360630 21364 consensus_queue.cc:237] T 94355fc1338f479e8798070aff04fcba P d870164263bb44b6b780a557daea3656 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:29.363086 19366 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "475b435b2967418f927fa172d2ff04ea" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74"
I20260130 02:58:29.363620 19366 raft_consensus.cc:3060] T 475b435b2967418f927fa172d2ff04ea P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:29.367126 21371 raft_consensus.cc:493] T 94355fc1338f479e8798070aff04fcba P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:29.367553 21371 raft_consensus.cc:515] T 94355fc1338f479e8798070aff04fcba P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:29.369321 21371 leader_election.cc:290] T 94355fc1338f479e8798070aff04fcba P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), d870164263bb44b6b780a557daea3656 (127.18.100.67:39571)
I20260130 02:58:29.375787 19366 raft_consensus.cc:2468] T 475b435b2967418f927fa172d2ff04ea P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d870164263bb44b6b780a557daea3656 in term 1.
I20260130 02:58:29.382267 19222 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "94355fc1338f479e8798070aff04fcba" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152" is_pre_election: true
I20260130 02:58:29.383272 19222 raft_consensus.cc:2393] T 94355fc1338f479e8798070aff04fcba P 371a36d219164974abdf848841528152 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 78c2c46c7520492db35d51c244fb4f74 in current term 2: Already voted for candidate d870164263bb44b6b780a557daea3656 in this term.
I20260130 02:58:29.345867 19807 consensus_queue.cc:260] T 3738385130c044e0a5a5803da74c567d P 371a36d219164974abdf848841528152 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:29.396026 19365 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "66a7487a7ab343a4abca5312afe78862" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74"
I20260130 02:58:29.396615 19365 raft_consensus.cc:3060] T 66a7487a7ab343a4abca5312afe78862 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:29.404558 19365 raft_consensus.cc:2468] T 66a7487a7ab343a4abca5312afe78862 P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d870164263bb44b6b780a557daea3656 in term 1.
I20260130 02:58:29.410804 19159 leader_election.cc:304] T 1129f72158ec4bae89da3c94cdb70839 P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 371a36d219164974abdf848841528152; no voters: 78c2c46c7520492db35d51c244fb4f74, d870164263bb44b6b780a557daea3656
I20260130 02:58:29.421416 19371 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "155ff0c35fc44cb292a28407e6ed937d" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74"
I20260130 02:58:29.421994 19371 raft_consensus.cc:3060] T 155ff0c35fc44cb292a28407e6ed937d P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:29.429051 19371 raft_consensus.cc:2468] T 155ff0c35fc44cb292a28407e6ed937d P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d870164263bb44b6b780a557daea3656 in term 1.
I20260130 02:58:29.430986 19379 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "94355fc1338f479e8798070aff04fcba" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74" is_pre_election: true
I20260130 02:58:29.431496 19379 raft_consensus.cc:2468] T 94355fc1338f479e8798070aff04fcba P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d870164263bb44b6b780a557daea3656 in term 1.
I20260130 02:58:29.436558 21353 raft_consensus.cc:2749] T 1129f72158ec4bae89da3c94cdb70839 P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260130 02:58:29.442118 19159 leader_election.cc:304] T 532e867fd5944d468d7672d2a5c92f3c P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 371a36d219164974abdf848841528152; no voters: 78c2c46c7520492db35d51c244fb4f74, d870164263bb44b6b780a557daea3656
I20260130 02:58:29.443080 19379 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "920f52b975ab480fbf42656171222f09" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74" is_pre_election: true
I20260130 02:58:29.443652 19379 raft_consensus.cc:2468] T 920f52b975ab480fbf42656171222f09 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d870164263bb44b6b780a557daea3656 in term 0.
I20260130 02:58:29.463042 21169 raft_consensus.cc:2749] T 532e867fd5944d468d7672d2a5c92f3c P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260130 02:58:29.468129 21364 consensus_queue.cc:1048] T 8b47653c7c3945a68e9bf4680bfc0496 P d870164263bb44b6b780a557daea3656 [LEADER]: Connected to new peer: Peer: permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260130 02:58:29.478415 19371 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "94355fc1338f479e8798070aff04fcba" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74"
I20260130 02:58:29.478997 19371 raft_consensus.cc:3060] T 94355fc1338f479e8798070aff04fcba P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Advancing to term 2
I20260130 02:58:29.491381 19371 raft_consensus.cc:2468] T 94355fc1338f479e8798070aff04fcba P 78c2c46c7520492db35d51c244fb4f74 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d870164263bb44b6b780a557daea3656 in term 2.
I20260130 02:58:29.488399 19807 ts_tablet_manager.cc:1434] T 3738385130c044e0a5a5803da74c567d P 371a36d219164974abdf848841528152: Time spent starting tablet: real 0.247s user 0.023s sys 0.003s
I20260130 02:58:29.520576 19222 raft_consensus.cc:3060] T dd377fab8c544bf69121dc9e9a060b29 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:29.538174 19807 tablet_bootstrap.cc:492] T e691c238199b4914bcc3fd248f31420c P 371a36d219164974abdf848841528152: Bootstrap starting.
I20260130 02:58:29.563853 19519 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "94355fc1338f479e8798070aff04fcba" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
I20260130 02:58:29.593093 19462 leader_election.cc:304] T 475b435b2967418f927fa172d2ff04ea P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 78c2c46c7520492db35d51c244fb4f74, d870164263bb44b6b780a557daea3656; no voters: 371a36d219164974abdf848841528152
I20260130 02:58:29.594224 21211 raft_consensus.cc:2804] T 475b435b2967418f927fa172d2ff04ea P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Leader election won for term 1
I20260130 02:58:29.594666 21211 raft_consensus.cc:697] T 475b435b2967418f927fa172d2ff04ea P d870164263bb44b6b780a557daea3656 [term 1 LEADER]: Becoming Leader. State: Replica: d870164263bb44b6b780a557daea3656, State: Running, Role: LEADER
I20260130 02:58:29.595351 21211 consensus_queue.cc:237] T 475b435b2967418f927fa172d2ff04ea P d870164263bb44b6b780a557daea3656 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:29.623363 19462 leader_election.cc:304] T 66a7487a7ab343a4abca5312afe78862 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 78c2c46c7520492db35d51c244fb4f74, d870164263bb44b6b780a557daea3656; no voters: 371a36d219164974abdf848841528152
I20260130 02:58:29.624058 21375 raft_consensus.cc:2804] T 66a7487a7ab343a4abca5312afe78862 P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Leader election won for term 1
I20260130 02:58:29.624486 21375 raft_consensus.cc:697] T 66a7487a7ab343a4abca5312afe78862 P d870164263bb44b6b780a557daea3656 [term 1 LEADER]: Becoming Leader. State: Replica: d870164263bb44b6b780a557daea3656, State: Running, Role: LEADER
I20260130 02:58:29.625227 21375 consensus_queue.cc:237] T 66a7487a7ab343a4abca5312afe78862 P d870164263bb44b6b780a557daea3656 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:29.636592 19462 leader_election.cc:304] T 155ff0c35fc44cb292a28407e6ed937d P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 78c2c46c7520492db35d51c244fb4f74, d870164263bb44b6b780a557daea3656; no voters: 371a36d219164974abdf848841528152
I20260130 02:58:29.637207 21375 raft_consensus.cc:2804] T 155ff0c35fc44cb292a28407e6ed937d P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Leader election won for term 1
I20260130 02:58:29.638031 21375 raft_consensus.cc:697] T 155ff0c35fc44cb292a28407e6ed937d P d870164263bb44b6b780a557daea3656 [term 1 LEADER]: Becoming Leader. State: Replica: d870164263bb44b6b780a557daea3656, State: Running, Role: LEADER
I20260130 02:58:29.638983 21375 consensus_queue.cc:237] T 155ff0c35fc44cb292a28407e6ed937d P d870164263bb44b6b780a557daea3656 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:29.653859 19462 leader_election.cc:304] T 920f52b975ab480fbf42656171222f09 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 78c2c46c7520492db35d51c244fb4f74, d870164263bb44b6b780a557daea3656; no voters: 371a36d219164974abdf848841528152
I20260130 02:58:29.654808 21211 raft_consensus.cc:2804] T 920f52b975ab480fbf42656171222f09 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260130 02:58:29.655107 21211 raft_consensus.cc:493] T 920f52b975ab480fbf42656171222f09 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260130 02:58:29.655360 21211 raft_consensus.cc:3060] T 920f52b975ab480fbf42656171222f09 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:29.673790 19807 tablet_bootstrap.cc:654] T e691c238199b4914bcc3fd248f31420c P 371a36d219164974abdf848841528152: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:29.693003 21411 consensus_queue.cc:1048] T 30cc52136bb3422895d95c0fed0c6798 P d870164263bb44b6b780a557daea3656 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260130 02:58:29.703085 21364 consensus_queue.cc:1048] T 8b47653c7c3945a68e9bf4680bfc0496 P d870164263bb44b6b780a557daea3656 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260130 02:58:29.674113 21427 raft_consensus.cc:493] T 00000000000000000000000000000000 P 3749baa287ed4f249329a74071781edd [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 658299197a424f53a37f14fadf089ae4)
I20260130 02:58:29.709157 21427 raft_consensus.cc:515] T 00000000000000000000000000000000 P 3749baa287ed4f249329a74071781edd [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "658299197a424f53a37f14fadf089ae4" member_type: VOTER last_known_addr { host: "127.18.100.126" port: 38191 } } peers { permanent_uuid: "107f274cdda543038832f06a56ab9130" member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39711 } } peers { permanent_uuid: "3749baa287ed4f249329a74071781edd" member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34629 } }
I20260130 02:58:29.719527 18960 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3749baa287ed4f249329a74071781edd" candidate_term: 7 candidate_status { last_received { term: 6 index: 374 } } ignore_live_leader: false dest_uuid: "107f274cdda543038832f06a56ab9130" is_pre_election: true
I20260130 02:58:29.723770 21292 consensus_queue.cc:1048] T bc04be1284414a8699d3c4b88bf62813 P 78c2c46c7520492db35d51c244fb4f74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260130 02:58:29.729008 21427 leader_election.cc:290] T 00000000000000000000000000000000 P 3749baa287ed4f249329a74071781edd [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 658299197a424f53a37f14fadf089ae4 (127.18.100.126:38191), 107f274cdda543038832f06a56ab9130 (127.18.100.125:39711)
I20260130 02:58:29.746601 19313 leader_election.cc:304] T 920f52b975ab480fbf42656171222f09 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 78c2c46c7520492db35d51c244fb4f74, d870164263bb44b6b780a557daea3656; no voters: 371a36d219164974abdf848841528152
I20260130 02:58:29.755191 21169 raft_consensus.cc:493] T 3738385130c044e0a5a5803da74c567d P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:29.755662 21169 raft_consensus.cc:515] T 3738385130c044e0a5a5803da74c567d P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:29.757401 21169 leader_election.cc:290] T 3738385130c044e0a5a5803da74c567d P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767), d870164263bb44b6b780a557daea3656 (127.18.100.67:39571)
I20260130 02:58:29.760078 21407 raft_consensus.cc:2804] T 920f52b975ab480fbf42656171222f09 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260130 02:58:29.760406 21407 raft_consensus.cc:493] T 920f52b975ab480fbf42656171222f09 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260130 02:58:29.760702 21407 raft_consensus.cc:3060] T 920f52b975ab480fbf42656171222f09 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:29.772790 19366 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3738385130c044e0a5a5803da74c567d" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74" is_pre_election: true
I20260130 02:58:29.782995 21211 raft_consensus.cc:515] T 920f52b975ab480fbf42656171222f09 P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:29.793162 21211 leader_election.cc:290] T 920f52b975ab480fbf42656171222f09 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 election: Requested vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767)
I20260130 02:58:29.799557 21407 raft_consensus.cc:515] T 920f52b975ab480fbf42656171222f09 P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:29.802564 19224 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "920f52b975ab480fbf42656171222f09" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152"
W20260130 02:58:29.804790 19314 leader_election.cc:343] T 920f52b975ab480fbf42656171222f09 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Illegal state: must be running to vote when last-logged opid is not known
W20260130 02:58:29.865201 19313 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.100.67:39571, user_credentials={real_user=slave}} blocked reactor thread for 83266us
I20260130 02:58:29.866106 19313 leader_election.cc:304] T 6f5a90b75a0f4851b5db672c41253f7b P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 78c2c46c7520492db35d51c244fb4f74, d870164263bb44b6b780a557daea3656; no voters: 371a36d219164974abdf848841528152
I20260130 02:58:29.867832 21372 raft_consensus.cc:2804] T 6f5a90b75a0f4851b5db672c41253f7b P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Leader election won for term 1
I20260130 02:58:29.868252 21372 raft_consensus.cc:697] T 6f5a90b75a0f4851b5db672c41253f7b P 78c2c46c7520492db35d51c244fb4f74 [term 1 LEADER]: Becoming Leader. State: Replica: 78c2c46c7520492db35d51c244fb4f74, State: Running, Role: LEADER
I20260130 02:58:29.868983 21372 consensus_queue.cc:237] T 6f5a90b75a0f4851b5db672c41253f7b P 78c2c46c7520492db35d51c244fb4f74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:29.877336 21407 leader_election.cc:290] T 920f52b975ab480fbf42656171222f09 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 election: Requested vote from peers d870164263bb44b6b780a557daea3656 (127.18.100.67:39571), 371a36d219164974abdf848841528152 (127.18.100.65:42021)
I20260130 02:58:29.879721 19519 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3738385130c044e0a5a5803da74c567d" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
I20260130 02:58:29.880471 19519 raft_consensus.cc:2393] T 3738385130c044e0a5a5803da74c567d P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 371a36d219164974abdf848841528152 in current term 1: Already voted for candidate 78c2c46c7520492db35d51c244fb4f74 in this term.
I20260130 02:58:29.880077 19313 leader_election.cc:304] T 94355fc1338f479e8798070aff04fcba P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78c2c46c7520492db35d51c244fb4f74; no voters: 371a36d219164974abdf848841528152, d870164263bb44b6b780a557daea3656
I20260130 02:58:29.882272 21371 raft_consensus.cc:2749] T 94355fc1338f479e8798070aff04fcba P 78c2c46c7520492db35d51c244fb4f74 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260130 02:58:29.885210 21371 consensus_queue.cc:1048] T bc04be1284414a8699d3c4b88bf62813 P 78c2c46c7520492db35d51c244fb4f74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260130 02:58:29.908536 19519 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "920f52b975ab480fbf42656171222f09" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656"
I20260130 02:58:29.909375 19519 raft_consensus.cc:2393] T 920f52b975ab480fbf42656171222f09 P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 78c2c46c7520492db35d51c244fb4f74 in current term 1: Already voted for candidate d870164263bb44b6b780a557daea3656 in this term.
I20260130 02:58:29.943217 19365 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "920f52b975ab480fbf42656171222f09" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74"
I20260130 02:58:29.944041 19365 raft_consensus.cc:2393] T 920f52b975ab480fbf42656171222f09 P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate d870164263bb44b6b780a557daea3656 in current term 1: Already voted for candidate 78c2c46c7520492db35d51c244fb4f74 in this term.
I20260130 02:58:29.951689 19224 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "920f52b975ab480fbf42656171222f09" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152"
I20260130 02:58:29.956070 19807 tablet_bootstrap.cc:492] T e691c238199b4914bcc3fd248f31420c P 371a36d219164974abdf848841528152: No bootstrap required, opened a new log
I20260130 02:58:29.956501 19807 ts_tablet_manager.cc:1403] T e691c238199b4914bcc3fd248f31420c P 371a36d219164974abdf848841528152: Time spent bootstrapping tablet: real 0.431s user 0.022s sys 0.009s
I20260130 02:58:29.958920 19807 raft_consensus.cc:359] T e691c238199b4914bcc3fd248f31420c P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:29.959506 19807 raft_consensus.cc:385] T e691c238199b4914bcc3fd248f31420c P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:29.959745 19807 raft_consensus.cc:740] T e691c238199b4914bcc3fd248f31420c P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 371a36d219164974abdf848841528152, State: Initialized, Role: FOLLOWER
I20260130 02:58:29.960366 19807 consensus_queue.cc:260] T e691c238199b4914bcc3fd248f31420c P 371a36d219164974abdf848841528152 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:29.962282 19807 ts_tablet_manager.cc:1434] T e691c238199b4914bcc3fd248f31420c P 371a36d219164974abdf848841528152: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260130 02:58:29.983017 19313 leader_election.cc:304] T 920f52b975ab480fbf42656171222f09 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78c2c46c7520492db35d51c244fb4f74; no voters: 371a36d219164974abdf848841528152, d870164263bb44b6b780a557daea3656
I20260130 02:58:29.986336 19159 leader_election.cc:304] T 3738385130c044e0a5a5803da74c567d P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 371a36d219164974abdf848841528152; no voters: 78c2c46c7520492db35d51c244fb4f74, d870164263bb44b6b780a557daea3656
I20260130 02:58:29.987038 21169 raft_consensus.cc:3060] T 3738385130c044e0a5a5803da74c567d P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:29.988773 21407 raft_consensus.cc:2749] T 920f52b975ab480fbf42656171222f09 P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
W20260130 02:58:30.019052 19463 consensus_peers.cc:597] T 30cc52136bb3422895d95c0fed0c6798 P d870164263bb44b6b780a557daea3656 -> Peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Couldn't send request to peer 371a36d219164974abdf848841528152. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20260130 02:58:30.020213 19463 leader_election.cc:343] T 920f52b975ab480fbf42656171222f09 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:58:30.020571 19463 leader_election.cc:304] T 920f52b975ab480fbf42656171222f09 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d870164263bb44b6b780a557daea3656; no voters: 371a36d219164974abdf848841528152, 78c2c46c7520492db35d51c244fb4f74
I20260130 02:58:30.021512 21375 raft_consensus.cc:2749] T 920f52b975ab480fbf42656171222f09 P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260130 02:58:30.028107 21169 raft_consensus.cc:2749] T 3738385130c044e0a5a5803da74c567d P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260130 02:58:30.065042 21372 consensus_queue.cc:1048] T 3738385130c044e0a5a5803da74c567d P 78c2c46c7520492db35d51c244fb4f74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260130 02:58:30.083498 19807 tablet_bootstrap.cc:492] T 30cc52136bb3422895d95c0fed0c6798 P 371a36d219164974abdf848841528152: Bootstrap starting.
I20260130 02:58:30.088814 19807 tablet_bootstrap.cc:654] T 30cc52136bb3422895d95c0fed0c6798 P 371a36d219164974abdf848841528152: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:30.132836 19222 raft_consensus.cc:3060] T e691c238199b4914bcc3fd248f31420c P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:30.166878 21438 consensus_queue.cc:1048] T 3738385130c044e0a5a5803da74c567d P 78c2c46c7520492db35d51c244fb4f74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260130 02:58:30.204710 19807 tablet_bootstrap.cc:492] T 30cc52136bb3422895d95c0fed0c6798 P 371a36d219164974abdf848841528152: No bootstrap required, opened a new log
I20260130 02:58:30.205119 19807 ts_tablet_manager.cc:1403] T 30cc52136bb3422895d95c0fed0c6798 P 371a36d219164974abdf848841528152: Time spent bootstrapping tablet: real 0.122s user 0.008s sys 0.000s
I20260130 02:58:30.207600 19807 raft_consensus.cc:359] T 30cc52136bb3422895d95c0fed0c6798 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:30.208216 19807 raft_consensus.cc:385] T 30cc52136bb3422895d95c0fed0c6798 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:30.208453 19807 raft_consensus.cc:740] T 30cc52136bb3422895d95c0fed0c6798 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 371a36d219164974abdf848841528152, State: Initialized, Role: FOLLOWER
I20260130 02:58:30.209045 19807 consensus_queue.cc:260] T 30cc52136bb3422895d95c0fed0c6798 P 371a36d219164974abdf848841528152 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:30.227602 19807 ts_tablet_manager.cc:1434] T 30cc52136bb3422895d95c0fed0c6798 P 371a36d219164974abdf848841528152: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20260130 02:58:30.274224 21409 raft_consensus.cc:493] T 30cc52136bb3422895d95c0fed0c6798 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:30.274734 21409 raft_consensus.cc:515] T 30cc52136bb3422895d95c0fed0c6798 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:30.276532 21409 leader_election.cc:290] T 30cc52136bb3422895d95c0fed0c6798 P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d870164263bb44b6b780a557daea3656 (127.18.100.67:39571), 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767)
I20260130 02:58:30.278270 19378 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "30cc52136bb3422895d95c0fed0c6798" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74" is_pre_election: true
I20260130 02:58:30.280035 19520 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "30cc52136bb3422895d95c0fed0c6798" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
I20260130 02:58:30.281819 19807 tablet_bootstrap.cc:492] T 920f52b975ab480fbf42656171222f09 P 371a36d219164974abdf848841528152: Bootstrap starting.
I20260130 02:58:30.282624 19159 leader_election.cc:304] T 30cc52136bb3422895d95c0fed0c6798 P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 371a36d219164974abdf848841528152; no voters: 78c2c46c7520492db35d51c244fb4f74, d870164263bb44b6b780a557daea3656
I20260130 02:58:30.283739 21409 raft_consensus.cc:2749] T 30cc52136bb3422895d95c0fed0c6798 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260130 02:58:30.309456 19378 raft_consensus.cc:1275] T 94355fc1338f479e8798070aff04fcba P 78c2c46c7520492db35d51c244fb4f74 [term 2 FOLLOWER]: Refusing update from remote peer d870164263bb44b6b780a557daea3656: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20260130 02:58:30.310987 21361 consensus_queue.cc:1048] T 94355fc1338f479e8798070aff04fcba P d870164263bb44b6b780a557daea3656 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260130 02:58:30.316248 20613 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.100.126:38191, user_credentials={real_user=slave}} blocked reactor thread for 47736.2us
W20260130 02:58:30.331684 21448 log.cc:927] Time spent T 3738385130c044e0a5a5803da74c567d P d870164263bb44b6b780a557daea3656: Append to log took a long time: real 0.065s user 0.003s sys 0.000s
I20260130 02:58:30.388306 19222 raft_consensus.cc:1275] T 94355fc1338f479e8798070aff04fcba P 371a36d219164974abdf848841528152 [term 2 FOLLOWER]: Refusing update from remote peer d870164263bb44b6b780a557daea3656: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20260130 02:58:30.391314 21436 consensus_queue.cc:1048] T 94355fc1338f479e8798070aff04fcba P d870164263bb44b6b780a557daea3656 [LEADER]: Connected to new peer: Peer: permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260130 02:58:30.392299 19807 tablet_bootstrap.cc:654] T 920f52b975ab480fbf42656171222f09 P 371a36d219164974abdf848841528152: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:30.438340 19807 tablet_bootstrap.cc:492] T 920f52b975ab480fbf42656171222f09 P 371a36d219164974abdf848841528152: No bootstrap required, opened a new log
I20260130 02:58:30.438791 19807 ts_tablet_manager.cc:1403] T 920f52b975ab480fbf42656171222f09 P 371a36d219164974abdf848841528152: Time spent bootstrapping tablet: real 0.157s user 0.013s sys 0.008s
I20260130 02:58:30.440979 19807 raft_consensus.cc:359] T 920f52b975ab480fbf42656171222f09 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:30.441596 19807 raft_consensus.cc:385] T 920f52b975ab480fbf42656171222f09 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:30.441852 19807 raft_consensus.cc:740] T 920f52b975ab480fbf42656171222f09 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 371a36d219164974abdf848841528152, State: Initialized, Role: FOLLOWER
I20260130 02:58:30.442386 19807 consensus_queue.cc:260] T 920f52b975ab480fbf42656171222f09 P 371a36d219164974abdf848841528152 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:30.461321 19807 ts_tablet_manager.cc:1434] T 920f52b975ab480fbf42656171222f09 P 371a36d219164974abdf848841528152: Time spent starting tablet: real 0.022s user 0.006s sys 0.000s
I20260130 02:58:30.466236 21361 consensus_queue.cc:1048] T 66a7487a7ab343a4abca5312afe78862 P d870164263bb44b6b780a557daea3656 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260130 02:58:30.472731 19807 tablet_bootstrap.cc:492] T 155ff0c35fc44cb292a28407e6ed937d P 371a36d219164974abdf848841528152: Bootstrap starting.
I20260130 02:58:30.477984 19807 tablet_bootstrap.cc:654] T 155ff0c35fc44cb292a28407e6ed937d P 371a36d219164974abdf848841528152: Neither blocks nor log segments found. Creating new log.
W20260130 02:58:30.533000 19463 consensus_peers.cc:597] T 66a7487a7ab343a4abca5312afe78862 P d870164263bb44b6b780a557daea3656 -> Peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Couldn't send request to peer 371a36d219164974abdf848841528152. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260130 02:58:30.640985 21375 consensus_queue.cc:1048] T 155ff0c35fc44cb292a28407e6ed937d P d870164263bb44b6b780a557daea3656 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260130 02:58:30.668448 19807 tablet_bootstrap.cc:492] T 155ff0c35fc44cb292a28407e6ed937d P 371a36d219164974abdf848841528152: No bootstrap required, opened a new log
I20260130 02:58:30.668870 19807 ts_tablet_manager.cc:1403] T 155ff0c35fc44cb292a28407e6ed937d P 371a36d219164974abdf848841528152: Time spent bootstrapping tablet: real 0.196s user 0.007s sys 0.004s
I20260130 02:58:30.672511 19807 raft_consensus.cc:359] T 155ff0c35fc44cb292a28407e6ed937d P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:30.675241 19807 raft_consensus.cc:385] T 155ff0c35fc44cb292a28407e6ed937d P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:30.676179 19807 raft_consensus.cc:740] T 155ff0c35fc44cb292a28407e6ed937d P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 371a36d219164974abdf848841528152, State: Initialized, Role: FOLLOWER
W20260130 02:58:30.708485 19463 consensus_peers.cc:597] T 475b435b2967418f927fa172d2ff04ea P d870164263bb44b6b780a557daea3656 -> Peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Couldn't send request to peer 371a36d219164974abdf848841528152. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20260130 02:58:30.711557 19463 consensus_peers.cc:597] T 155ff0c35fc44cb292a28407e6ed937d P d870164263bb44b6b780a557daea3656 -> Peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Couldn't send request to peer 371a36d219164974abdf848841528152. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
W20260130 02:58:30.735388 18934 leader_election.cc:336] T 00000000000000000000000000000000 P 107f274cdda543038832f06a56ab9130 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 658299197a424f53a37f14fadf089ae4 (127.18.100.126:38191): Timed out: RequestConsensusVote RPC to 127.18.100.126:38191 timed out after 0.707s (SENT)
I20260130 02:58:30.735805 18934 leader_election.cc:304] T 00000000000000000000000000000000 P 107f274cdda543038832f06a56ab9130 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 107f274cdda543038832f06a56ab9130; no voters: 3749baa287ed4f249329a74071781edd, 658299197a424f53a37f14fadf089ae4
I20260130 02:58:30.678658 19807 consensus_queue.cc:260] T 155ff0c35fc44cb292a28407e6ed937d P 371a36d219164974abdf848841528152 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:30.755694 19807 ts_tablet_manager.cc:1434] T 155ff0c35fc44cb292a28407e6ed937d P 371a36d219164974abdf848841528152: Time spent starting tablet: real 0.087s user 0.007s sys 0.003s
I20260130 02:58:30.756652 19807 tablet_bootstrap.cc:492] T 475b435b2967418f927fa172d2ff04ea P 371a36d219164974abdf848841528152: Bootstrap starting.
W20260130 02:58:30.818938 18859 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet 657e9b0d8c05413f904129dce745710b (table default.Table_526677c8ef4144bd9de7a54a5e78e2be [id=b16908b75587459597a978c71756451c], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 657e9b0d8c05413f904129dce745710b consensus state has no leader
I20260130 02:58:30.837522 21436 consensus_queue.cc:1048] T 475b435b2967418f927fa172d2ff04ea P d870164263bb44b6b780a557daea3656 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260130 02:58:30.853485 21462 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 107f274cdda543038832f06a56ab9130 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260130 02:58:30.854539 18934 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.100.126:38191, user_credentials={real_user=slave}} blocked reactor thread for 119187us
I20260130 02:58:30.912082 19807 tablet_bootstrap.cc:654] T 475b435b2967418f927fa172d2ff04ea P 371a36d219164974abdf848841528152: Neither blocks nor log segments found. Creating new log.
W20260130 02:58:30.924422 20613 leader_election.cc:336] T 00000000000000000000000000000000 P 3749baa287ed4f249329a74071781edd [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 658299197a424f53a37f14fadf089ae4 (127.18.100.126:38191): Timed out: RequestConsensusVote RPC to 127.18.100.126:38191 timed out after 0.557s (SENT)
I20260130 02:58:30.925028 20613 leader_election.cc:304] T 00000000000000000000000000000000 P 3749baa287ed4f249329a74071781edd [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3749baa287ed4f249329a74071781edd; no voters: 107f274cdda543038832f06a56ab9130, 658299197a424f53a37f14fadf089ae4
I20260130 02:58:30.931453 19224 raft_consensus.cc:3060] T 30cc52136bb3422895d95c0fed0c6798 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:30.954830 21465 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 3749baa287ed4f249329a74071781edd [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260130 02:58:30.963920 19313 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.100.67:39571, user_credentials={real_user=slave}} blocked reactor thread for 127263us
I20260130 02:58:31.015991 19807 tablet_bootstrap.cc:492] T 475b435b2967418f927fa172d2ff04ea P 371a36d219164974abdf848841528152: No bootstrap required, opened a new log
I20260130 02:58:31.016420 19807 ts_tablet_manager.cc:1403] T 475b435b2967418f927fa172d2ff04ea P 371a36d219164974abdf848841528152: Time spent bootstrapping tablet: real 0.260s user 0.001s sys 0.011s
I20260130 02:58:31.041096 19807 raft_consensus.cc:359] T 475b435b2967418f927fa172d2ff04ea P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:31.041806 19807 raft_consensus.cc:385] T 475b435b2967418f927fa172d2ff04ea P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:31.042065 19807 raft_consensus.cc:740] T 475b435b2967418f927fa172d2ff04ea P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 371a36d219164974abdf848841528152, State: Initialized, Role: FOLLOWER
I20260130 02:58:31.059074 19807 consensus_queue.cc:260] T 475b435b2967418f927fa172d2ff04ea P 371a36d219164974abdf848841528152 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:31.061450 19807 ts_tablet_manager.cc:1434] T 475b435b2967418f927fa172d2ff04ea P 371a36d219164974abdf848841528152: Time spent starting tablet: real 0.045s user 0.005s sys 0.000s
I20260130 02:58:31.093951 19807 tablet_bootstrap.cc:492] T 66a7487a7ab343a4abca5312afe78862 P 371a36d219164974abdf848841528152: Bootstrap starting.
I20260130 02:58:31.104831 19807 tablet_bootstrap.cc:654] T 66a7487a7ab343a4abca5312afe78862 P 371a36d219164974abdf848841528152: Neither blocks nor log segments found. Creating new log.
W20260130 02:58:31.155527 19314 consensus_peers.cc:597] T 6f5a90b75a0f4851b5db672c41253f7b P 78c2c46c7520492db35d51c244fb4f74 -> Peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Couldn't send request to peer 371a36d219164974abdf848841528152. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260130 02:58:31.168960 21463 raft_consensus.cc:493] T 920f52b975ab480fbf42656171222f09 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:31.169382 21463 raft_consensus.cc:515] T 920f52b975ab480fbf42656171222f09 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:31.171347 21463 leader_election.cc:290] T 920f52b975ab480fbf42656171222f09 P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d870164263bb44b6b780a557daea3656 (127.18.100.67:39571), 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767)
I20260130 02:58:31.172366 19523 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "920f52b975ab480fbf42656171222f09" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
I20260130 02:58:31.173063 19523 raft_consensus.cc:2393] T 920f52b975ab480fbf42656171222f09 P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 371a36d219164974abdf848841528152 in current term 1: Already voted for candidate d870164263bb44b6b780a557daea3656 in this term.
I20260130 02:58:31.193651 19366 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "920f52b975ab480fbf42656171222f09" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74" is_pre_election: true
I20260130 02:58:31.195000 19366 raft_consensus.cc:2393] T 920f52b975ab480fbf42656171222f09 P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 371a36d219164974abdf848841528152 in current term 1: Already voted for candidate 78c2c46c7520492db35d51c244fb4f74 in this term.
I20260130 02:58:31.233496 19159 leader_election.cc:304] T 920f52b975ab480fbf42656171222f09 P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 371a36d219164974abdf848841528152; no voters: 78c2c46c7520492db35d51c244fb4f74, d870164263bb44b6b780a557daea3656
I20260130 02:58:31.234381 21353 raft_consensus.cc:3060] T 920f52b975ab480fbf42656171222f09 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:31.240998 21353 raft_consensus.cc:2749] T 920f52b975ab480fbf42656171222f09 P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260130 02:58:31.266626 19807 tablet_bootstrap.cc:492] T 66a7487a7ab343a4abca5312afe78862 P 371a36d219164974abdf848841528152: No bootstrap required, opened a new log
I20260130 02:58:31.267117 19807 ts_tablet_manager.cc:1403] T 66a7487a7ab343a4abca5312afe78862 P 371a36d219164974abdf848841528152: Time spent bootstrapping tablet: real 0.173s user 0.007s sys 0.003s
W20260130 02:58:31.272442 19462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.100.66:40767, user_credentials={real_user=slave}} blocked reactor thread for 61666.2us
I20260130 02:58:31.285816 21473 consensus_queue.cc:1048] T 6f5a90b75a0f4851b5db672c41253f7b P 78c2c46c7520492db35d51c244fb4f74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260130 02:58:31.301275 21476 raft_consensus.cc:493] T 155ff0c35fc44cb292a28407e6ed937d P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:31.301792 21476 raft_consensus.cc:515] T 155ff0c35fc44cb292a28407e6ed937d P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:31.303665 21476 leader_election.cc:290] T 155ff0c35fc44cb292a28407e6ed937d P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d870164263bb44b6b780a557daea3656 (127.18.100.67:39571), 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767)
I20260130 02:58:31.305538 19520 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "155ff0c35fc44cb292a28407e6ed937d" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
I20260130 02:58:31.307865 19366 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "155ff0c35fc44cb292a28407e6ed937d" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74" is_pre_election: true
I20260130 02:58:31.311703 19159 leader_election.cc:304] T 155ff0c35fc44cb292a28407e6ed937d P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 371a36d219164974abdf848841528152; no voters: 78c2c46c7520492db35d51c244fb4f74, d870164263bb44b6b780a557daea3656
I20260130 02:58:31.312989 21476 raft_consensus.cc:2749] T 155ff0c35fc44cb292a28407e6ed937d P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260130 02:58:31.350730 19807 raft_consensus.cc:359] T 66a7487a7ab343a4abca5312afe78862 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:31.351393 19807 raft_consensus.cc:385] T 66a7487a7ab343a4abca5312afe78862 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:31.351649 19807 raft_consensus.cc:740] T 66a7487a7ab343a4abca5312afe78862 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 371a36d219164974abdf848841528152, State: Initialized, Role: FOLLOWER
I20260130 02:58:31.387449 19807 consensus_queue.cc:260] T 66a7487a7ab343a4abca5312afe78862 P 371a36d219164974abdf848841528152 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:31.389587 19807 ts_tablet_manager.cc:1434] T 66a7487a7ab343a4abca5312afe78862 P 371a36d219164974abdf848841528152: Time spent starting tablet: real 0.122s user 0.001s sys 0.005s
I20260130 02:58:31.425812 19807 tablet_bootstrap.cc:492] T 6f5a90b75a0f4851b5db672c41253f7b P 371a36d219164974abdf848841528152: Bootstrap starting.
I20260130 02:58:31.542599 21353 raft_consensus.cc:493] T 66a7487a7ab343a4abca5312afe78862 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:31.543027 21353 raft_consensus.cc:515] T 66a7487a7ab343a4abca5312afe78862 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:31.544564 21353 leader_election.cc:290] T 66a7487a7ab343a4abca5312afe78862 P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d870164263bb44b6b780a557daea3656 (127.18.100.67:39571), 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767)
I20260130 02:58:31.552608 19366 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "66a7487a7ab343a4abca5312afe78862" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74" is_pre_election: true
I20260130 02:58:31.557969 19807 tablet_bootstrap.cc:654] T 6f5a90b75a0f4851b5db672c41253f7b P 371a36d219164974abdf848841528152: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:31.558105 19520 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "66a7487a7ab343a4abca5312afe78862" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
I20260130 02:58:31.565357 19159 leader_election.cc:304] T 66a7487a7ab343a4abca5312afe78862 P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 371a36d219164974abdf848841528152; no voters: 78c2c46c7520492db35d51c244fb4f74, d870164263bb44b6b780a557daea3656
I20260130 02:58:31.566948 21353 raft_consensus.cc:2749] T 66a7487a7ab343a4abca5312afe78862 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260130 02:58:31.645576 19807 tablet_bootstrap.cc:492] T 6f5a90b75a0f4851b5db672c41253f7b P 371a36d219164974abdf848841528152: No bootstrap required, opened a new log
I20260130 02:58:31.645987 19807 ts_tablet_manager.cc:1403] T 6f5a90b75a0f4851b5db672c41253f7b P 371a36d219164974abdf848841528152: Time spent bootstrapping tablet: real 0.220s user 0.004s sys 0.006s
I20260130 02:58:31.654407 19807 raft_consensus.cc:359] T 6f5a90b75a0f4851b5db672c41253f7b P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:31.656136 19807 raft_consensus.cc:385] T 6f5a90b75a0f4851b5db672c41253f7b P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:31.656375 19807 raft_consensus.cc:740] T 6f5a90b75a0f4851b5db672c41253f7b P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 371a36d219164974abdf848841528152, State: Initialized, Role: FOLLOWER
I20260130 02:58:31.657418 19807 consensus_queue.cc:260] T 6f5a90b75a0f4851b5db672c41253f7b P 371a36d219164974abdf848841528152 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:31.659475 19807 ts_tablet_manager.cc:1434] T 6f5a90b75a0f4851b5db672c41253f7b P 371a36d219164974abdf848841528152: Time spent starting tablet: real 0.013s user 0.007s sys 0.000s
I20260130 02:58:31.738672 19222 raft_consensus.cc:3060] T 155ff0c35fc44cb292a28407e6ed937d P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:31.763454 19224 raft_consensus.cc:3060] T 66a7487a7ab343a4abca5312afe78862 P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:31.822184 19221 raft_consensus.cc:3060] T 475b435b2967418f927fa172d2ff04ea P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Advancing to term 1
W20260130 02:58:31.842103 18860 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.100.125:39711, user_credentials={real_user=slave}} blocked reactor thread for 66512.2us
I20260130 02:58:31.992034 21463 raft_consensus.cc:493] T 6f5a90b75a0f4851b5db672c41253f7b P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:31.992470 21463 raft_consensus.cc:515] T 6f5a90b75a0f4851b5db672c41253f7b P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:31.994141 21463 leader_election.cc:290] T 6f5a90b75a0f4851b5db672c41253f7b P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d870164263bb44b6b780a557daea3656 (127.18.100.67:39571), 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767)
I20260130 02:58:32.005316 19366 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6f5a90b75a0f4851b5db672c41253f7b" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74" is_pre_election: true
I20260130 02:58:32.010974 19519 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6f5a90b75a0f4851b5db672c41253f7b" candidate_uuid: "371a36d219164974abdf848841528152" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
I20260130 02:58:32.028863 19224 raft_consensus.cc:3060] T 6f5a90b75a0f4851b5db672c41253f7b P 371a36d219164974abdf848841528152 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:32.031594 19159 leader_election.cc:304] T 6f5a90b75a0f4851b5db672c41253f7b P 371a36d219164974abdf848841528152 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 371a36d219164974abdf848841528152; no voters: 78c2c46c7520492db35d51c244fb4f74, d870164263bb44b6b780a557daea3656
W20260130 02:58:32.127104 19159 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.100.67:39571, user_credentials={real_user=slave}} blocked reactor thread for 49795.5us
I20260130 02:58:32.189282 21463 raft_consensus.cc:2749] T 6f5a90b75a0f4851b5db672c41253f7b P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260130 02:58:32.239574 19314 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.100.65:42021, user_credentials={real_user=slave}} blocked reactor thread for 49674.7us
I20260130 02:58:32.636024 21249 catalog_manager.cc:679] Loaded metadata for table default.Table_5e70ab0e32bc4048bd19388f9d7cd302 [id=030c8ac23aac4a1fb7acc954d09434db]
I20260130 02:58:32.637573 21249 catalog_manager.cc:679] Loaded metadata for table default.Table_dabf999abe834151a0a5b6747a05cac3 [id=038ba9aa798741f6858fa09002d6c3d6]
I20260130 02:58:32.639792 21249 catalog_manager.cc:679] Loaded metadata for table default.Table_ed6e2e20d89542199e5cbc7842df964d [id=047c39bd14664d1099e043b973766d32]
I20260130 02:58:32.663187 21249 catalog_manager.cc:679] Loaded metadata for table default.Table_fec49c37fdd045e7801325ace326f367 [id=06f1da4d6e0e46e4a141914a3aa68c07]
I20260130 02:58:32.668809 21249 catalog_manager.cc:679] Loaded metadata for table default.Table_4ee03db7d0064d96a53d4f261bd12722 [id=07fe2931234d4888a524b52171b03f0a]
I20260130 02:58:32.670356 21249 catalog_manager.cc:679] Loaded metadata for table default.Table_0480357296ac4a27812d802ee4566351 [id=0da6197ab32e4645aa8d8787662ee5e6]
I20260130 02:58:32.673002 21249 catalog_manager.cc:679] Loaded metadata for table default.Table_85611716686c4097bacb64df677d5793 [id=122fe719b1d64ec8b531783539f85df1]
I20260130 02:58:32.676960 21249 catalog_manager.cc:679] Loaded metadata for table default.Table_805e07e3546e447d8c8b8029477013cc [id=1515db389f284e2f9ed1a3cd0061d0f1]
I20260130 02:58:32.678336 21249 catalog_manager.cc:679] Loaded metadata for table default.Table_3a858a3cb886427cb8268f42c953ad4c [id=17150cb73e374ff9a1e92c43e0c3f2c1]
I20260130 02:58:32.681772 21249 catalog_manager.cc:679] Loaded metadata for table default.Table_ea2e0aa925a0486bb292304ca21bb819 [id=18580b9c54e448d1bdaae84f697f7b0b]
I20260130 02:58:32.688913 21249 catalog_manager.cc:679] Loaded metadata for table default.Table_9e942a1ce3944e9aa028e75656ce066d [id=18ed268eb9ad4450abf9874310ace074]
I20260130 02:58:32.694908 21502 raft_consensus.cc:493] T 00000000000000000000000000000000 P 3749baa287ed4f249329a74071781edd [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 658299197a424f53a37f14fadf089ae4)
I20260130 02:58:32.697971 21249 catalog_manager.cc:679] Loaded metadata for table default.Table_a132cf962cfc4ed8a6a8c177e088daeb [id=1c4937ca6e014fec8596a3dc25779600]
I20260130 02:58:32.698473 21502 raft_consensus.cc:515] T 00000000000000000000000000000000 P 3749baa287ed4f249329a74071781edd [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "658299197a424f53a37f14fadf089ae4" member_type: VOTER last_known_addr { host: "127.18.100.126" port: 38191 } } peers { permanent_uuid: "107f274cdda543038832f06a56ab9130" member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39711 } } peers { permanent_uuid: "3749baa287ed4f249329a74071781edd" member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34629 } }
I20260130 02:58:32.705466 21501 raft_consensus.cc:493] T 00000000000000000000000000000000 P 107f274cdda543038832f06a56ab9130 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 658299197a424f53a37f14fadf089ae4)
I20260130 02:58:32.706108 21501 raft_consensus.cc:515] T 00000000000000000000000000000000 P 107f274cdda543038832f06a56ab9130 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "658299197a424f53a37f14fadf089ae4" member_type: VOTER last_known_addr { host: "127.18.100.126" port: 38191 } } peers { permanent_uuid: "107f274cdda543038832f06a56ab9130" member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39711 } } peers { permanent_uuid: "3749baa287ed4f249329a74071781edd" member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34629 } }
I20260130 02:58:32.712077 18960 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3749baa287ed4f249329a74071781edd" candidate_term: 7 candidate_status { last_received { term: 6 index: 374 } } ignore_live_leader: false dest_uuid: "107f274cdda543038832f06a56ab9130" is_pre_election: true
I20260130 02:58:32.712733 18960 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 107f274cdda543038832f06a56ab9130 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3749baa287ed4f249329a74071781edd in term 6.
I20260130 02:58:32.714201 20627 leader_election.cc:304] T 00000000000000000000000000000000 P 3749baa287ed4f249329a74071781edd [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 107f274cdda543038832f06a56ab9130, 3749baa287ed4f249329a74071781edd; no voters:
I20260130 02:58:32.711972 21502 leader_election.cc:290] T 00000000000000000000000000000000 P 3749baa287ed4f249329a74071781edd [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 658299197a424f53a37f14fadf089ae4 (127.18.100.126:38191), 107f274cdda543038832f06a56ab9130 (127.18.100.125:39711)
I20260130 02:58:32.724817 21502 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 3749baa287ed4f249329a74071781edd [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260130 02:58:32.727726 21249 catalog_manager.cc:679] Loaded metadata for table default.Table_ddfb5df90fdb40a087ac434d088bbf0e [id=1e72b7dd669d4539b61884ba63ff7183]
I20260130 02:58:32.727427 21502 raft_consensus.cc:493] T 00000000000000000000000000000000 P 3749baa287ed4f249329a74071781edd [term 6 FOLLOWER]: Starting leader election (detected failure of leader 658299197a424f53a37f14fadf089ae4)
I20260130 02:58:32.732843 21502 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 3749baa287ed4f249329a74071781edd [term 6 FOLLOWER]: Advancing to term 7
I20260130 02:58:32.760181 20807 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "107f274cdda543038832f06a56ab9130" candidate_term: 7 candidate_status { last_received { term: 6 index: 374 } } ignore_live_leader: false dest_uuid: "3749baa287ed4f249329a74071781edd" is_pre_election: true
I20260130 02:58:32.783861 21501 leader_election.cc:290] T 00000000000000000000000000000000 P 107f274cdda543038832f06a56ab9130 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 658299197a424f53a37f14fadf089ae4 (127.18.100.126:38191), 3749baa287ed4f249329a74071781edd (127.18.100.124:34629)
I20260130 02:58:32.821853 21249 catalog_manager.cc:679] Loaded metadata for table default.Table_e03bdbb221e94bd6b535ddc02c863bad [id=2382eeebdc9540d5b588835705b2a4f1]
I20260130 02:58:32.833961 21249 catalog_manager.cc:679] Loaded metadata for table default.Table_ebc8b8379c274c18b0a06a919dbe0ee0 [id=28156e0843674fce88f5914a3e4195ae]
I20260130 02:58:32.841113 21249 catalog_manager.cc:679] Loaded metadata for table default.Table_bc168ff8b5f34cacbd398117647ba25a [id=2a21ee52b7d246f1b44eda02bf5a411e]
I20260130 02:58:32.842447 21249 catalog_manager.cc:679] Loaded metadata for table default.Table_f619de75da8e4d23b4d641d30285e0e7 [id=2dc3a57f750e4104b99f4589fdb29e3c]
I20260130 02:58:32.897852 21249 catalog_manager.cc:679] Loaded metadata for table default.Table_fa63c9eff99d4ac1b185a4c4ce2df031 [id=2e8b378422c74abc939955d110737316]
I20260130 02:58:32.894196 21502 raft_consensus.cc:515] T 00000000000000000000000000000000 P 3749baa287ed4f249329a74071781edd [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "658299197a424f53a37f14fadf089ae4" member_type: VOTER last_known_addr { host: "127.18.100.126" port: 38191 } } peers { permanent_uuid: "107f274cdda543038832f06a56ab9130" member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39711 } } peers { permanent_uuid: "3749baa287ed4f249329a74071781edd" member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34629 } }
I20260130 02:58:32.933279 20807 raft_consensus.cc:2393] T 00000000000000000000000000000000 P 3749baa287ed4f249329a74071781edd [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 107f274cdda543038832f06a56ab9130 in current term 7: Already voted for candidate 3749baa287ed4f249329a74071781edd in this term.
I20260130 02:58:32.938949 20810 raft_consensus.cc:1240] T 00000000000000000000000000000000 P 3749baa287ed4f249329a74071781edd [term 7 FOLLOWER]: Rejecting Update request from peer 658299197a424f53a37f14fadf089ae4 for earlier term 6. Current term is 7. Ops: []
I20260130 02:58:32.941926 21507 consensus_queue.cc:1059] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3749baa287ed4f249329a74071781edd" member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34629 }, Status: INVALID_TERM, Last received: 6.374, Next index: 375, Last known committed idx: 374, Time since last communication: 0.001s
I20260130 02:58:32.943297 21508 raft_consensus.cc:3055] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4 [term 6 LEADER]: Stepping down as leader of term 6
I20260130 02:58:32.943729 21508 raft_consensus.cc:740] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 658299197a424f53a37f14fadf089ae4, State: Running, Role: LEADER
I20260130 02:58:32.944903 21508 consensus_queue.cc:260] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 374, Committed index: 374, Last appended: 6.374, Last appended by leader: 374, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "658299197a424f53a37f14fadf089ae4" member_type: VOTER last_known_addr { host: "127.18.100.126" port: 38191 } } peers { permanent_uuid: "107f274cdda543038832f06a56ab9130" member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39711 } } peers { permanent_uuid: "3749baa287ed4f249329a74071781edd" member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34629 } }
I20260130 02:58:32.946008 21502 leader_election.cc:290] T 00000000000000000000000000000000 P 3749baa287ed4f249329a74071781edd [CANDIDATE]: Term 7 election: Requested vote from peers 658299197a424f53a37f14fadf089ae4 (127.18.100.126:38191), 107f274cdda543038832f06a56ab9130 (127.18.100.125:39711)
I20260130 02:58:32.946182 21508 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4 [term 6 FOLLOWER]: Advancing to term 7
I20260130 02:58:32.947962 18960 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3749baa287ed4f249329a74071781edd" candidate_term: 7 candidate_status { last_received { term: 6 index: 374 } } ignore_live_leader: false dest_uuid: "107f274cdda543038832f06a56ab9130"
I20260130 02:58:32.963922 21249 catalog_manager.cc:679] Loaded metadata for table default.Table_1e42d571631143088cf33d4c3aafed28 [id=30ba2af6dffd4f7e8dcbfdd2f3798157]
I20260130 02:58:32.990264 21249 catalog_manager.cc:679] Loaded metadata for table default.Table_cda216a3bfc14ec0862b998bedd5b300 [id=35978c9d901c47eb81e565e37493b1ae]
I20260130 02:58:32.996155 21249 catalog_manager.cc:679] Loaded metadata for table default.Table_19677b801d1f4a63b22139b9fc80c4ef [id=3992cda3a7bb49ae86b8f84ee12f1c8b]
I20260130 02:58:33.026614 21249 catalog_manager.cc:679] Loaded metadata for table default.Table_0f3e44ae93bc4c028ed3929b94a49a24 [id=3b9c62fc8bbb44ffb7d66e9604974c30]
I20260130 02:58:33.038085 21249 catalog_manager.cc:679] Loaded metadata for table default.Table_8f632ff84d624b72a10cb90fb9e78636 [id=3cf79022ea954f6397f0a0497e6e6a6c]
I20260130 02:58:33.049075 21249 catalog_manager.cc:679] Loaded metadata for table default.Table_d764cbaecfee476ba5e734dd1e2f50fd [id=3f9236f0631a4e4c94ca7456f0478a45]
I20260130 02:58:33.062381 21249 catalog_manager.cc:679] Loaded metadata for table default.Table_0ca1d7370d2f416782b386c5b47111ab [id=3ffe1a273d4f452c835278a8e7f10102]
I20260130 02:58:33.078528 21249 catalog_manager.cc:679] Loaded metadata for table default.Table_f17d8e4d9f25449b84aa751af93936b2 [id=43254e466e7b4a2fb59fbcec68cfde36]
I20260130 02:58:33.090786 21249 catalog_manager.cc:679] Loaded metadata for table default.Table_6f53c17cfd144031b2d4b112fb2d88eb [id=443cbad5510d463fa8ca878b8628dbda]
I20260130 02:58:33.103426 21249 catalog_manager.cc:679] Loaded metadata for table default.Table_8896365e792f4d24b2c16c2d269b0a93 [id=44816e07033849479a7fea0f51d00fad]
I20260130 02:58:33.113226 21249 catalog_manager.cc:679] Loaded metadata for table default.Table_2d345d0f0957486c86c2cf2e34b00941 [id=4c3e0d3ecd5642a3b7fc7d20e2521418]
I20260130 02:58:33.124481 21249 catalog_manager.cc:679] Loaded metadata for table default.Table_8f311c5878744ef692b496ecfdfc3ede [id=4f499357303d464aafc17679ffcaaa07]
I20260130 02:58:33.136094 21249 catalog_manager.cc:679] Loaded metadata for table default.Table_d2a3438ed66741eab245ae6d6c5c2f37 [id=4fb5c26265e94083a8714be512fcd222]
I20260130 02:58:33.144703 21249 catalog_manager.cc:679] Loaded metadata for table default.Table_21178f657dca43a1912ce596834d1618 [id=53b79067a6034189aac5f1554df542c4]
I20260130 02:58:33.152012 21249 catalog_manager.cc:679] Loaded metadata for table default.Table_12b505700b3d44e28353172e1786e5c8 [id=562776405d6f4c7aab0622f3866bf45b]
I20260130 02:58:33.163848 21249 catalog_manager.cc:679] Loaded metadata for table default.Table_aa4e5fd22945475da1dad48e1508595e [id=5b41834c6d8b4d89b773c2ea4c4ca7e6]
I20260130 02:58:33.185832 21249 catalog_manager.cc:679] Loaded metadata for table default.Table_9cb9a9f9e5d145a6a42d7ccbc42aced9 [id=5f1f179fadc247da9382dd7796dcf2c2]
I20260130 02:58:33.188031 21249 catalog_manager.cc:679] Loaded metadata for table default.Table_9d0cd9756483406081c8bdb18565a42f [id=5f6488df8ba94ef293b181328cc3b680]
I20260130 02:58:33.208386 21249 catalog_manager.cc:679] Loaded metadata for table default.Table_d95381c590c94802bdc83acef0f0df96 [id=64a8e0e079f34806aa68795f3d548c5e]
I20260130 02:58:33.209882 21249 catalog_manager.cc:679] Loaded metadata for table default.Table_dee034aec0b24982b8dc2a3fa1b326f1 [id=66033e75b3814a8584311cf7e4a07ad3]
I20260130 02:58:33.211937 21249 catalog_manager.cc:679] Loaded metadata for table default.Table_6a7faee4c75c4eb7862fb4c05afc17cb [id=676ece9dc36b418a8b1400e8406d6b98]
I20260130 02:58:33.213196 21249 catalog_manager.cc:679] Loaded metadata for table default.Table_12c2fe20cc3e45d29b7be37f900f68a9 [id=67ac4f4eda324e799b5867b2b011a16b]
I20260130 02:58:33.229363 21249 catalog_manager.cc:679] Loaded metadata for table default.Table_6c46e6995d4741dcaaa6ed7bdc765e29 [id=6d91a161cf484239967af58db160c952]
I20260130 02:58:33.235627 21249 catalog_manager.cc:679] Loaded metadata for table default.Table_e8fd7516280f4dc4a73eb1f88fe916a2 [id=6efb836791414f3888f714aa8f65f0a4]
I20260130 02:58:33.237103 21249 catalog_manager.cc:679] Loaded metadata for table default.Table_5c1c447e85524ac5a61ffd8a48509d36 [id=70aaf21ad9bd4d959f46939c6b4a4880]
W20260130 02:58:33.268944 18934 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.18.100.126:38191 timed out after 0.556s (SENT) [suppressed 1 similar messages]
I20260130 02:58:33.277201 21249 catalog_manager.cc:679] Loaded metadata for table default.Table_7fe8c354c0dc4cfba0e84ed33e4c2506 [id=7523d671e9ac4814acff5060fd282eea]
I20260130 02:58:33.280611 21249 catalog_manager.cc:679] Loaded metadata for table default.Table_f3e214a955f6447482099175b92ecb5c [id=76a688022fd9421a84de059b3c1da77a]
I20260130 02:58:33.282040 21249 catalog_manager.cc:679] Loaded metadata for table default.Table_dd22e9dc63f84d8688c0071f427f0b25 [id=77979d498aa040828de415129cb15a06]
I20260130 02:58:33.283438 21249 catalog_manager.cc:679] Loaded metadata for table default.Table_007e969f70d345a7bcb3dfbaabbc025d [id=77d4fa8705a3419cb0a84d3ca1bb4ce3]
I20260130 02:58:33.284796 21249 catalog_manager.cc:679] Loaded metadata for table default.Table_aa496bf2019748f29823a9a253b1f3ca [id=780188e2335f43fbb11de8212eb38d56]
I20260130 02:58:33.291117 21249 catalog_manager.cc:679] Loaded metadata for table default.Table_c1886dd9a7934e7797e0e7e0b065d692 [id=7a2d5367224b492b844199088b01b149]
I20260130 02:58:33.294003 21249 catalog_manager.cc:679] Loaded metadata for table default.Table_69267d31df644f4396a44130fa3d499d [id=7efa53800a974e31a38a4c60c0a1b466]
I20260130 02:58:33.297042 21249 catalog_manager.cc:679] Loaded metadata for table default.Table_b85cb876f25740bfaaab01949ac26b38 [id=839e292e040c4aeb832f3fc0258c32ec]
I20260130 02:58:33.298385 21249 catalog_manager.cc:679] Loaded metadata for table default.Table_1ae68ae3f0404843aca3c72f02678ac6 [id=8a9182f49e18489fafec8dca190c1b2f]
I20260130 02:58:33.299599 21249 catalog_manager.cc:679] Loaded metadata for table default.Table_333d2efbf068485a871f566d50e563ce [id=8b7db40afb894ae99625a756c75ef1de]
I20260130 02:58:33.300771 21249 catalog_manager.cc:679] Loaded metadata for table default.Table_1703c69112744c0e951c1ac9667bbac6 [id=8d76f4328edc491291491c78ba205d30]
I20260130 02:58:33.301921 21249 catalog_manager.cc:679] Loaded metadata for table default.Table_6773a0ad5d6a429086a5818d731f3f4a [id=90b72499c7174fd580914d7fe3714891]
I20260130 02:58:33.306463 21249 catalog_manager.cc:679] Loaded metadata for table default.Table_829d1e187e8b4db5b8f4f835e84246e5 [id=a24b4761037d47a383f454a82517131f]
I20260130 02:58:33.307741 21249 catalog_manager.cc:679] Loaded metadata for table default.Table_f8856f06ce97440bae4abdf34164716d [id=a3303ada20a94f99b2f37a02eec361ef]
I20260130 02:58:33.308959 21249 catalog_manager.cc:679] Loaded metadata for table default.Table_2e81f0a0cbc449a397141b717db17025 [id=a9c919f57c0c437cb35a6921103e45a6]
I20260130 02:58:33.310073 21249 catalog_manager.cc:679] Loaded metadata for table default.Table_7c48a0d054354501947d11c5fd7b1864 [id=aa4ad1394ab344df8c54ba0031af5590]
I20260130 02:58:33.311888 21249 catalog_manager.cc:679] Loaded metadata for table default.Table_477fb92965284222aa1ecaa0107d1ec2 [id=ad82881a19e748edb569e74d692bb2bc]
I20260130 02:58:33.315534 21249 catalog_manager.cc:679] Loaded metadata for table default.Table_2414288f0dc54e5f835f876060927a9b [id=b0c6f2d9483c47da8ff3e65f0400a593]
I20260130 02:58:33.316658 21249 catalog_manager.cc:679] Loaded metadata for table default.Table_526677c8ef4144bd9de7a54a5e78e2be [id=b16908b75587459597a978c71756451c]
I20260130 02:58:33.318255 21249 catalog_manager.cc:679] Loaded metadata for table default.Table_78b080efddd94878b9d8b77b5ef3af35 [id=b654822c58224a859ad2dcdd8d0123c2]
I20260130 02:58:33.321110 21249 catalog_manager.cc:679] Loaded metadata for table default.Table_7d1c27ab3515406ba5b5333c90df093e [id=b6fabeeae79941dc8094d0f2b51704f9]
I20260130 02:58:33.328810 21249 catalog_manager.cc:679] Loaded metadata for table default.Table_95f0bff8e17d4ca995c0af0d0c24e84d [id=b7795905d0c440579c59d6ac00473f5a]
I20260130 02:58:33.337647 21249 catalog_manager.cc:679] Loaded metadata for table default.Table_da3d581b15a94840898777f74611994e [id=b7e241ced53e4160ac6fa1f65ef3674e]
W20260130 02:58:33.338428 18934 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.100.126:38191, user_credentials={real_user=slave}} blocked reactor thread for 69613.4us
I20260130 02:58:33.339506 21249 catalog_manager.cc:679] Loaded metadata for table default.Table_7e6ab9964f4c4e0f906d875a937a781d [id=bcf41af45d284bd89800977d14194812]
I20260130 02:58:33.340862 21249 catalog_manager.cc:679] Loaded metadata for table default.Table_d3f82845a18f4a4a830460cb1cbbb2de [id=c3d0c57ad165460eae9c20c32557fd2e]
I20260130 02:58:33.343855 21249 catalog_manager.cc:679] Loaded metadata for table default.Table_c4f4904b6fac4b9880979cf0314e71c1 [id=c456c6418c93499a9d08f652720854ca]
I20260130 02:58:33.351743 21249 catalog_manager.cc:679] Loaded metadata for table default.Table_81553b46d7804fd99ef36f431793b001 [id=cdfc1cfcc48f41f8a7a13f865174c3d8]
I20260130 02:58:33.354758 21249 catalog_manager.cc:679] Loaded metadata for table default.Table_aaf139c1a55d4a1783298ca12ae0d5c0 [id=d128767e303145a885eec3c5f714a27f]
I20260130 02:58:33.358161 21249 catalog_manager.cc:679] Loaded metadata for table default.Table_94b23422c82d419d97dc097a3efb92fc [id=d2f35437eae140dab4150079eae46377]
I20260130 02:58:33.361256 21249 catalog_manager.cc:679] Loaded metadata for table default.Table_e894e5560f194692bac730600c6bf5f4 [id=d5e1b2a45122429aabb4fd8653d71f0f]
I20260130 02:58:33.363231 21249 catalog_manager.cc:679] Loaded metadata for table default.Table_22d10332a83f4700ac21892bab67f46b [id=da3a9cb1f6d5419abc1afc93659b5324]
I20260130 02:58:33.364606 21249 catalog_manager.cc:679] Loaded metadata for table default.Table_9f83f87711d94e8394abe1925ac84ad1 [id=db23c1be42c44b6f9eb774f14d2b8fd6]
I20260130 02:58:33.365728 21249 catalog_manager.cc:679] Loaded metadata for table default.Table_109c8b6c17d4442b939bde9441d0476f [id=dcf82ea15d6044c2b023f81f0a579ade]
I20260130 02:58:33.368049 21249 catalog_manager.cc:679] Loaded metadata for table default.Table_baaf376733384fdbbac49ec8f2357432 [id=dea97f19a50141878f97820194afa59e]
I20260130 02:58:33.369839 21249 catalog_manager.cc:679] Loaded metadata for table default.Table_8c27a7d10b7847e78029f7e95321e9b6 [id=e31c1bf0c3b9457fa63efe71138c259c]
I20260130 02:58:33.371172 21249 catalog_manager.cc:679] Loaded metadata for table default.Table_92836e7fcf3a48d8bac2160c2ff7c482 [id=e3ae4b66fa0048668cb1e4d9da7d16fa]
I20260130 02:58:33.372325 21249 catalog_manager.cc:679] Loaded metadata for table default.Table_d24e400f7451443a82d23140a90adc57 [id=e478df970dac40f7a576ce584c3abf2c]
I20260130 02:58:33.373430 21249 catalog_manager.cc:679] Loaded metadata for table default.Table_2f60fd0292ab40c8a3ae9b842cae8faf [id=e648e521ef5044a0b1967452cc3a31b2]
I20260130 02:58:33.374547 21249 catalog_manager.cc:679] Loaded metadata for table default.Table_ba19837ced4c45d79be506e93308449b [id=e732719fbe544e38a15b35017010c7b3]
I20260130 02:58:33.376593 21249 catalog_manager.cc:679] Loaded metadata for table default.Table_c238448062ad4cef8ddd0bd199d5e7f2 [id=e8d3be5336974867a5d65d1ceb4433d6]
I20260130 02:58:33.379740 21249 catalog_manager.cc:679] Loaded metadata for table default.Table_a3ecda6d64704b7fad065a5845683ad8 [id=e93360fd2af141a28d92401ed449c19d]
I20260130 02:58:33.383765 21249 catalog_manager.cc:679] Loaded metadata for table default.Table_3046889233c24f0a8e8058e1591d63b8 [id=edd6505c0bf3462cbf65c3df5649c16c]
I20260130 02:58:33.385641 21249 catalog_manager.cc:679] Loaded metadata for table default.Table_d0b4516d24384e5386c1debaf2543adb [id=f1342f9497b74e7a9ecad1de8082578d]
I20260130 02:58:33.387491 21249 catalog_manager.cc:679] Loaded metadata for table default.Table_5d5e26f93a5a4fb084831add317d90f9 [id=f237b1c90a1743d597d87c5cf1abc89a]
I20260130 02:58:33.390945 21249 catalog_manager.cc:679] Loaded metadata for table default.Table_1e9e20a1bfec4bcf81f9eb6219c33d59 [id=f675b7a3ec744533903d8834eeed51d7]
I20260130 02:58:33.396607 21249 catalog_manager.cc:679] Loaded metadata for table default.Table_f246882197f44c69b72ca421cbdf201f [id=f778784dc88a4e4890113916098ad20f]
I20260130 02:58:33.397825 21249 catalog_manager.cc:679] Loaded metadata for table default.Table_2c737f14bf854cfca58d4a9f83fb2e9e [id=fa4e8955bebb40b6adf6ed4ce0e43bfb]
I20260130 02:58:33.399055 21249 catalog_manager.cc:679] Loaded metadata for table default.Table_e2480022cdae4429b710395eebae7fec [id=fa72411018324474aa9a7265d9f1a2e4]
I20260130 02:58:33.400175 21249 catalog_manager.cc:679] Loaded metadata for table default.Table_e4d07634d84a4678b1c94374e3adc476 [id=fcbd7e3a7fc64d89aa740494d7e1a0f7]
I20260130 02:58:33.401288 21249 catalog_manager.cc:679] Loaded metadata for table default.Table_203b3a960d3e4b5cb978b6168435f3a4 [id=ffe264c47e31462a8deaf81a8b96d34c]
W20260130 02:58:33.415917 20613 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.18.100.126:38191 timed out after 0.708s (SENT) [suppressed 1 similar messages]
I20260130 02:58:33.458652 21249 tablet_loader.cc:96] loaded metadata for tablet 00ddbfe6218c44d788fbf65245a31dc6 (table default.Table_9cb9a9f9e5d145a6a42d7ccbc42aced9 [id=5f1f179fadc247da9382dd7796dcf2c2])
I20260130 02:58:33.461737 21249 tablet_loader.cc:96] loaded metadata for tablet 03025d13f0e545cfa6288677a2715343 (table default.Table_21178f657dca43a1912ce596834d1618 [id=53b79067a6034189aac5f1554df542c4])
I20260130 02:58:33.464946 21249 tablet_loader.cc:96] loaded metadata for tablet 0855d88806914a36b4bf6b49036f0ad0 (table default.Table_6773a0ad5d6a429086a5818d731f3f4a [id=90b72499c7174fd580914d7fe3714891])
I20260130 02:58:33.466058 21249 tablet_loader.cc:96] loaded metadata for tablet 0a763e48583c4e748a46a325746ce69a (table default.Table_94b23422c82d419d97dc097a3efb92fc [id=d2f35437eae140dab4150079eae46377])
I20260130 02:58:33.474076 21249 tablet_loader.cc:96] loaded metadata for tablet 0b1877e5f9244913bcd639f1336df7ee (table default.Table_19677b801d1f4a63b22139b9fc80c4ef [id=3992cda3a7bb49ae86b8f84ee12f1c8b])
I20260130 02:58:33.475212 21249 tablet_loader.cc:96] loaded metadata for tablet 0d14fb95ace9447f810ccbed4d5befdf (table default.Table_c238448062ad4cef8ddd0bd199d5e7f2 [id=e8d3be5336974867a5d65d1ceb4433d6])
I20260130 02:58:33.476208 21249 tablet_loader.cc:96] loaded metadata for tablet 0e8018c094024c4788c484e8f5466e00 (table default.Table_f17d8e4d9f25449b84aa751af93936b2 [id=43254e466e7b4a2fb59fbcec68cfde36])
I20260130 02:58:33.477205 21249 tablet_loader.cc:96] loaded metadata for tablet 111287b5557d4abda1c5cd560fb0d019 (table default.Table_fec49c37fdd045e7801325ace326f367 [id=06f1da4d6e0e46e4a141914a3aa68c07])
I20260130 02:58:33.479104 21249 tablet_loader.cc:96] loaded metadata for tablet 127a84346b9f473d9e7f739b1e65feb8 (table default.Table_805e07e3546e447d8c8b8029477013cc [id=1515db389f284e2f9ed1a3cd0061d0f1])
I20260130 02:58:33.480333 21249 tablet_loader.cc:96] loaded metadata for tablet 12c3f3ec523040e7b7cd356e3da57e0d (table default.Table_81553b46d7804fd99ef36f431793b001 [id=cdfc1cfcc48f41f8a7a13f865174c3d8])
I20260130 02:58:33.481359 21249 tablet_loader.cc:96] loaded metadata for tablet 133e9bfe175f44aab1a53114abe67e4d (table default.Table_d3f82845a18f4a4a830460cb1cbbb2de [id=c3d0c57ad165460eae9c20c32557fd2e])
I20260130 02:58:33.482692 21249 tablet_loader.cc:96] loaded metadata for tablet 13f0ace4d9de43abab08eca32cdb30c1 (table default.Table_9d0cd9756483406081c8bdb18565a42f [id=5f6488df8ba94ef293b181328cc3b680])
I20260130 02:58:33.483722 21249 tablet_loader.cc:96] loaded metadata for tablet 155ff0c35fc44cb292a28407e6ed937d (table default.Table_dabf999abe834151a0a5b6747a05cac3 [id=038ba9aa798741f6858fa09002d6c3d6])
I20260130 02:58:33.484759 21249 tablet_loader.cc:96] loaded metadata for tablet 15918c90ffdc43a5a38b2950e9834188 (table default.Table_333d2efbf068485a871f566d50e563ce [id=8b7db40afb894ae99625a756c75ef1de])
I20260130 02:58:33.485769 21249 tablet_loader.cc:96] loaded metadata for tablet 16691c9315e94253bb9b3fb5058e5636 (table default.Table_8f311c5878744ef692b496ecfdfc3ede [id=4f499357303d464aafc17679ffcaaa07])
I20260130 02:58:33.488086 21249 tablet_loader.cc:96] loaded metadata for tablet 1fc8f09d6329407c8078542b4123ef4d (table default.Table_7d1c27ab3515406ba5b5333c90df093e [id=b6fabeeae79941dc8094d0f2b51704f9])
I20260130 02:58:33.489715 21249 tablet_loader.cc:96] loaded metadata for tablet 25fd48e693424537b91ba7c3fcc96878 (table default.Table_ebc8b8379c274c18b0a06a919dbe0ee0 [id=28156e0843674fce88f5914a3e4195ae])
I20260130 02:58:33.491393 21249 tablet_loader.cc:96] loaded metadata for tablet 274530359b174ff79f046c2d4a6afa76 (table default.Table_c4f4904b6fac4b9880979cf0314e71c1 [id=c456c6418c93499a9d08f652720854ca])
I20260130 02:58:33.493094 21249 tablet_loader.cc:96] loaded metadata for tablet 2b0edaad6877460cb7982efc62b00b66 (table default.Table_d95381c590c94802bdc83acef0f0df96 [id=64a8e0e079f34806aa68795f3d548c5e])
I20260130 02:58:33.495802 21249 tablet_loader.cc:96] loaded metadata for tablet 2b8034a92b2043e5b57cf02f1ee6296d (table default.Table_e2480022cdae4429b710395eebae7fec [id=fa72411018324474aa9a7265d9f1a2e4])
I20260130 02:58:33.499089 21249 tablet_loader.cc:96] loaded metadata for tablet 2ca0df71836145748cada4e4559a5656 (table default.Table_d764cbaecfee476ba5e734dd1e2f50fd [id=3f9236f0631a4e4c94ca7456f0478a45])
I20260130 02:58:33.500902 21249 tablet_loader.cc:96] loaded metadata for tablet 306452e208ca4e719609f67345114a13 (table default.Table_fa63c9eff99d4ac1b185a4c4ce2df031 [id=2e8b378422c74abc939955d110737316])
I20260130 02:58:33.501914 21249 tablet_loader.cc:96] loaded metadata for tablet 30cc52136bb3422895d95c0fed0c6798 (table default.Table_f619de75da8e4d23b4d641d30285e0e7 [id=2dc3a57f750e4104b99f4589fdb29e3c])
I20260130 02:58:33.503558 21249 tablet_loader.cc:96] loaded metadata for tablet 320dbe98a32a43b9899bbeed80b20f55 (table default.Table_cda216a3bfc14ec0862b998bedd5b300 [id=35978c9d901c47eb81e565e37493b1ae])
I20260130 02:58:33.504419 21249 tablet_loader.cc:96] loaded metadata for tablet 3379641837df4b6ea360e8e27dfe4f8b (table default.Table_8f632ff84d624b72a10cb90fb9e78636 [id=3cf79022ea954f6397f0a0497e6e6a6c])
I20260130 02:58:33.506844 21249 tablet_loader.cc:96] loaded metadata for tablet 341041cb0be2456e88403367c0814704 (table default.Table_1703c69112744c0e951c1ac9667bbac6 [id=8d76f4328edc491291491c78ba205d30])
I20260130 02:58:33.511207 21249 tablet_loader.cc:96] loaded metadata for tablet 345eaaa5625c48d099280b7fbb2e514b (table default.Table_c1886dd9a7934e7797e0e7e0b065d692 [id=7a2d5367224b492b844199088b01b149])
I20260130 02:58:33.512244 21249 tablet_loader.cc:96] loaded metadata for tablet 3593f8b89da24ff9b6cf549d5ac58bc6 (table default.Table_2c737f14bf854cfca58d4a9f83fb2e9e [id=fa4e8955bebb40b6adf6ed4ce0e43bfb])
I20260130 02:58:33.513259 21249 tablet_loader.cc:96] loaded metadata for tablet 3738385130c044e0a5a5803da74c567d (table default.Table_ddfb5df90fdb40a087ac434d088bbf0e [id=1e72b7dd669d4539b61884ba63ff7183])
I20260130 02:58:33.516665 21249 tablet_loader.cc:96] loaded metadata for tablet 39bcc41f10ae4241b370529364454586 (table default.Table_a132cf962cfc4ed8a6a8c177e088daeb [id=1c4937ca6e014fec8596a3dc25779600])
I20260130 02:58:33.517889 21249 tablet_loader.cc:96] loaded metadata for tablet 3aa76993beb94b12bc706a76db26a03b (table default.Table_477fb92965284222aa1ecaa0107d1ec2 [id=ad82881a19e748edb569e74d692bb2bc])
I20260130 02:58:33.518947 21249 tablet_loader.cc:96] loaded metadata for tablet 3ae5472cba1e4613abe4cda42ccb2496 (table default.Table_69267d31df644f4396a44130fa3d499d [id=7efa53800a974e31a38a4c60c0a1b466])
I20260130 02:58:33.519977 21249 tablet_loader.cc:96] loaded metadata for tablet 3b02480d9c6f4328af83f7940716d698 (table default.Table_1e9e20a1bfec4bcf81f9eb6219c33d59 [id=f675b7a3ec744533903d8834eeed51d7])
I20260130 02:58:33.530195 21249 tablet_loader.cc:96] loaded metadata for tablet 475b435b2967418f927fa172d2ff04ea (table default.Table_dee034aec0b24982b8dc2a3fa1b326f1 [id=66033e75b3814a8584311cf7e4a07ad3])
I20260130 02:58:33.534231 21249 tablet_loader.cc:96] loaded metadata for tablet 47f574674aa7414588605ca38eb7200a (table default.Table_d2a3438ed66741eab245ae6d6c5c2f37 [id=4fb5c26265e94083a8714be512fcd222])
I20260130 02:58:33.536996 21249 tablet_loader.cc:96] loaded metadata for tablet 4907294442c145c6b03eb1de057c66a8 (table default.Table_ea2e0aa925a0486bb292304ca21bb819 [id=18580b9c54e448d1bdaae84f697f7b0b])
I20260130 02:58:33.538067 21249 tablet_loader.cc:96] loaded metadata for tablet 49efcfced0b74c48962393149ac2a500 (table default.Table_2e81f0a0cbc449a397141b717db17025 [id=a9c919f57c0c437cb35a6921103e45a6])
I20260130 02:58:33.561131 21508 raft_consensus.cc:493] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:33.561643 21508 raft_consensus.cc:515] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "658299197a424f53a37f14fadf089ae4" member_type: VOTER last_known_addr { host: "127.18.100.126" port: 38191 } } peers { permanent_uuid: "107f274cdda543038832f06a56ab9130" member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39711 } } peers { permanent_uuid: "3749baa287ed4f249329a74071781edd" member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34629 } }
I20260130 02:58:33.565274 18960 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "658299197a424f53a37f14fadf089ae4" candidate_term: 8 candidate_status { last_received { term: 6 index: 374 } } ignore_live_leader: false dest_uuid: "107f274cdda543038832f06a56ab9130" is_pre_election: true
I20260130 02:58:33.565824 18960 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 107f274cdda543038832f06a56ab9130 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 658299197a424f53a37f14fadf089ae4 in term 6.
I20260130 02:58:33.566452 21508 leader_election.cc:290] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 107f274cdda543038832f06a56ab9130 (127.18.100.125:39711), 3749baa287ed4f249329a74071781edd (127.18.100.124:34629)
I20260130 02:58:33.567451 20810 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "658299197a424f53a37f14fadf089ae4" candidate_term: 8 candidate_status { last_received { term: 6 index: 374 } } ignore_live_leader: false dest_uuid: "3749baa287ed4f249329a74071781edd" is_pre_election: true
I20260130 02:58:33.567981 20810 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 3749baa287ed4f249329a74071781edd [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 658299197a424f53a37f14fadf089ae4 in term 7.
I20260130 02:58:33.569000 18860 leader_election.cc:304] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3749baa287ed4f249329a74071781edd, 658299197a424f53a37f14fadf089ae4; no voters:
I20260130 02:58:33.569847 21508 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260130 02:58:33.570214 21508 raft_consensus.cc:493] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260130 02:58:33.570561 21508 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4 [term 7 FOLLOWER]: Advancing to term 8
I20260130 02:58:33.576668 21249 tablet_loader.cc:96] loaded metadata for tablet 4c139aa8cd184f488cec4e05ef7dcada (table default.Table_e4d07634d84a4678b1c94374e3adc476 [id=fcbd7e3a7fc64d89aa740494d7e1a0f7])
I20260130 02:58:33.577837 21249 tablet_loader.cc:96] loaded metadata for tablet 532e867fd5944d468d7672d2a5c92f3c (table default.Table_0ca1d7370d2f416782b386c5b47111ab [id=3ffe1a273d4f452c835278a8e7f10102])
I20260130 02:58:33.585008 21249 tablet_loader.cc:96] loaded metadata for tablet 54be22352e404ff9b798a075693f7f21 (table default.Table_3a858a3cb886427cb8268f42c953ad4c [id=17150cb73e374ff9a1e92c43e0c3f2c1])
I20260130 02:58:33.603891 21249 tablet_loader.cc:96] loaded metadata for tablet 5fbe92a2227d44d88b4531a7f8aa0e08 (table default.Table_a3ecda6d64704b7fad065a5845683ad8 [id=e93360fd2af141a28d92401ed449c19d])
I20260130 02:58:33.605053 21249 tablet_loader.cc:96] loaded metadata for tablet 6218f2f7d73a42428b57a45858f53afd (table default.Table_95f0bff8e17d4ca995c0af0d0c24e84d [id=b7795905d0c440579c59d6ac00473f5a])
I20260130 02:58:33.605974 21493 raft_consensus.cc:493] T 920f52b975ab480fbf42656171222f09 P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:33.606423 21493 raft_consensus.cc:515] T 920f52b975ab480fbf42656171222f09 P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:33.608457 21493 leader_election.cc:290] T 920f52b975ab480fbf42656171222f09 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d870164263bb44b6b780a557daea3656 (127.18.100.67:39571), 371a36d219164974abdf848841528152 (127.18.100.65:42021)
I20260130 02:58:33.609292 19519 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "920f52b975ab480fbf42656171222f09" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656" is_pre_election: true
I20260130 02:58:33.610092 19519 raft_consensus.cc:2468] T 920f52b975ab480fbf42656171222f09 P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78c2c46c7520492db35d51c244fb4f74 in term 1.
I20260130 02:58:33.611370 19313 leader_election.cc:304] T 920f52b975ab480fbf42656171222f09 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78c2c46c7520492db35d51c244fb4f74, d870164263bb44b6b780a557daea3656; no voters:
I20260130 02:58:33.612560 19224 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "920f52b975ab480fbf42656171222f09" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152" is_pre_election: true
I20260130 02:58:33.613070 19224 raft_consensus.cc:2468] T 920f52b975ab480fbf42656171222f09 P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78c2c46c7520492db35d51c244fb4f74 in term 1.
I20260130 02:58:33.614032 21510 raft_consensus.cc:2804] T 920f52b975ab480fbf42656171222f09 P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260130 02:58:33.614363 21510 raft_consensus.cc:493] T 920f52b975ab480fbf42656171222f09 P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260130 02:58:33.614672 21510 raft_consensus.cc:3060] T 920f52b975ab480fbf42656171222f09 P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Advancing to term 2
I20260130 02:58:33.628011 21249 tablet_loader.cc:96] loaded metadata for tablet 657e9b0d8c05413f904129dce745710b (table default.Table_526677c8ef4144bd9de7a54a5e78e2be [id=b16908b75587459597a978c71756451c])
I20260130 02:58:33.629164 21249 tablet_loader.cc:96] loaded metadata for tablet 66a7487a7ab343a4abca5312afe78862 (table default.Table_6c46e6995d4741dcaaa6ed7bdc765e29 [id=6d91a161cf484239967af58db160c952])
I20260130 02:58:33.630115 21249 tablet_loader.cc:96] loaded metadata for tablet 6ad6bb8c986a433d94c745f2b47266ff (table default.Table_bc168ff8b5f34cacbd398117647ba25a [id=2a21ee52b7d246f1b44eda02bf5a411e])
I20260130 02:58:33.631245 21249 tablet_loader.cc:96] loaded metadata for tablet 6d1f121f49cd47c2b8247daaf6874e01 (table default.Table_0480357296ac4a27812d802ee4566351 [id=0da6197ab32e4645aa8d8787662ee5e6])
I20260130 02:58:33.632472 21249 tablet_loader.cc:96] loaded metadata for tablet 6e12a8eeb3204325a9a8b7779f72ab29 (table default.Table_8c27a7d10b7847e78029f7e95321e9b6 [id=e31c1bf0c3b9457fa63efe71138c259c])
I20260130 02:58:33.633764 21249 tablet_loader.cc:96] loaded metadata for tablet 6f5a90b75a0f4851b5db672c41253f7b (table default.Table_6f53c17cfd144031b2d4b112fb2d88eb [id=443cbad5510d463fa8ca878b8628dbda])
I20260130 02:58:33.635520 21249 tablet_loader.cc:96] loaded metadata for tablet 753344737b5941d9a93b6779e72f63d4 (table default.Table_8896365e792f4d24b2c16c2d269b0a93 [id=44816e07033849479a7fea0f51d00fad])
I20260130 02:58:33.636598 21249 tablet_loader.cc:96] loaded metadata for tablet 7783db3568f4434e91e9e456799cc312 (table default.Table_2d345d0f0957486c86c2cf2e34b00941 [id=4c3e0d3ecd5642a3b7fc7d20e2521418])
I20260130 02:58:33.640413 21510 raft_consensus.cc:515] T 920f52b975ab480fbf42656171222f09 P 78c2c46c7520492db35d51c244fb4f74 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:33.647670 21249 tablet_loader.cc:96] loaded metadata for tablet 7caf7cbad5054396b5617094e4d536a0 (table default.Table_007e969f70d345a7bcb3dfbaabbc025d [id=77d4fa8705a3419cb0a84d3ca1bb4ce3])
I20260130 02:58:33.650949 21249 tablet_loader.cc:96] loaded metadata for tablet 838e49e5e8874db393bdc15a1c92f87e (table default.Table_da3d581b15a94840898777f74611994e [id=b7e241ced53e4160ac6fa1f65ef3674e])
I20260130 02:58:33.653702 21249 tablet_loader.cc:96] loaded metadata for tablet 8b47653c7c3945a68e9bf4680bfc0496 (table default.Table_1ae68ae3f0404843aca3c72f02678ac6 [id=8a9182f49e18489fafec8dca190c1b2f])
I20260130 02:58:33.656960 21249 tablet_loader.cc:96] loaded metadata for tablet 91f954f3ef4244c9a575a7421a7e75eb (table default.Table_9f83f87711d94e8394abe1925ac84ad1 [id=db23c1be42c44b6f9eb774f14d2b8fd6])
I20260130 02:58:33.658290 21249 tablet_loader.cc:96] loaded metadata for tablet 920f52b975ab480fbf42656171222f09 (table default.Table_78b080efddd94878b9d8b77b5ef3af35 [id=b654822c58224a859ad2dcdd8d0123c2])
I20260130 02:58:33.660754 21249 tablet_loader.cc:96] loaded metadata for tablet 94355fc1338f479e8798070aff04fcba (table default.Table_12c2fe20cc3e45d29b7be37f900f68a9 [id=67ac4f4eda324e799b5867b2b011a16b])
I20260130 02:58:33.661868 21249 tablet_loader.cc:96] loaded metadata for tablet 955b9547f49f434699fe1d86b260638c (table default.Table_f246882197f44c69b72ca421cbdf201f [id=f778784dc88a4e4890113916098ad20f])
I20260130 02:58:33.665762 21249 tablet_loader.cc:96] loaded metadata for tablet 9896ff9364744857968e87e80e776dd2 (table default.Table_5d5e26f93a5a4fb084831add317d90f9 [id=f237b1c90a1743d597d87c5cf1abc89a])
I20260130 02:58:33.668278 21249 tablet_loader.cc:96] loaded metadata for tablet 9fe0448bcc4a4346b3c67fc3f626ed3c (table default.Table_5c1c447e85524ac5a61ffd8a48509d36 [id=70aaf21ad9bd4d959f46939c6b4a4880])
I20260130 02:58:33.669422 21249 tablet_loader.cc:96] loaded metadata for tablet a3258852aed24ed582a79f77860e6f0d (table default.Table_7c48a0d054354501947d11c5fd7b1864 [id=aa4ad1394ab344df8c54ba0031af5590])
I20260130 02:58:33.672895 21249 tablet_loader.cc:96] loaded metadata for tablet a90712651a604b83ba187f893ab17711 (table default.Table_0f3e44ae93bc4c028ed3929b94a49a24 [id=3b9c62fc8bbb44ffb7d66e9604974c30])
I20260130 02:58:33.674013 21249 tablet_loader.cc:96] loaded metadata for tablet a90e92d569254bbf8a5fbf286baf5cce (table default.Table_1e42d571631143088cf33d4c3aafed28 [id=30ba2af6dffd4f7e8dcbfdd2f3798157])
I20260130 02:58:33.675125 21249 tablet_loader.cc:96] loaded metadata for tablet aae68390dbbe4dc58ce3d3bd01725ae6 (table default.Table_f3e214a955f6447482099175b92ecb5c [id=76a688022fd9421a84de059b3c1da77a])
I20260130 02:58:33.682937 19519 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "920f52b975ab480fbf42656171222f09" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d870164263bb44b6b780a557daea3656"
I20260130 02:58:33.683480 19519 raft_consensus.cc:3060] T 920f52b975ab480fbf42656171222f09 P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Advancing to term 2
I20260130 02:58:33.720249 19224 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "920f52b975ab480fbf42656171222f09" candidate_uuid: "78c2c46c7520492db35d51c244fb4f74" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371a36d219164974abdf848841528152"
I20260130 02:58:33.720779 19224 raft_consensus.cc:3060] T 920f52b975ab480fbf42656171222f09 P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Advancing to term 2
I20260130 02:58:33.727739 19519 raft_consensus.cc:2468] T 920f52b975ab480fbf42656171222f09 P d870164263bb44b6b780a557daea3656 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78c2c46c7520492db35d51c244fb4f74 in term 2.
I20260130 02:58:33.665683 21508 raft_consensus.cc:515] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "658299197a424f53a37f14fadf089ae4" member_type: VOTER last_known_addr { host: "127.18.100.126" port: 38191 } } peers { permanent_uuid: "107f274cdda543038832f06a56ab9130" member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39711 } } peers { permanent_uuid: "3749baa287ed4f249329a74071781edd" member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34629 } }
I20260130 02:58:33.755003 18960 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "658299197a424f53a37f14fadf089ae4" candidate_term: 8 candidate_status { last_received { term: 6 index: 374 } } ignore_live_leader: false dest_uuid: "107f274cdda543038832f06a56ab9130"
I20260130 02:58:33.755178 20810 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "658299197a424f53a37f14fadf089ae4" candidate_term: 8 candidate_status { last_received { term: 6 index: 374 } } ignore_live_leader: false dest_uuid: "3749baa287ed4f249329a74071781edd"
I20260130 02:58:33.755515 18960 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 107f274cdda543038832f06a56ab9130 [term 6 FOLLOWER]: Advancing to term 8
I20260130 02:58:33.755666 20810 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 3749baa287ed4f249329a74071781edd [term 7 FOLLOWER]: Advancing to term 8
I20260130 02:58:33.755234 21508 leader_election.cc:290] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4 [CANDIDATE]: Term 8 election: Requested vote from peers 107f274cdda543038832f06a56ab9130 (127.18.100.125:39711), 3749baa287ed4f249329a74071781edd (127.18.100.124:34629)
I20260130 02:58:33.761597 20810 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 3749baa287ed4f249329a74071781edd [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 658299197a424f53a37f14fadf089ae4 in term 8.
I20260130 02:58:33.762750 18860 leader_election.cc:304] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3749baa287ed4f249329a74071781edd, 658299197a424f53a37f14fadf089ae4; no voters:
I20260130 02:58:33.764415 18960 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 107f274cdda543038832f06a56ab9130 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 658299197a424f53a37f14fadf089ae4 in term 8.
I20260130 02:58:33.772706 19313 leader_election.cc:304] T 920f52b975ab480fbf42656171222f09 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78c2c46c7520492db35d51c244fb4f74, d870164263bb44b6b780a557daea3656; no voters:
I20260130 02:58:33.721175 21510 leader_election.cc:290] T 920f52b975ab480fbf42656171222f09 P 78c2c46c7520492db35d51c244fb4f74 [CANDIDATE]: Term 2 election: Requested vote from peers d870164263bb44b6b780a557daea3656 (127.18.100.67:39571), 371a36d219164974abdf848841528152 (127.18.100.65:42021)
I20260130 02:58:33.777571 21470 raft_consensus.cc:2804] T 920f52b975ab480fbf42656171222f09 P 78c2c46c7520492db35d51c244fb4f74 [term 2 FOLLOWER]: Leader election won for term 2
I20260130 02:58:33.778031 21470 raft_consensus.cc:697] T 920f52b975ab480fbf42656171222f09 P 78c2c46c7520492db35d51c244fb4f74 [term 2 LEADER]: Becoming Leader. State: Replica: 78c2c46c7520492db35d51c244fb4f74, State: Running, Role: LEADER
I20260130 02:58:33.778883 21470 consensus_queue.cc:237] T 920f52b975ab480fbf42656171222f09 P 78c2c46c7520492db35d51c244fb4f74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } }
I20260130 02:58:33.794305 21508 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4 [term 8 FOLLOWER]: Leader election won for term 8
I20260130 02:58:33.805824 19224 raft_consensus.cc:2468] T 920f52b975ab480fbf42656171222f09 P 371a36d219164974abdf848841528152 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78c2c46c7520492db35d51c244fb4f74 in term 2.
I20260130 02:58:33.850459 21249 tablet_loader.cc:96] loaded metadata for tablet aedf72892b7248e2ace8772283c2b476 (table default.Table_203b3a960d3e4b5cb978b6168435f3a4 [id=ffe264c47e31462a8deaf81a8b96d34c])
W20260130 02:58:33.850737 19463 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.100.65:42021, user_credentials={real_user=slave}} blocked reactor thread for 75394.1us
I20260130 02:58:33.851764 21249 tablet_loader.cc:96] loaded metadata for tablet af0f90071a334016aa17fae2e3a97fc8 (table default.Table_22d10332a83f4700ac21892bab67f46b [id=da3a9cb1f6d5419abc1afc93659b5324])
I20260130 02:58:33.863762 21249 tablet_loader.cc:96] loaded metadata for tablet b1e8b78eff0d4a0b92059adb2de4a3ce (table default.Table_d0b4516d24384e5386c1debaf2543adb [id=f1342f9497b74e7a9ecad1de8082578d])
W20260130 02:58:33.875247 19462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.100.66:40767, user_credentials={real_user=slave}} blocked reactor thread for 129632us
I20260130 02:58:33.806396 21508 raft_consensus.cc:697] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4 [term 8 LEADER]: Becoming Leader. State: Replica: 658299197a424f53a37f14fadf089ae4, State: Running, Role: LEADER
I20260130 02:58:33.876734 21508 consensus_queue.cc:237] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 374, Committed index: 374, Last appended: 6.374, Last appended by leader: 374, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "658299197a424f53a37f14fadf089ae4" member_type: VOTER last_known_addr { host: "127.18.100.126" port: 38191 } } peers { permanent_uuid: "107f274cdda543038832f06a56ab9130" member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39711 } } peers { permanent_uuid: "3749baa287ed4f249329a74071781edd" member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34629 } }
I20260130 02:58:33.881351 21249 tablet_loader.cc:96] loaded metadata for tablet b89ed298e3dc49d789d166c613ad7fb8 (table default.Table_f8856f06ce97440bae4abdf34164716d [id=a3303ada20a94f99b2f37a02eec361ef])
I20260130 02:58:33.882503 21249 tablet_loader.cc:96] loaded metadata for tablet b8c579ff5fa84651ba9125457d927162 (table default.Table_3046889233c24f0a8e8058e1591d63b8 [id=edd6505c0bf3462cbf65c3df5649c16c])
I20260130 02:58:33.887796 21249 tablet_loader.cc:96] loaded metadata for tablet bac739bf4cac4272975f5587ad9c71ff (table default.Table_9e942a1ce3944e9aa028e75656ce066d [id=18ed268eb9ad4450abf9874310ace074])
I20260130 02:58:33.894982 21249 tablet_loader.cc:96] loaded metadata for tablet bc04be1284414a8699d3c4b88bf62813 (table default.Table_2414288f0dc54e5f835f876060927a9b [id=b0c6f2d9483c47da8ff3e65f0400a593])
W20260130 02:58:33.897773 18934 leader_election.cc:336] T 00000000000000000000000000000000 P 107f274cdda543038832f06a56ab9130 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 658299197a424f53a37f14fadf089ae4 (127.18.100.126:38191): Timed out: RequestConsensusVote RPC to 127.18.100.126:38191 timed out after 0.556s (SENT)
I20260130 02:58:33.898190 18934 leader_election.cc:304] T 00000000000000000000000000000000 P 107f274cdda543038832f06a56ab9130 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 107f274cdda543038832f06a56ab9130; no voters: 3749baa287ed4f249329a74071781edd, 658299197a424f53a37f14fadf089ae4
I20260130 02:58:33.927712 21249 tablet_loader.cc:96] loaded metadata for tablet c2c927e461f04fb48bfa9e3a685b077d (table default.Table_aa496bf2019748f29823a9a253b1f3ca [id=780188e2335f43fbb11de8212eb38d56])
I20260130 02:58:33.929652 21249 tablet_loader.cc:96] loaded metadata for tablet c76f0dc7762c49c389fa6fef41f02141 (table default.Table_e894e5560f194692bac730600c6bf5f4 [id=d5e1b2a45122429aabb4fd8653d71f0f])
I20260130 02:58:33.940658 21518 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 107f274cdda543038832f06a56ab9130 [term 8 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260130 02:58:33.955675 21249 tablet_loader.cc:96] loaded metadata for tablet ca06f07bb98f45a7bf63aebc28348e08 (table default.Table_829d1e187e8b4db5b8f4f835e84246e5 [id=a24b4761037d47a383f454a82517131f])
I20260130 02:58:33.956815 21249 tablet_loader.cc:96] loaded metadata for tablet cd30d7adcc4545848e3501a205a10293 (table default.Table_b85cb876f25740bfaaab01949ac26b38 [id=839e292e040c4aeb832f3fc0258c32ec])
W20260130 02:58:33.976264 20613 leader_election.cc:336] T 00000000000000000000000000000000 P 3749baa287ed4f249329a74071781edd [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer 658299197a424f53a37f14fadf089ae4 (127.18.100.126:38191): Timed out: RequestConsensusVote RPC to 127.18.100.126:38191 timed out after 0.513s (SENT)
I20260130 02:58:33.976697 20613 leader_election.cc:304] T 00000000000000000000000000000000 P 3749baa287ed4f249329a74071781edd [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3749baa287ed4f249329a74071781edd; no voters: 107f274cdda543038832f06a56ab9130, 658299197a424f53a37f14fadf089ae4
I20260130 02:58:33.984895 21249 tablet_loader.cc:96] loaded metadata for tablet cf1b66a20fce4c4d96bc0bc32a7bafa8 (table default.Table_d24e400f7451443a82d23140a90adc57 [id=e478df970dac40f7a576ce584c3abf2c])
I20260130 02:58:33.987888 21249 tablet_loader.cc:96] loaded metadata for tablet cfd12a72726846e587cff9d6a4d61bd6 (table default.Table_4ee03db7d0064d96a53d4f261bd12722 [id=07fe2931234d4888a524b52171b03f0a])
I20260130 02:58:34.007189 21514 sys_catalog.cc:455] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 658299197a424f53a37f14fadf089ae4. Latest consensus state: current_term: 8 leader_uuid: "658299197a424f53a37f14fadf089ae4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "658299197a424f53a37f14fadf089ae4" member_type: VOTER last_known_addr { host: "127.18.100.126" port: 38191 } } peers { permanent_uuid: "107f274cdda543038832f06a56ab9130" member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39711 } } peers { permanent_uuid: "3749baa287ed4f249329a74071781edd" member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34629 } } }
I20260130 02:58:34.007953 21514 sys_catalog.cc:458] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4 [sys.catalog]: This master's current role is: LEADER
I20260130 02:58:34.012354 21520 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 3749baa287ed4f249329a74071781edd [term 8 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260130 02:58:34.002871 21249 tablet_loader.cc:96] loaded metadata for tablet d3ba2df0b15c44f882070570f91b254d (table default.Table_92836e7fcf3a48d8bac2160c2ff7c482 [id=e3ae4b66fa0048668cb1e4d9da7d16fa])
I20260130 02:58:34.055951 21249 tablet_loader.cc:96] loaded metadata for tablet d480bf18140245e3870a62582d5a8318 (table default.Table_109c8b6c17d4442b939bde9441d0476f [id=dcf82ea15d6044c2b023f81f0a579ade])
I20260130 02:58:34.099278 21249 tablet_loader.cc:96] loaded metadata for tablet dc09a0f0cecf49738bab04596944c0f3 (table default.Table_6a7faee4c75c4eb7862fb4c05afc17cb [id=676ece9dc36b418a8b1400e8406d6b98])
I20260130 02:58:34.100348 21249 tablet_loader.cc:96] loaded metadata for tablet dc4bdd8e4f104d4a86c0af6ac6accda0 (table default.Table_ba19837ced4c45d79be506e93308449b [id=e732719fbe544e38a15b35017010c7b3])
I20260130 02:58:34.101454 21249 tablet_loader.cc:96] loaded metadata for tablet dcba6b534f9d406dad4fb502be40921c (table default.Table_baaf376733384fdbbac49ec8f2357432 [id=dea97f19a50141878f97820194afa59e])
I20260130 02:58:34.138624 21249 tablet_loader.cc:96] loaded metadata for tablet dd377fab8c544bf69121dc9e9a060b29 (table default.Table_5e70ab0e32bc4048bd19388f9d7cd302 [id=030c8ac23aac4a1fb7acc954d09434db])
I20260130 02:58:34.142252 21249 tablet_loader.cc:96] loaded metadata for tablet dfbbe9045e7c4aa5b520218687e7e4c1 (table default.Table_e03bdbb221e94bd6b535ddc02c863bad [id=2382eeebdc9540d5b588835705b2a4f1])
I20260130 02:58:34.143409 21249 tablet_loader.cc:96] loaded metadata for tablet e026dee22493424692d00f14848ab57c (table default.Table_85611716686c4097bacb64df677d5793 [id=122fe719b1d64ec8b531783539f85df1])
W20260130 02:58:34.163480 20613 leader_election.cc:336] T 00000000000000000000000000000000 P 3749baa287ed4f249329a74071781edd [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 658299197a424f53a37f14fadf089ae4 (127.18.100.126:38191): Timed out: RequestConsensusVote RPC to 127.18.100.126:38191 timed out after 0.708s (SENT)
I20260130 02:58:34.177084 21249 tablet_loader.cc:96] loaded metadata for tablet e34d5da0ad864b2eaea0d575c982e447 (table default.Table_aa4e5fd22945475da1dad48e1508595e [id=5b41834c6d8b4d89b773c2ea4c4ca7e6])
I20260130 02:58:34.180732 21249 tablet_loader.cc:96] loaded metadata for tablet e691c238199b4914bcc3fd248f31420c (table default.Table_7fe8c354c0dc4cfba0e84ed33e4c2506 [id=7523d671e9ac4814acff5060fd282eea])
I20260130 02:58:34.182428 21249 tablet_loader.cc:96] loaded metadata for tablet eb7a755ab62b4552bb8588264c5f7b75 (table default.Table_7e6ab9964f4c4e0f906d875a937a781d [id=bcf41af45d284bd89800977d14194812])
I20260130 02:58:34.216483 21249 tablet_loader.cc:96] loaded metadata for tablet ee7c8ff92ca9465fa18be6eae63d78ca (table default.Table_dd22e9dc63f84d8688c0071f427f0b25 [id=77979d498aa040828de415129cb15a06])
I20260130 02:58:34.217680 21249 tablet_loader.cc:96] loaded metadata for tablet f2d01bfdc9c4494386bd02bed655fe63 (table default.Table_aaf139c1a55d4a1783298ca12ae0d5c0 [id=d128767e303145a885eec3c5f714a27f])
I20260130 02:58:34.242914 21249 tablet_loader.cc:96] loaded metadata for tablet f41f4cff7dd4489f948700a45e9a7057 (table default.Table_ed6e2e20d89542199e5cbc7842df964d [id=047c39bd14664d1099e043b973766d32])
I20260130 02:58:34.263626 21249 tablet_loader.cc:96] loaded metadata for tablet f7607df5efd543438fb2dff3667cc8c8 (table default.Table_e8fd7516280f4dc4a73eb1f88fe916a2 [id=6efb836791414f3888f714aa8f65f0a4])
I20260130 02:58:34.264930 21249 tablet_loader.cc:96] loaded metadata for tablet f9e5f01e69254d658c369f3e21cef523 (table default.Table_12b505700b3d44e28353172e1786e5c8 [id=562776405d6f4c7aab0622f3866bf45b])
I20260130 02:58:34.266083 21249 tablet_loader.cc:96] loaded metadata for tablet ffa1142fc7274aecbdbee1a0802b95da (table default.Table_2f60fd0292ab40c8a3ae9b842cae8faf [id=e648e521ef5044a0b1967452cc3a31b2])
I20260130 02:58:34.446043 20810 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 3749baa287ed4f249329a74071781edd [term 8 FOLLOWER]: Refusing update from remote peer 658299197a424f53a37f14fadf089ae4: Log matching property violated. Preceding OpId in replica: term: 6 index: 374. Preceding OpId from leader: term: 8 index: 375. (index mismatch)
I20260130 02:58:34.447479 21508 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3749baa287ed4f249329a74071781edd" member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 375, Last known committed idx: 374, Time since last communication: 0.000s
I20260130 02:58:34.460212 18960 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 107f274cdda543038832f06a56ab9130 [term 8 FOLLOWER]: Refusing update from remote peer 658299197a424f53a37f14fadf089ae4: Log matching property violated. Preceding OpId in replica: term: 6 index: 374. Preceding OpId from leader: term: 8 index: 375. (index mismatch)
I20260130 02:58:34.461638 21508 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "107f274cdda543038832f06a56ab9130" member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39711 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 375, Last known committed idx: 374, Time since last communication: 0.000s
I20260130 02:58:34.504665 21523 sys_catalog.cc:455] T 00000000000000000000000000000000 P 107f274cdda543038832f06a56ab9130 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 658299197a424f53a37f14fadf089ae4. Latest consensus state: current_term: 8 leader_uuid: "658299197a424f53a37f14fadf089ae4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "658299197a424f53a37f14fadf089ae4" member_type: VOTER last_known_addr { host: "127.18.100.126" port: 38191 } } peers { permanent_uuid: "107f274cdda543038832f06a56ab9130" member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39711 } } peers { permanent_uuid: "3749baa287ed4f249329a74071781edd" member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34629 } } }
I20260130 02:58:34.505501 21523 sys_catalog.cc:458] T 00000000000000000000000000000000 P 107f274cdda543038832f06a56ab9130 [sys.catalog]: This master's current role is: FOLLOWER
I20260130 02:58:34.523676 21520 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3749baa287ed4f249329a74071781edd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 658299197a424f53a37f14fadf089ae4. Latest consensus state: current_term: 8 leader_uuid: "658299197a424f53a37f14fadf089ae4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "658299197a424f53a37f14fadf089ae4" member_type: VOTER last_known_addr { host: "127.18.100.126" port: 38191 } } peers { permanent_uuid: "107f274cdda543038832f06a56ab9130" member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39711 } } peers { permanent_uuid: "3749baa287ed4f249329a74071781edd" member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34629 } } }
I20260130 02:58:34.524446 21520 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3749baa287ed4f249329a74071781edd [sys.catalog]: This master's current role is: FOLLOWER
I20260130 02:58:34.527566 21514 sys_catalog.cc:455] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 8 leader_uuid: "658299197a424f53a37f14fadf089ae4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "658299197a424f53a37f14fadf089ae4" member_type: VOTER last_known_addr { host: "127.18.100.126" port: 38191 } } peers { permanent_uuid: "107f274cdda543038832f06a56ab9130" member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39711 } } peers { permanent_uuid: "3749baa287ed4f249329a74071781edd" member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34629 } } }
I20260130 02:58:34.528280 21514 sys_catalog.cc:458] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4 [sys.catalog]: This master's current role is: LEADER
I20260130 02:58:34.550084 21508 sys_catalog.cc:455] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 8 leader_uuid: "658299197a424f53a37f14fadf089ae4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "658299197a424f53a37f14fadf089ae4" member_type: VOTER last_known_addr { host: "127.18.100.126" port: 38191 } } peers { permanent_uuid: "107f274cdda543038832f06a56ab9130" member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39711 } } peers { permanent_uuid: "3749baa287ed4f249329a74071781edd" member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34629 } } }
I20260130 02:58:34.550858 21508 sys_catalog.cc:458] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4 [sys.catalog]: This master's current role is: LEADER
I20260130 02:58:34.595373 21523 sys_catalog.cc:455] T 00000000000000000000000000000000 P 107f274cdda543038832f06a56ab9130 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 8 leader_uuid: "658299197a424f53a37f14fadf089ae4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "658299197a424f53a37f14fadf089ae4" member_type: VOTER last_known_addr { host: "127.18.100.126" port: 38191 } } peers { permanent_uuid: "107f274cdda543038832f06a56ab9130" member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39711 } } peers { permanent_uuid: "3749baa287ed4f249329a74071781edd" member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34629 } } }
I20260130 02:58:34.596043 21523 sys_catalog.cc:458] T 00000000000000000000000000000000 P 107f274cdda543038832f06a56ab9130 [sys.catalog]: This master's current role is: FOLLOWER
I20260130 02:58:34.687799 21520 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3749baa287ed4f249329a74071781edd [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 8 leader_uuid: "658299197a424f53a37f14fadf089ae4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "658299197a424f53a37f14fadf089ae4" member_type: VOTER last_known_addr { host: "127.18.100.126" port: 38191 } } peers { permanent_uuid: "107f274cdda543038832f06a56ab9130" member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39711 } } peers { permanent_uuid: "3749baa287ed4f249329a74071781edd" member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34629 } } }
I20260130 02:58:34.688484 21520 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3749baa287ed4f249329a74071781edd [sys.catalog]: This master's current role is: FOLLOWER
I20260130 02:58:34.712720 18876 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "107f274cdda543038832f06a56ab9130" candidate_term: 7 candidate_status { last_received { term: 6 index: 374 } } ignore_live_leader: false dest_uuid: "658299197a424f53a37f14fadf089ae4" is_pre_election: true
W20260130 02:58:34.713546 18876 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58704 (request call id 23) took 2001 ms (client timeout 556 ms). Trace:
W20260130 02:58:34.713912 18876 rpcz_store.cc:269] 0130 02:58:32.711878 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:58:32.712096 (+ 218us) service_pool.cc:225] Handling call
0130 02:58:34.713463 (+2001367us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260130 02:58:34.714955 18876 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35042 (request call id 10) took 1770 ms (client timeout 513 ms). Trace:
W20260130 02:58:34.715267 18876 rpcz_store.cc:269] 0130 02:58:32.944839 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:58:34.714777 (+1769938us) service_pool.cc:212] Skipping call since client already timed out
0130 02:58:34.714897 (+ 120us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20260130 02:58:34.715993 18876 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58704 (request call id 24) took 1376 ms (client timeout 556 ms). Trace:
W20260130 02:58:34.716303 18876 rpcz_store.cc:269] 0130 02:58:33.339663 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:58:34.715852 (+1376189us) service_pool.cc:212] Skipping call since client already timed out
0130 02:58:34.715944 (+ 92us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20260130 02:58:34.717020 18876 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35042 (request call id 12) took 1254 ms (client timeout 513 ms). Trace:
W20260130 02:58:34.717337 18876 rpcz_store.cc:269] 0130 02:58:33.462630 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:58:34.716879 (+1254249us) service_pool.cc:212] Skipping call since client already timed out
0130 02:58:34.716973 (+ 94us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20260130 02:58:34.718026 18876 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35042 (request call id 11) took 1260 ms (client timeout 708 ms). Trace:
I20260130 02:58:34.710973 18877 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3749baa287ed4f249329a74071781edd" candidate_term: 7 candidate_status { last_received { term: 6 index: 374 } } ignore_live_leader: false dest_uuid: "658299197a424f53a37f14fadf089ae4" is_pre_election: true
W20260130 02:58:34.718336 18876 rpcz_store.cc:269] 0130 02:58:33.457212 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:58:34.717886 (+1260674us) service_pool.cc:212] Skipping call since client already timed out
0130 02:58:34.717983 (+ 97us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20260130 02:58:34.727499 18884 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3749baa287ed4f249329a74071781edd" candidate_term: 7 candidate_status { last_received { term: 6 index: 374 } } ignore_live_leader: false dest_uuid: "658299197a424f53a37f14fadf089ae4" is_pre_election: true
W20260130 02:58:34.729184 18884 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35042 (request call id 5) took 7204 ms (client timeout 530 ms). Trace:
W20260130 02:58:34.729521 18884 rpcz_store.cc:269] 0130 02:58:27.524971 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:58:27.525126 (+ 155us) service_pool.cc:225] Handling call
0130 02:58:34.729109 (+7203983us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260130 02:58:34.730432 18881 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "107f274cdda543038832f06a56ab9130" candidate_term: 7 candidate_status { last_received { term: 6 index: 374 } } ignore_live_leader: false dest_uuid: "658299197a424f53a37f14fadf089ae4" is_pre_election: true
W20260130 02:58:34.736819 18881 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58704 (request call id 21) took 5458 ms (client timeout 707 ms). Trace:
W20260130 02:58:34.737172 18881 rpcz_store.cc:269] 0130 02:58:29.278692 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:58:29.279224 (+ 532us) service_pool.cc:225] Handling call
0130 02:58:34.736733 (+5457509us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260130 02:58:34.726927 21249 catalog_manager.cc:1486] Time spent T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4: Loading table and tablet metadata into memory: real 7.957s user 0.394s sys 0.066s
I20260130 02:58:34.740299 18885 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "107f274cdda543038832f06a56ab9130" candidate_term: 7 candidate_status { last_received { term: 6 index: 374 } } ignore_live_leader: false dest_uuid: "658299197a424f53a37f14fadf089ae4" is_pre_election: true
I20260130 02:58:34.738560 21249 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260130 02:58:34.738883 18882 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3749baa287ed4f249329a74071781edd" candidate_term: 7 candidate_status { last_received { term: 6 index: 374 } } ignore_live_leader: false dest_uuid: "658299197a424f53a37f14fadf089ae4" is_pre_election: true
I20260130 02:58:34.741142 18879 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "107f274cdda543038832f06a56ab9130" candidate_term: 7 candidate_status { last_received { term: 6 index: 374 } } ignore_live_leader: false dest_uuid: "658299197a424f53a37f14fadf089ae4" is_pre_election: true
W20260130 02:58:34.742798 18877 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35042 (request call id 9) took 2035 ms (client timeout 708 ms). Trace:
W20260130 02:58:34.742908 18882 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35042 (request call id 6) took 6671 ms (client timeout 530 ms). Trace:
W20260130 02:58:34.743144 18877 rpcz_store.cc:269] 0130 02:58:32.707544 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:58:32.707848 (+ 304us) service_pool.cc:225] Handling call
0130 02:58:34.742732 (+2034884us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260130 02:58:34.743234 18882 rpcz_store.cc:269] 0130 02:58:28.071780 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:58:28.071924 (+ 144us) service_pool.cc:225] Handling call
0130 02:58:34.742834 (+6670910us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260130 02:58:34.744244 18868 rpcz_store.cc:275] Call kudu.master.MasterService.ConnectToMaster from 127.0.0.1:50284 (request call id 279) took 7943 ms. Trace:
I20260130 02:58:34.738943 18880 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3749baa287ed4f249329a74071781edd" candidate_term: 7 candidate_status { last_received { term: 6 index: 374 } } ignore_live_leader: false dest_uuid: "658299197a424f53a37f14fadf089ae4" is_pre_election: true
W20260130 02:58:34.742352 18885 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58704 (request call id 19) took 7681 ms (client timeout 577 ms). Trace:
I20260130 02:58:34.744585 18868 rpcz_store.cc:276] 0130 02:58:26.800662 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:58:26.800803 (+ 141us) service_pool.cc:225] Handling call
0130 02:58:34.744031 (+7943228us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260130 02:58:34.748682 18880 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35042 (request call id 7) took 5037 ms (client timeout 557 ms). Trace:
W20260130 02:58:34.749274 18880 rpcz_store.cc:269] 0130 02:58:29.711366 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:58:29.711524 (+ 158us) service_pool.cc:225] Handling call
0130 02:58:34.748609 (+5037085us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260130 02:58:34.738449 18883 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "107f274cdda543038832f06a56ab9130" candidate_term: 7 candidate_status { last_received { term: 6 index: 374 } } ignore_live_leader: false dest_uuid: "658299197a424f53a37f14fadf089ae4" is_pre_election: true
W20260130 02:58:34.750475 18883 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58704 (request call id 20) took 7022 ms (client timeout 577 ms). Trace:
I20260130 02:58:34.727438 18878 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3749baa287ed4f249329a74071781edd" candidate_term: 7 candidate_status { last_received { term: 6 index: 374 } } ignore_live_leader: false dest_uuid: "658299197a424f53a37f14fadf089ae4" is_pre_election: true
W20260130 02:58:34.748193 18885 rpcz_store.cc:269] 0130 02:58:27.060742 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:58:27.060877 (+ 135us) service_pool.cc:225] Handling call
0130 02:58:34.741415 (+7680538us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260130 02:58:34.752611 18878 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35042 (request call id 8) took 4434 ms (client timeout 557 ms). Trace:
W20260130 02:58:34.753007 18878 rpcz_store.cc:269] 0130 02:58:30.318161 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:58:30.318315 (+ 154us) service_pool.cc:225] Handling call
0130 02:58:34.752545 (+4434230us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260130 02:58:34.761516 18879 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58704 (request call id 22) took 4733 ms (client timeout 707 ms). Trace:
W20260130 02:58:34.750850 18883 rpcz_store.cc:269] 0130 02:58:27.728499 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:58:27.728672 (+ 173us) service_pool.cc:225] Handling call
0130 02:58:34.750409 (+7021737us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260130 02:58:34.761895 18879 rpcz_store.cc:269] 0130 02:58:30.027649 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:58:30.027793 (+ 144us) service_pool.cc:225] Handling call
0130 02:58:34.761445 (+4733652us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260130 02:58:34.887356 21249 catalog_manager.cc:1269] Loaded cluster ID: db1c0e7709154c10a8c9a0ea68ffe38b
I20260130 02:58:34.887645 21249 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260130 02:58:34.957283 19224 raft_consensus.cc:1275] T 920f52b975ab480fbf42656171222f09 P 371a36d219164974abdf848841528152 [term 2 FOLLOWER]: Refusing update from remote peer 78c2c46c7520492db35d51c244fb4f74: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20260130 02:58:34.958829 21470 consensus_queue.cc:1048] T 920f52b975ab480fbf42656171222f09 P 78c2c46c7520492db35d51c244fb4f74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260130 02:58:34.987089 21249 catalog_manager.cc:1514] Loading token signing keys...
I20260130 02:58:35.011651 21523 raft_consensus.cc:493] T 00000000000000000000000000000000 P 107f274cdda543038832f06a56ab9130 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 658299197a424f53a37f14fadf089ae4)
I20260130 02:58:35.012126 21523 raft_consensus.cc:515] T 00000000000000000000000000000000 P 107f274cdda543038832f06a56ab9130 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "658299197a424f53a37f14fadf089ae4" member_type: VOTER last_known_addr { host: "127.18.100.126" port: 38191 } } peers { permanent_uuid: "107f274cdda543038832f06a56ab9130" member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39711 } } peers { permanent_uuid: "3749baa287ed4f249329a74071781edd" member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34629 } }
I20260130 02:58:35.013913 21523 leader_election.cc:290] T 00000000000000000000000000000000 P 107f274cdda543038832f06a56ab9130 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 658299197a424f53a37f14fadf089ae4 (127.18.100.126:38191), 3749baa287ed4f249329a74071781edd (127.18.100.124:34629)
I20260130 02:58:35.016777 18877 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "107f274cdda543038832f06a56ab9130" candidate_term: 9 candidate_status { last_received { term: 8 index: 375 } } ignore_live_leader: false dest_uuid: "658299197a424f53a37f14fadf089ae4" is_pre_election: true
I20260130 02:58:35.022200 19523 raft_consensus.cc:1275] T 920f52b975ab480fbf42656171222f09 P d870164263bb44b6b780a557daea3656 [term 2 FOLLOWER]: Refusing update from remote peer 78c2c46c7520492db35d51c244fb4f74: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20260130 02:58:35.024204 20810 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "107f274cdda543038832f06a56ab9130" candidate_term: 9 candidate_status { last_received { term: 8 index: 375 } } ignore_live_leader: false dest_uuid: "3749baa287ed4f249329a74071781edd" is_pre_election: true
I20260130 02:58:35.036008 18935 leader_election.cc:304] T 00000000000000000000000000000000 P 107f274cdda543038832f06a56ab9130 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 107f274cdda543038832f06a56ab9130; no voters: 3749baa287ed4f249329a74071781edd, 658299197a424f53a37f14fadf089ae4
I20260130 02:58:35.041131 21517 consensus_queue.cc:1048] T 920f52b975ab480fbf42656171222f09 P 78c2c46c7520492db35d51c244fb4f74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260130 02:58:35.056661 21523 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 107f274cdda543038832f06a56ab9130 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260130 02:58:35.350890 21249 catalog_manager.cc:6038] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4: Loaded TSK: 0
I20260130 02:58:35.351938 21249 catalog_manager.cc:1524] Initializing in-progress tserver states...
W20260130 02:58:35.357563 19462 consensus_peers.cc:597] T 2c79a3aecab844698e70834999e1caac P d870164263bb44b6b780a557daea3656 -> Peer 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767): Couldn't send request to peer 78c2c46c7520492db35d51c244fb4f74. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2c79a3aecab844698e70834999e1caac. This is attempt 6: this message will repeat every 5th retry.
I20260130 02:58:35.490221 21249 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260130 02:58:35.494961 21539 raft_consensus.cc:493] T 00000000000000000000000000000000 P 3749baa287ed4f249329a74071781edd [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 658299197a424f53a37f14fadf089ae4)
I20260130 02:58:35.496318 21539 raft_consensus.cc:515] T 00000000000000000000000000000000 P 3749baa287ed4f249329a74071781edd [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "658299197a424f53a37f14fadf089ae4" member_type: VOTER last_known_addr { host: "127.18.100.126" port: 38191 } } peers { permanent_uuid: "107f274cdda543038832f06a56ab9130" member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39711 } } peers { permanent_uuid: "3749baa287ed4f249329a74071781edd" member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34629 } }
I20260130 02:58:35.499953 18960 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3749baa287ed4f249329a74071781edd" candidate_term: 9 candidate_status { last_received { term: 8 index: 375 } } ignore_live_leader: false dest_uuid: "107f274cdda543038832f06a56ab9130" is_pre_election: true
I20260130 02:58:35.510689 21539 leader_election.cc:290] T 00000000000000000000000000000000 P 3749baa287ed4f249329a74071781edd [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 658299197a424f53a37f14fadf089ae4 (127.18.100.126:38191), 107f274cdda543038832f06a56ab9130 (127.18.100.125:39711)
I20260130 02:58:35.632503 21249 catalog_manager.cc:679] Loaded metadata for table default.Table_5e70ab0e32bc4048bd19388f9d7cd302 [id=030c8ac23aac4a1fb7acc954d09434db]
I20260130 02:58:35.633894 21249 catalog_manager.cc:679] Loaded metadata for table default.Table_dabf999abe834151a0a5b6747a05cac3 [id=038ba9aa798741f6858fa09002d6c3d6]
I20260130 02:58:35.635035 21249 catalog_manager.cc:679] Loaded metadata for table default.Table_ed6e2e20d89542199e5cbc7842df964d [id=047c39bd14664d1099e043b973766d32]
I20260130 02:58:35.636129 21249 catalog_manager.cc:679] Loaded metadata for table default.Table_fec49c37fdd045e7801325ace326f367 [id=06f1da4d6e0e46e4a141914a3aa68c07]
I20260130 02:58:35.637215 21249 catalog_manager.cc:679] Loaded metadata for table default.Table_4ee03db7d0064d96a53d4f261bd12722 [id=07fe2931234d4888a524b52171b03f0a]
I20260130 02:58:35.638298 21249 catalog_manager.cc:679] Loaded metadata for table default.Table_0480357296ac4a27812d802ee4566351 [id=0da6197ab32e4645aa8d8787662ee5e6]
I20260130 02:58:35.639393 21249 catalog_manager.cc:679] Loaded metadata for table default.Table_85611716686c4097bacb64df677d5793 [id=122fe719b1d64ec8b531783539f85df1]
I20260130 02:58:35.640514 21249 catalog_manager.cc:679] Loaded metadata for table default.Table_805e07e3546e447d8c8b8029477013cc [id=1515db389f284e2f9ed1a3cd0061d0f1]
I20260130 02:58:35.641608 21249 catalog_manager.cc:679] Loaded metadata for table default.Table_3a858a3cb886427cb8268f42c953ad4c [id=17150cb73e374ff9a1e92c43e0c3f2c1]
I20260130 02:58:35.654779 21249 catalog_manager.cc:679] Loaded metadata for table default.Table_ea2e0aa925a0486bb292304ca21bb819 [id=18580b9c54e448d1bdaae84f697f7b0b]
I20260130 02:58:35.656719 21249 catalog_manager.cc:679] Loaded metadata for table default.Table_9e942a1ce3944e9aa028e75656ce066d [id=18ed268eb9ad4450abf9874310ace074]
I20260130 02:58:35.659134 21249 catalog_manager.cc:679] Loaded metadata for table default.Table_a132cf962cfc4ed8a6a8c177e088daeb [id=1c4937ca6e014fec8596a3dc25779600]
I20260130 02:58:35.660214 21249 catalog_manager.cc:679] Loaded metadata for table default.Table_ddfb5df90fdb40a087ac434d088bbf0e [id=1e72b7dd669d4539b61884ba63ff7183]
I20260130 02:58:35.663224 21249 catalog_manager.cc:679] Loaded metadata for table default.Table_e03bdbb221e94bd6b535ddc02c863bad [id=2382eeebdc9540d5b588835705b2a4f1]
I20260130 02:58:35.665611 21249 catalog_manager.cc:679] Loaded metadata for table default.Table_ebc8b8379c274c18b0a06a919dbe0ee0 [id=28156e0843674fce88f5914a3e4195ae]
I20260130 02:58:35.678833 21249 catalog_manager.cc:679] Loaded metadata for table default.Table_bc168ff8b5f34cacbd398117647ba25a [id=2a21ee52b7d246f1b44eda02bf5a411e]
I20260130 02:58:35.680141 21249 catalog_manager.cc:679] Loaded metadata for table default.Table_f619de75da8e4d23b4d641d30285e0e7 [id=2dc3a57f750e4104b99f4589fdb29e3c]
I20260130 02:58:35.681296 21249 catalog_manager.cc:679] Loaded metadata for table default.Table_fa63c9eff99d4ac1b185a4c4ce2df031 [id=2e8b378422c74abc939955d110737316]
I20260130 02:58:35.682392 21249 catalog_manager.cc:679] Loaded metadata for table default.Table_1e42d571631143088cf33d4c3aafed28 [id=30ba2af6dffd4f7e8dcbfdd2f3798157]
I20260130 02:58:35.683488 21249 catalog_manager.cc:679] Loaded metadata for table default.Table_cda216a3bfc14ec0862b998bedd5b300 [id=35978c9d901c47eb81e565e37493b1ae]
I20260130 02:58:35.684561 21249 catalog_manager.cc:679] Loaded metadata for table default.Table_19677b801d1f4a63b22139b9fc80c4ef [id=3992cda3a7bb49ae86b8f84ee12f1c8b]
I20260130 02:58:35.685622 21249 catalog_manager.cc:679] Loaded metadata for table default.Table_0f3e44ae93bc4c028ed3929b94a49a24 [id=3b9c62fc8bbb44ffb7d66e9604974c30]
I20260130 02:58:35.686755 21249 catalog_manager.cc:679] Loaded metadata for table default.Table_8f632ff84d624b72a10cb90fb9e78636 [id=3cf79022ea954f6397f0a0497e6e6a6c]
I20260130 02:58:35.687829 21249 catalog_manager.cc:679] Loaded metadata for table default.Table_d764cbaecfee476ba5e734dd1e2f50fd [id=3f9236f0631a4e4c94ca7456f0478a45]
I20260130 02:58:35.688879 21249 catalog_manager.cc:679] Loaded metadata for table default.Table_0ca1d7370d2f416782b386c5b47111ab [id=3ffe1a273d4f452c835278a8e7f10102]
I20260130 02:58:35.689942 21249 catalog_manager.cc:679] Loaded metadata for table default.Table_f17d8e4d9f25449b84aa751af93936b2 [id=43254e466e7b4a2fb59fbcec68cfde36]
I20260130 02:58:35.701781 21249 catalog_manager.cc:679] Loaded metadata for table default.Table_6f53c17cfd144031b2d4b112fb2d88eb [id=443cbad5510d463fa8ca878b8628dbda]
I20260130 02:58:35.704130 21249 catalog_manager.cc:679] Loaded metadata for table default.Table_8896365e792f4d24b2c16c2d269b0a93 [id=44816e07033849479a7fea0f51d00fad]
I20260130 02:58:35.705983 21249 catalog_manager.cc:679] Loaded metadata for table default.Table_2d345d0f0957486c86c2cf2e34b00941 [id=4c3e0d3ecd5642a3b7fc7d20e2521418]
I20260130 02:58:35.708388 21249 catalog_manager.cc:679] Loaded metadata for table default.Table_8f311c5878744ef692b496ecfdfc3ede [id=4f499357303d464aafc17679ffcaaa07]
I20260130 02:58:35.711447 21249 catalog_manager.cc:679] Loaded metadata for table default.Table_d2a3438ed66741eab245ae6d6c5c2f37 [id=4fb5c26265e94083a8714be512fcd222]
I20260130 02:58:35.713186 21249 catalog_manager.cc:679] Loaded metadata for table default.Table_21178f657dca43a1912ce596834d1618 [id=53b79067a6034189aac5f1554df542c4]
I20260130 02:58:35.727085 21249 catalog_manager.cc:679] Loaded metadata for table default.Table_12b505700b3d44e28353172e1786e5c8 [id=562776405d6f4c7aab0622f3866bf45b]
I20260130 02:58:35.729081 21249 catalog_manager.cc:679] Loaded metadata for table default.Table_aa4e5fd22945475da1dad48e1508595e [id=5b41834c6d8b4d89b773c2ea4c4ca7e6]
I20260130 02:58:35.730973 21249 catalog_manager.cc:679] Loaded metadata for table default.Table_9cb9a9f9e5d145a6a42d7ccbc42aced9 [id=5f1f179fadc247da9382dd7796dcf2c2]
I20260130 02:58:35.732717 21249 catalog_manager.cc:679] Loaded metadata for table default.Table_9d0cd9756483406081c8bdb18565a42f [id=5f6488df8ba94ef293b181328cc3b680]
I20260130 02:58:35.734474 21249 catalog_manager.cc:679] Loaded metadata for table default.Table_d95381c590c94802bdc83acef0f0df96 [id=64a8e0e079f34806aa68795f3d548c5e]
I20260130 02:58:35.735622 21249 catalog_manager.cc:679] Loaded metadata for table default.Table_dee034aec0b24982b8dc2a3fa1b326f1 [id=66033e75b3814a8584311cf7e4a07ad3]
I20260130 02:58:35.737326 21249 catalog_manager.cc:679] Loaded metadata for table default.Table_6a7faee4c75c4eb7862fb4c05afc17cb [id=676ece9dc36b418a8b1400e8406d6b98]
I20260130 02:58:35.738420 21249 catalog_manager.cc:679] Loaded metadata for table default.Table_12c2fe20cc3e45d29b7be37f900f68a9 [id=67ac4f4eda324e799b5867b2b011a16b]
I20260130 02:58:35.739547 21249 catalog_manager.cc:679] Loaded metadata for table default.Table_6c46e6995d4741dcaaa6ed7bdc765e29 [id=6d91a161cf484239967af58db160c952]
I20260130 02:58:35.740615 21249 catalog_manager.cc:679] Loaded metadata for table default.Table_e8fd7516280f4dc4a73eb1f88fe916a2 [id=6efb836791414f3888f714aa8f65f0a4]
I20260130 02:58:35.741768 21249 catalog_manager.cc:679] Loaded metadata for table default.Table_5c1c447e85524ac5a61ffd8a48509d36 [id=70aaf21ad9bd4d959f46939c6b4a4880]
I20260130 02:58:35.761682 21249 catalog_manager.cc:679] Loaded metadata for table default.Table_7fe8c354c0dc4cfba0e84ed33e4c2506 [id=7523d671e9ac4814acff5060fd282eea]
I20260130 02:58:35.762840 21249 catalog_manager.cc:679] Loaded metadata for table default.Table_f3e214a955f6447482099175b92ecb5c [id=76a688022fd9421a84de059b3c1da77a]
I20260130 02:58:35.763962 21249 catalog_manager.cc:679] Loaded metadata for table default.Table_dd22e9dc63f84d8688c0071f427f0b25 [id=77979d498aa040828de415129cb15a06]
I20260130 02:58:35.765072 21249 catalog_manager.cc:679] Loaded metadata for table default.Table_007e969f70d345a7bcb3dfbaabbc025d [id=77d4fa8705a3419cb0a84d3ca1bb4ce3]
I20260130 02:58:35.766189 21249 catalog_manager.cc:679] Loaded metadata for table default.Table_aa496bf2019748f29823a9a253b1f3ca [id=780188e2335f43fbb11de8212eb38d56]
I20260130 02:58:35.768673 21249 catalog_manager.cc:679] Loaded metadata for table default.Table_c1886dd9a7934e7797e0e7e0b065d692 [id=7a2d5367224b492b844199088b01b149]
I20260130 02:58:35.783219 21249 catalog_manager.cc:679] Loaded metadata for table default.Table_69267d31df644f4396a44130fa3d499d [id=7efa53800a974e31a38a4c60c0a1b466]
I20260130 02:58:35.784451 21249 catalog_manager.cc:679] Loaded metadata for table default.Table_b85cb876f25740bfaaab01949ac26b38 [id=839e292e040c4aeb832f3fc0258c32ec]
I20260130 02:58:35.785588 21249 catalog_manager.cc:679] Loaded metadata for table default.Table_1ae68ae3f0404843aca3c72f02678ac6 [id=8a9182f49e18489fafec8dca190c1b2f]
I20260130 02:58:35.786725 21249 catalog_manager.cc:679] Loaded metadata for table default.Table_333d2efbf068485a871f566d50e563ce [id=8b7db40afb894ae99625a756c75ef1de]
I20260130 02:58:35.787813 21249 catalog_manager.cc:679] Loaded metadata for table default.Table_1703c69112744c0e951c1ac9667bbac6 [id=8d76f4328edc491291491c78ba205d30]
I20260130 02:58:35.788870 21249 catalog_manager.cc:679] Loaded metadata for table default.Table_6773a0ad5d6a429086a5818d731f3f4a [id=90b72499c7174fd580914d7fe3714891]
I20260130 02:58:35.803452 21249 catalog_manager.cc:679] Loaded metadata for table default.Table_829d1e187e8b4db5b8f4f835e84246e5 [id=a24b4761037d47a383f454a82517131f]
I20260130 02:58:35.804755 21249 catalog_manager.cc:679] Loaded metadata for table default.Table_f8856f06ce97440bae4abdf34164716d [id=a3303ada20a94f99b2f37a02eec361ef]
I20260130 02:58:35.805894 21249 catalog_manager.cc:679] Loaded metadata for table default.Table_2e81f0a0cbc449a397141b717db17025 [id=a9c919f57c0c437cb35a6921103e45a6]
I20260130 02:58:35.807063 21249 catalog_manager.cc:679] Loaded metadata for table default.Table_7c48a0d054354501947d11c5fd7b1864 [id=aa4ad1394ab344df8c54ba0031af5590]
I20260130 02:58:35.808791 21249 catalog_manager.cc:679] Loaded metadata for table default.Table_477fb92965284222aa1ecaa0107d1ec2 [id=ad82881a19e748edb569e74d692bb2bc]
I20260130 02:58:35.811182 21249 catalog_manager.cc:679] Loaded metadata for table default.Table_2414288f0dc54e5f835f876060927a9b [id=b0c6f2d9483c47da8ff3e65f0400a593]
I20260130 02:58:35.812268 21249 catalog_manager.cc:679] Loaded metadata for table default.Table_526677c8ef4144bd9de7a54a5e78e2be [id=b16908b75587459597a978c71756451c]
I20260130 02:58:35.813340 21249 catalog_manager.cc:679] Loaded metadata for table default.Table_78b080efddd94878b9d8b77b5ef3af35 [id=b654822c58224a859ad2dcdd8d0123c2]
I20260130 02:58:35.823004 21249 catalog_manager.cc:679] Loaded metadata for table default.Table_7d1c27ab3515406ba5b5333c90df093e [id=b6fabeeae79941dc8094d0f2b51704f9]
I20260130 02:58:35.824186 21249 catalog_manager.cc:679] Loaded metadata for table default.Table_95f0bff8e17d4ca995c0af0d0c24e84d [id=b7795905d0c440579c59d6ac00473f5a]
I20260130 02:58:35.826643 21249 catalog_manager.cc:679] Loaded metadata for table default.Table_da3d581b15a94840898777f74611994e [id=b7e241ced53e4160ac6fa1f65ef3674e]
I20260130 02:58:35.829145 21249 catalog_manager.cc:679] Loaded metadata for table default.Table_7e6ab9964f4c4e0f906d875a937a781d [id=bcf41af45d284bd89800977d14194812]
I20260130 02:58:35.830204 21249 catalog_manager.cc:679] Loaded metadata for table default.Table_d3f82845a18f4a4a830460cb1cbbb2de [id=c3d0c57ad165460eae9c20c32557fd2e]
I20260130 02:58:35.839452 21249 catalog_manager.cc:679] Loaded metadata for table default.Table_c4f4904b6fac4b9880979cf0314e71c1 [id=c456c6418c93499a9d08f652720854ca]
I20260130 02:58:35.841374 21249 catalog_manager.cc:679] Loaded metadata for table default.Table_81553b46d7804fd99ef36f431793b001 [id=cdfc1cfcc48f41f8a7a13f865174c3d8]
I20260130 02:58:35.843439 21249 catalog_manager.cc:679] Loaded metadata for table default.Table_aaf139c1a55d4a1783298ca12ae0d5c0 [id=d128767e303145a885eec3c5f714a27f]
I20260130 02:58:35.844606 21249 catalog_manager.cc:679] Loaded metadata for table default.Table_94b23422c82d419d97dc097a3efb92fc [id=d2f35437eae140dab4150079eae46377]
I20260130 02:58:35.846299 21249 catalog_manager.cc:679] Loaded metadata for table default.Table_e894e5560f194692bac730600c6bf5f4 [id=d5e1b2a45122429aabb4fd8653d71f0f]
I20260130 02:58:35.847411 21249 catalog_manager.cc:679] Loaded metadata for table default.Table_22d10332a83f4700ac21892bab67f46b [id=da3a9cb1f6d5419abc1afc93659b5324]
I20260130 02:58:35.848511 21249 catalog_manager.cc:679] Loaded metadata for table default.Table_9f83f87711d94e8394abe1925ac84ad1 [id=db23c1be42c44b6f9eb774f14d2b8fd6]
I20260130 02:58:35.849577 21249 catalog_manager.cc:679] Loaded metadata for table default.Table_109c8b6c17d4442b939bde9441d0476f [id=dcf82ea15d6044c2b023f81f0a579ade]
I20260130 02:58:35.885376 21249 catalog_manager.cc:679] Loaded metadata for table default.Table_baaf376733384fdbbac49ec8f2357432 [id=dea97f19a50141878f97820194afa59e]
I20260130 02:58:35.887311 21249 catalog_manager.cc:679] Loaded metadata for table default.Table_8c27a7d10b7847e78029f7e95321e9b6 [id=e31c1bf0c3b9457fa63efe71138c259c]
I20260130 02:58:35.888394 21249 catalog_manager.cc:679] Loaded metadata for table default.Table_92836e7fcf3a48d8bac2160c2ff7c482 [id=e3ae4b66fa0048668cb1e4d9da7d16fa]
I20260130 02:58:35.889446 21249 catalog_manager.cc:679] Loaded metadata for table default.Table_d24e400f7451443a82d23140a90adc57 [id=e478df970dac40f7a576ce584c3abf2c]
I20260130 02:58:35.890543 21249 catalog_manager.cc:679] Loaded metadata for table default.Table_2f60fd0292ab40c8a3ae9b842cae8faf [id=e648e521ef5044a0b1967452cc3a31b2]
I20260130 02:58:35.891685 21249 catalog_manager.cc:679] Loaded metadata for table default.Table_ba19837ced4c45d79be506e93308449b [id=e732719fbe544e38a15b35017010c7b3]
I20260130 02:58:35.893447 21249 catalog_manager.cc:679] Loaded metadata for table default.Table_c238448062ad4cef8ddd0bd199d5e7f2 [id=e8d3be5336974867a5d65d1ceb4433d6]
I20260130 02:58:35.894505 21249 catalog_manager.cc:679] Loaded metadata for table default.Table_a3ecda6d64704b7fad065a5845683ad8 [id=e93360fd2af141a28d92401ed449c19d]
I20260130 02:58:35.931977 21249 catalog_manager.cc:679] Loaded metadata for table default.Table_3046889233c24f0a8e8058e1591d63b8 [id=edd6505c0bf3462cbf65c3df5649c16c]
I20260130 02:58:35.933892 21249 catalog_manager.cc:679] Loaded metadata for table default.Table_d0b4516d24384e5386c1debaf2543adb [id=f1342f9497b74e7a9ecad1de8082578d]
I20260130 02:58:35.935791 21249 catalog_manager.cc:679] Loaded metadata for table default.Table_5d5e26f93a5a4fb084831add317d90f9 [id=f237b1c90a1743d597d87c5cf1abc89a]
I20260130 02:58:35.937526 21249 catalog_manager.cc:679] Loaded metadata for table default.Table_1e9e20a1bfec4bcf81f9eb6219c33d59 [id=f675b7a3ec744533903d8834eeed51d7]
I20260130 02:58:35.972846 21249 catalog_manager.cc:679] Loaded metadata for table default.Table_f246882197f44c69b72ca421cbdf201f [id=f778784dc88a4e4890113916098ad20f]
I20260130 02:58:35.974086 21249 catalog_manager.cc:679] Loaded metadata for table default.Table_2c737f14bf854cfca58d4a9f83fb2e9e [id=fa4e8955bebb40b6adf6ed4ce0e43bfb]
I20260130 02:58:35.975226 21249 catalog_manager.cc:679] Loaded metadata for table default.Table_e2480022cdae4429b710395eebae7fec [id=fa72411018324474aa9a7265d9f1a2e4]
I20260130 02:58:35.976336 21249 catalog_manager.cc:679] Loaded metadata for table default.Table_e4d07634d84a4678b1c94374e3adc476 [id=fcbd7e3a7fc64d89aa740494d7e1a0f7]
I20260130 02:58:35.977419 21249 catalog_manager.cc:679] Loaded metadata for table default.Table_203b3a960d3e4b5cb978b6168435f3a4 [id=ffe264c47e31462a8deaf81a8b96d34c]
I20260130 02:58:36.069700 21542 raft_consensus.cc:493] T 00000000000000000000000000000000 P 107f274cdda543038832f06a56ab9130 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 658299197a424f53a37f14fadf089ae4)
I20260130 02:58:36.070179 21542 raft_consensus.cc:515] T 00000000000000000000000000000000 P 107f274cdda543038832f06a56ab9130 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "658299197a424f53a37f14fadf089ae4" member_type: VOTER last_known_addr { host: "127.18.100.126" port: 38191 } } peers { permanent_uuid: "107f274cdda543038832f06a56ab9130" member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39711 } } peers { permanent_uuid: "3749baa287ed4f249329a74071781edd" member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34629 } }
I20260130 02:58:36.087325 21542 leader_election.cc:290] T 00000000000000000000000000000000 P 107f274cdda543038832f06a56ab9130 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 658299197a424f53a37f14fadf089ae4 (127.18.100.126:38191), 3749baa287ed4f249329a74071781edd (127.18.100.124:34629)
I20260130 02:58:36.087525 20810 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "107f274cdda543038832f06a56ab9130" candidate_term: 9 candidate_status { last_received { term: 8 index: 375 } } ignore_live_leader: false dest_uuid: "3749baa287ed4f249329a74071781edd" is_pre_election: true
I20260130 02:58:36.096967 21249 tablet_loader.cc:96] loaded metadata for tablet 00ddbfe6218c44d788fbf65245a31dc6 (table default.Table_9cb9a9f9e5d145a6a42d7ccbc42aced9 [id=5f1f179fadc247da9382dd7796dcf2c2])
I20260130 02:58:36.099586 21249 tablet_loader.cc:96] loaded metadata for tablet 03025d13f0e545cfa6288677a2715343 (table default.Table_21178f657dca43a1912ce596834d1618 [id=53b79067a6034189aac5f1554df542c4])
I20260130 02:58:36.110877 21249 tablet_loader.cc:96] loaded metadata for tablet 0855d88806914a36b4bf6b49036f0ad0 (table default.Table_6773a0ad5d6a429086a5818d731f3f4a [id=90b72499c7174fd580914d7fe3714891])
I20260130 02:58:36.112059 21249 tablet_loader.cc:96] loaded metadata for tablet 0a763e48583c4e748a46a325746ce69a (table default.Table_94b23422c82d419d97dc097a3efb92fc [id=d2f35437eae140dab4150079eae46377])
I20260130 02:58:36.113154 21249 tablet_loader.cc:96] loaded metadata for tablet 0b1877e5f9244913bcd639f1336df7ee (table default.Table_19677b801d1f4a63b22139b9fc80c4ef [id=3992cda3a7bb49ae86b8f84ee12f1c8b])
I20260130 02:58:36.114219 21249 tablet_loader.cc:96] loaded metadata for tablet 0d14fb95ace9447f810ccbed4d5befdf (table default.Table_c238448062ad4cef8ddd0bd199d5e7f2 [id=e8d3be5336974867a5d65d1ceb4433d6])
I20260130 02:58:36.123435 21249 tablet_loader.cc:96] loaded metadata for tablet 0e8018c094024c4788c484e8f5466e00 (table default.Table_f17d8e4d9f25449b84aa751af93936b2 [id=43254e466e7b4a2fb59fbcec68cfde36])
I20260130 02:58:36.124661 21249 tablet_loader.cc:96] loaded metadata for tablet 111287b5557d4abda1c5cd560fb0d019 (table default.Table_fec49c37fdd045e7801325ace326f367 [id=06f1da4d6e0e46e4a141914a3aa68c07])
I20260130 02:58:36.126461 21249 tablet_loader.cc:96] loaded metadata for tablet 127a84346b9f473d9e7f739b1e65feb8 (table default.Table_805e07e3546e447d8c8b8029477013cc [id=1515db389f284e2f9ed1a3cd0061d0f1])
I20260130 02:58:36.127560 21249 tablet_loader.cc:96] loaded metadata for tablet 12c3f3ec523040e7b7cd356e3da57e0d (table default.Table_81553b46d7804fd99ef36f431793b001 [id=cdfc1cfcc48f41f8a7a13f865174c3d8])
I20260130 02:58:36.128618 21249 tablet_loader.cc:96] loaded metadata for tablet 133e9bfe175f44aab1a53114abe67e4d (table default.Table_d3f82845a18f4a4a830460cb1cbbb2de [id=c3d0c57ad165460eae9c20c32557fd2e])
I20260130 02:58:36.147179 21249 tablet_loader.cc:96] loaded metadata for tablet 13f0ace4d9de43abab08eca32cdb30c1 (table default.Table_9d0cd9756483406081c8bdb18565a42f [id=5f6488df8ba94ef293b181328cc3b680])
I20260130 02:58:36.149317 21249 tablet_loader.cc:96] loaded metadata for tablet 155ff0c35fc44cb292a28407e6ed937d (table default.Table_dabf999abe834151a0a5b6747a05cac3 [id=038ba9aa798741f6858fa09002d6c3d6])
I20260130 02:58:36.150417 21249 tablet_loader.cc:96] loaded metadata for tablet 15918c90ffdc43a5a38b2950e9834188 (table default.Table_333d2efbf068485a871f566d50e563ce [id=8b7db40afb894ae99625a756c75ef1de])
I20260130 02:58:36.151487 21249 tablet_loader.cc:96] loaded metadata for tablet 16691c9315e94253bb9b3fb5058e5636 (table default.Table_8f311c5878744ef692b496ecfdfc3ede [id=4f499357303d464aafc17679ffcaaa07])
I20260130 02:58:36.153929 21249 tablet_loader.cc:96] loaded metadata for tablet 1fc8f09d6329407c8078542b4123ef4d (table default.Table_7d1c27ab3515406ba5b5333c90df093e [id=b6fabeeae79941dc8094d0f2b51704f9])
I20260130 02:58:36.166993 21249 tablet_loader.cc:96] loaded metadata for tablet 25fd48e693424537b91ba7c3fcc96878 (table default.Table_ebc8b8379c274c18b0a06a919dbe0ee0 [id=28156e0843674fce88f5914a3e4195ae])
I20260130 02:58:36.168841 21249 tablet_loader.cc:96] loaded metadata for tablet 274530359b174ff79f046c2d4a6afa76 (table default.Table_c4f4904b6fac4b9880979cf0314e71c1 [id=c456c6418c93499a9d08f652720854ca])
I20260130 02:58:36.170611 21249 tablet_loader.cc:96] loaded metadata for tablet 2b0edaad6877460cb7982efc62b00b66 (table default.Table_d95381c590c94802bdc83acef0f0df96 [id=64a8e0e079f34806aa68795f3d548c5e])
I20260130 02:58:36.171659 21249 tablet_loader.cc:96] loaded metadata for tablet 2b8034a92b2043e5b57cf02f1ee6296d (table default.Table_e2480022cdae4429b710395eebae7fec [id=fa72411018324474aa9a7265d9f1a2e4])
I20260130 02:58:36.173444 21249 tablet_loader.cc:96] loaded metadata for tablet 2ca0df71836145748cada4e4559a5656 (table default.Table_d764cbaecfee476ba5e734dd1e2f50fd [id=3f9236f0631a4e4c94ca7456f0478a45])
I20260130 02:58:36.175122 21249 tablet_loader.cc:96] loaded metadata for tablet 306452e208ca4e719609f67345114a13 (table default.Table_fa63c9eff99d4ac1b185a4c4ce2df031 [id=2e8b378422c74abc939955d110737316])
I20260130 02:58:36.176148 21249 tablet_loader.cc:96] loaded metadata for tablet 30cc52136bb3422895d95c0fed0c6798 (table default.Table_f619de75da8e4d23b4d641d30285e0e7 [id=2dc3a57f750e4104b99f4589fdb29e3c])
I20260130 02:58:36.177892 21249 tablet_loader.cc:96] loaded metadata for tablet 320dbe98a32a43b9899bbeed80b20f55 (table default.Table_cda216a3bfc14ec0862b998bedd5b300 [id=35978c9d901c47eb81e565e37493b1ae])
I20260130 02:58:36.190903 21249 tablet_loader.cc:96] loaded metadata for tablet 3379641837df4b6ea360e8e27dfe4f8b (table default.Table_8f632ff84d624b72a10cb90fb9e78636 [id=3cf79022ea954f6397f0a0497e6e6a6c])
I20260130 02:58:36.192776 21249 tablet_loader.cc:96] loaded metadata for tablet 341041cb0be2456e88403367c0814704 (table default.Table_1703c69112744c0e951c1ac9667bbac6 [id=8d76f4328edc491291491c78ba205d30])
I20260130 02:58:36.194775 21249 tablet_loader.cc:96] loaded metadata for tablet 345eaaa5625c48d099280b7fbb2e514b (table default.Table_c1886dd9a7934e7797e0e7e0b065d692 [id=7a2d5367224b492b844199088b01b149])
I20260130 02:58:36.195840 21249 tablet_loader.cc:96] loaded metadata for tablet 3593f8b89da24ff9b6cf549d5ac58bc6 (table default.Table_2c737f14bf854cfca58d4a9f83fb2e9e [id=fa4e8955bebb40b6adf6ed4ce0e43bfb])
I20260130 02:58:36.196894 21249 tablet_loader.cc:96] loaded metadata for tablet 3738385130c044e0a5a5803da74c567d (table default.Table_ddfb5df90fdb40a087ac434d088bbf0e [id=1e72b7dd669d4539b61884ba63ff7183])
I20260130 02:58:36.199994 21249 tablet_loader.cc:96] loaded metadata for tablet 39bcc41f10ae4241b370529364454586 (table default.Table_a132cf962cfc4ed8a6a8c177e088daeb [id=1c4937ca6e014fec8596a3dc25779600])
I20260130 02:58:36.201035 21249 tablet_loader.cc:96] loaded metadata for tablet 3aa76993beb94b12bc706a76db26a03b (table default.Table_477fb92965284222aa1ecaa0107d1ec2 [id=ad82881a19e748edb569e74d692bb2bc])
I20260130 02:58:36.230623 21249 tablet_loader.cc:96] loaded metadata for tablet 3ae5472cba1e4613abe4cda42ccb2496 (table default.Table_69267d31df644f4396a44130fa3d499d [id=7efa53800a974e31a38a4c60c0a1b466])
I20260130 02:58:36.231820 21249 tablet_loader.cc:96] loaded metadata for tablet 3b02480d9c6f4328af83f7940716d698 (table default.Table_1e9e20a1bfec4bcf81f9eb6219c33d59 [id=f675b7a3ec744533903d8834eeed51d7])
I20260130 02:58:36.237031 21249 tablet_loader.cc:96] loaded metadata for tablet 475b435b2967418f927fa172d2ff04ea (table default.Table_dee034aec0b24982b8dc2a3fa1b326f1 [id=66033e75b3814a8584311cf7e4a07ad3])
I20260130 02:58:36.237988 21249 tablet_loader.cc:96] loaded metadata for tablet 47f574674aa7414588605ca38eb7200a (table default.Table_d2a3438ed66741eab245ae6d6c5c2f37 [id=4fb5c26265e94083a8714be512fcd222])
I20260130 02:58:36.239872 21249 tablet_loader.cc:96] loaded metadata for tablet 4907294442c145c6b03eb1de057c66a8 (table default.Table_ea2e0aa925a0486bb292304ca21bb819 [id=18580b9c54e448d1bdaae84f697f7b0b])
I20260130 02:58:36.240917 21249 tablet_loader.cc:96] loaded metadata for tablet 49efcfced0b74c48962393149ac2a500 (table default.Table_2e81f0a0cbc449a397141b717db17025 [id=a9c919f57c0c437cb35a6921103e45a6])
I20260130 02:58:36.241986 21249 tablet_loader.cc:96] loaded metadata for tablet 4c139aa8cd184f488cec4e05ef7dcada (table default.Table_e4d07634d84a4678b1c94374e3adc476 [id=fcbd7e3a7fc64d89aa740494d7e1a0f7])
I20260130 02:58:36.243057 21249 tablet_loader.cc:96] loaded metadata for tablet 532e867fd5944d468d7672d2a5c92f3c (table default.Table_0ca1d7370d2f416782b386c5b47111ab [id=3ffe1a273d4f452c835278a8e7f10102])
I20260130 02:58:36.244848 21249 tablet_loader.cc:96] loaded metadata for tablet 54be22352e404ff9b798a075693f7f21 (table default.Table_3a858a3cb886427cb8268f42c953ad4c [id=17150cb73e374ff9a1e92c43e0c3f2c1])
I20260130 02:58:36.247412 21249 tablet_loader.cc:96] loaded metadata for tablet 5fbe92a2227d44d88b4531a7f8aa0e08 (table default.Table_a3ecda6d64704b7fad065a5845683ad8 [id=e93360fd2af141a28d92401ed449c19d])
I20260130 02:58:36.248515 21249 tablet_loader.cc:96] loaded metadata for tablet 6218f2f7d73a42428b57a45858f53afd (table default.Table_95f0bff8e17d4ca995c0af0d0c24e84d [id=b7795905d0c440579c59d6ac00473f5a])
I20260130 02:58:36.250366 21249 tablet_loader.cc:96] loaded metadata for tablet 657e9b0d8c05413f904129dce745710b (table default.Table_526677c8ef4144bd9de7a54a5e78e2be [id=b16908b75587459597a978c71756451c])
I20260130 02:58:36.251464 21249 tablet_loader.cc:96] loaded metadata for tablet 66a7487a7ab343a4abca5312afe78862 (table default.Table_6c46e6995d4741dcaaa6ed7bdc765e29 [id=6d91a161cf484239967af58db160c952])
I20260130 02:58:36.252523 21249 tablet_loader.cc:96] loaded metadata for tablet 6ad6bb8c986a433d94c745f2b47266ff (table default.Table_bc168ff8b5f34cacbd398117647ba25a [id=2a21ee52b7d246f1b44eda02bf5a411e])
I20260130 02:58:36.253739 21249 tablet_loader.cc:96] loaded metadata for tablet 6d1f121f49cd47c2b8247daaf6874e01 (table default.Table_0480357296ac4a27812d802ee4566351 [id=0da6197ab32e4645aa8d8787662ee5e6])
I20260130 02:58:36.254773 21249 tablet_loader.cc:96] loaded metadata for tablet 6e12a8eeb3204325a9a8b7779f72ab29 (table default.Table_8c27a7d10b7847e78029f7e95321e9b6 [id=e31c1bf0c3b9457fa63efe71138c259c])
I20260130 02:58:36.255788 21249 tablet_loader.cc:96] loaded metadata for tablet 6f5a90b75a0f4851b5db672c41253f7b (table default.Table_6f53c17cfd144031b2d4b112fb2d88eb [id=443cbad5510d463fa8ca878b8628dbda])
I20260130 02:58:36.257536 21249 tablet_loader.cc:96] loaded metadata for tablet 753344737b5941d9a93b6779e72f63d4 (table default.Table_8896365e792f4d24b2c16c2d269b0a93 [id=44816e07033849479a7fea0f51d00fad])
I20260130 02:58:36.258647 21249 tablet_loader.cc:96] loaded metadata for tablet 7783db3568f4434e91e9e456799cc312 (table default.Table_2d345d0f0957486c86c2cf2e34b00941 [id=4c3e0d3ecd5642a3b7fc7d20e2521418])
I20260130 02:58:36.262346 21249 tablet_loader.cc:96] loaded metadata for tablet 7caf7cbad5054396b5617094e4d536a0 (table default.Table_007e969f70d345a7bcb3dfbaabbc025d [id=77d4fa8705a3419cb0a84d3ca1bb4ce3])
I20260130 02:58:36.265316 21249 tablet_loader.cc:96] loaded metadata for tablet 838e49e5e8874db393bdc15a1c92f87e (table default.Table_da3d581b15a94840898777f74611994e [id=b7e241ced53e4160ac6fa1f65ef3674e])
I20260130 02:58:36.267092 21249 tablet_loader.cc:96] loaded metadata for tablet 8b47653c7c3945a68e9bf4680bfc0496 (table default.Table_1ae68ae3f0404843aca3c72f02678ac6 [id=8a9182f49e18489fafec8dca190c1b2f])
I20260130 02:58:36.270042 21249 tablet_loader.cc:96] loaded metadata for tablet 91f954f3ef4244c9a575a7421a7e75eb (table default.Table_9f83f87711d94e8394abe1925ac84ad1 [id=db23c1be42c44b6f9eb774f14d2b8fd6])
I20260130 02:58:36.271090 21249 tablet_loader.cc:96] loaded metadata for tablet 920f52b975ab480fbf42656171222f09 (table default.Table_78b080efddd94878b9d8b77b5ef3af35 [id=b654822c58224a859ad2dcdd8d0123c2])
I20260130 02:58:36.273494 21249 tablet_loader.cc:96] loaded metadata for tablet 94355fc1338f479e8798070aff04fcba (table default.Table_12c2fe20cc3e45d29b7be37f900f68a9 [id=67ac4f4eda324e799b5867b2b011a16b])
I20260130 02:58:36.274547 21249 tablet_loader.cc:96] loaded metadata for tablet 955b9547f49f434699fe1d86b260638c (table default.Table_f246882197f44c69b72ca421cbdf201f [id=f778784dc88a4e4890113916098ad20f])
I20260130 02:58:36.287937 21249 tablet_loader.cc:96] loaded metadata for tablet 9896ff9364744857968e87e80e776dd2 (table default.Table_5d5e26f93a5a4fb084831add317d90f9 [id=f237b1c90a1743d597d87c5cf1abc89a])
I20260130 02:58:36.289076 21249 tablet_loader.cc:96] loaded metadata for tablet 9fe0448bcc4a4346b3c67fc3f626ed3c (table default.Table_5c1c447e85524ac5a61ffd8a48509d36 [id=70aaf21ad9bd4d959f46939c6b4a4880])
I20260130 02:58:36.290109 21249 tablet_loader.cc:96] loaded metadata for tablet a3258852aed24ed582a79f77860e6f0d (table default.Table_7c48a0d054354501947d11c5fd7b1864 [id=aa4ad1394ab344df8c54ba0031af5590])
I20260130 02:58:36.293236 21249 tablet_loader.cc:96] loaded metadata for tablet a90712651a604b83ba187f893ab17711 (table default.Table_0f3e44ae93bc4c028ed3929b94a49a24 [id=3b9c62fc8bbb44ffb7d66e9604974c30])
I20260130 02:58:36.294276 21249 tablet_loader.cc:96] loaded metadata for tablet a90e92d569254bbf8a5fbf286baf5cce (table default.Table_1e42d571631143088cf33d4c3aafed28 [id=30ba2af6dffd4f7e8dcbfdd2f3798157])
I20260130 02:58:36.303445 21249 tablet_loader.cc:96] loaded metadata for tablet aae68390dbbe4dc58ce3d3bd01725ae6 (table default.Table_f3e214a955f6447482099175b92ecb5c [id=76a688022fd9421a84de059b3c1da77a])
I20260130 02:58:36.305259 21249 tablet_loader.cc:96] loaded metadata for tablet aedf72892b7248e2ace8772283c2b476 (table default.Table_203b3a960d3e4b5cb978b6168435f3a4 [id=ffe264c47e31462a8deaf81a8b96d34c])
I20260130 02:58:36.306316 21249 tablet_loader.cc:96] loaded metadata for tablet af0f90071a334016aa17fae2e3a97fc8 (table default.Table_22d10332a83f4700ac21892bab67f46b [id=da3a9cb1f6d5419abc1afc93659b5324])
I20260130 02:58:36.309170 21249 tablet_loader.cc:96] loaded metadata for tablet b1e8b78eff0d4a0b92059adb2de4a3ce (table default.Table_d0b4516d24384e5386c1debaf2543adb [id=f1342f9497b74e7a9ecad1de8082578d])
I20260130 02:58:36.312348 21249 tablet_loader.cc:96] loaded metadata for tablet b89ed298e3dc49d789d166c613ad7fb8 (table default.Table_f8856f06ce97440bae4abdf34164716d [id=a3303ada20a94f99b2f37a02eec361ef])
I20260130 02:58:36.313414 21249 tablet_loader.cc:96] loaded metadata for tablet b8c579ff5fa84651ba9125457d927162 (table default.Table_3046889233c24f0a8e8058e1591d63b8 [id=edd6505c0bf3462cbf65c3df5649c16c])
I20260130 02:58:36.315388 21249 tablet_loader.cc:96] loaded metadata for tablet bac739bf4cac4272975f5587ad9c71ff (table default.Table_9e942a1ce3944e9aa028e75656ce066d [id=18ed268eb9ad4450abf9874310ace074])
I20260130 02:58:36.316416 21249 tablet_loader.cc:96] loaded metadata for tablet bc04be1284414a8699d3c4b88bf62813 (table default.Table_2414288f0dc54e5f835f876060927a9b [id=b0c6f2d9483c47da8ff3e65f0400a593])
I20260130 02:58:36.318809 21249 tablet_loader.cc:96] loaded metadata for tablet c2c927e461f04fb48bfa9e3a685b077d (table default.Table_aa496bf2019748f29823a9a253b1f3ca [id=780188e2335f43fbb11de8212eb38d56])
I20260130 02:58:36.320494 21249 tablet_loader.cc:96] loaded metadata for tablet c76f0dc7762c49c389fa6fef41f02141 (table default.Table_e894e5560f194692bac730600c6bf5f4 [id=d5e1b2a45122429aabb4fd8653d71f0f])
I20260130 02:58:36.322237 21249 tablet_loader.cc:96] loaded metadata for tablet ca06f07bb98f45a7bf63aebc28348e08 (table default.Table_829d1e187e8b4db5b8f4f835e84246e5 [id=a24b4761037d47a383f454a82517131f])
I20260130 02:58:36.323293 21249 tablet_loader.cc:96] loaded metadata for tablet cd30d7adcc4545848e3501a205a10293 (table default.Table_b85cb876f25740bfaaab01949ac26b38 [id=839e292e040c4aeb832f3fc0258c32ec])
I20260130 02:58:36.325045 21249 tablet_loader.cc:96] loaded metadata for tablet cf1b66a20fce4c4d96bc0bc32a7bafa8 (table default.Table_d24e400f7451443a82d23140a90adc57 [id=e478df970dac40f7a576ce584c3abf2c])
I20260130 02:58:36.326071 21249 tablet_loader.cc:96] loaded metadata for tablet cfd12a72726846e587cff9d6a4d61bd6 (table default.Table_4ee03db7d0064d96a53d4f261bd12722 [id=07fe2931234d4888a524b52171b03f0a])
I20260130 02:58:36.327879 21249 tablet_loader.cc:96] loaded metadata for tablet d3ba2df0b15c44f882070570f91b254d (table default.Table_92836e7fcf3a48d8bac2160c2ff7c482 [id=e3ae4b66fa0048668cb1e4d9da7d16fa])
I20260130 02:58:36.328913 21249 tablet_loader.cc:96] loaded metadata for tablet d480bf18140245e3870a62582d5a8318 (table default.Table_109c8b6c17d4442b939bde9441d0476f [id=dcf82ea15d6044c2b023f81f0a579ade])
I20260130 02:58:36.332265 21249 tablet_loader.cc:96] loaded metadata for tablet dc09a0f0cecf49738bab04596944c0f3 (table default.Table_6a7faee4c75c4eb7862fb4c05afc17cb [id=676ece9dc36b418a8b1400e8406d6b98])
I20260130 02:58:36.333191 21249 tablet_loader.cc:96] loaded metadata for tablet dc4bdd8e4f104d4a86c0af6ac6accda0 (table default.Table_ba19837ced4c45d79be506e93308449b [id=e732719fbe544e38a15b35017010c7b3])
I20260130 02:58:36.334228 21249 tablet_loader.cc:96] loaded metadata for tablet dcba6b534f9d406dad4fb502be40921c (table default.Table_baaf376733384fdbbac49ec8f2357432 [id=dea97f19a50141878f97820194afa59e])
I20260130 02:58:36.335305 21249 tablet_loader.cc:96] loaded metadata for tablet dd377fab8c544bf69121dc9e9a060b29 (table default.Table_5e70ab0e32bc4048bd19388f9d7cd302 [id=030c8ac23aac4a1fb7acc954d09434db])
I20260130 02:58:36.337790 21249 tablet_loader.cc:96] loaded metadata for tablet dfbbe9045e7c4aa5b520218687e7e4c1 (table default.Table_e03bdbb221e94bd6b535ddc02c863bad [id=2382eeebdc9540d5b588835705b2a4f1])
I20260130 02:58:36.338896 21249 tablet_loader.cc:96] loaded metadata for tablet e026dee22493424692d00f14848ab57c (table default.Table_85611716686c4097bacb64df677d5793 [id=122fe719b1d64ec8b531783539f85df1])
I20260130 02:58:36.351742 21249 tablet_loader.cc:96] loaded metadata for tablet e34d5da0ad864b2eaea0d575c982e447 (table default.Table_aa4e5fd22945475da1dad48e1508595e [id=5b41834c6d8b4d89b773c2ea4c4ca7e6])
I20260130 02:58:36.354266 21249 tablet_loader.cc:96] loaded metadata for tablet e691c238199b4914bcc3fd248f31420c (table default.Table_7fe8c354c0dc4cfba0e84ed33e4c2506 [id=7523d671e9ac4814acff5060fd282eea])
I20260130 02:58:36.370486 21249 tablet_loader.cc:96] loaded metadata for tablet eb7a755ab62b4552bb8588264c5f7b75 (table default.Table_7e6ab9964f4c4e0f906d875a937a781d [id=bcf41af45d284bd89800977d14194812])
I20260130 02:58:36.372529 21249 tablet_loader.cc:96] loaded metadata for tablet ee7c8ff92ca9465fa18be6eae63d78ca (table default.Table_dd22e9dc63f84d8688c0071f427f0b25 [id=77979d498aa040828de415129cb15a06])
I20260130 02:58:36.373651 21249 tablet_loader.cc:96] loaded metadata for tablet f2d01bfdc9c4494386bd02bed655fe63 (table default.Table_aaf139c1a55d4a1783298ca12ae0d5c0 [id=d128767e303145a885eec3c5f714a27f])
I20260130 02:58:36.374758 21249 tablet_loader.cc:96] loaded metadata for tablet f41f4cff7dd4489f948700a45e9a7057 (table default.Table_ed6e2e20d89542199e5cbc7842df964d [id=047c39bd14664d1099e043b973766d32])
I20260130 02:58:36.377190 21249 tablet_loader.cc:96] loaded metadata for tablet f7607df5efd543438fb2dff3667cc8c8 (table default.Table_e8fd7516280f4dc4a73eb1f88fe916a2 [id=6efb836791414f3888f714aa8f65f0a4])
I20260130 02:58:36.378257 21249 tablet_loader.cc:96] loaded metadata for tablet f9e5f01e69254d658c369f3e21cef523 (table default.Table_12b505700b3d44e28353172e1786e5c8 [id=562776405d6f4c7aab0622f3866bf45b])
I20260130 02:58:36.397936 21249 tablet_loader.cc:96] loaded metadata for tablet ffa1142fc7274aecbdbee1a0802b95da (table default.Table_2f60fd0292ab40c8a3ae9b842cae8faf [id=e648e521ef5044a0b1967452cc3a31b2])
I20260130 02:58:36.455829 21249 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260130 02:58:36.455906 18885 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "107f274cdda543038832f06a56ab9130" candidate_term: 9 candidate_status { last_received { term: 8 index: 375 } } ignore_live_leader: false dest_uuid: "658299197a424f53a37f14fadf089ae4" is_pre_election: true
I20260130 02:58:36.457975 18934 leader_election.cc:304] T 00000000000000000000000000000000 P 107f274cdda543038832f06a56ab9130 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 107f274cdda543038832f06a56ab9130; no voters: 3749baa287ed4f249329a74071781edd, 658299197a424f53a37f14fadf089ae4
I20260130 02:58:36.458046 18877 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3749baa287ed4f249329a74071781edd" candidate_term: 9 candidate_status { last_received { term: 8 index: 375 } } ignore_live_leader: false dest_uuid: "658299197a424f53a37f14fadf089ae4" is_pre_election: true
W20260130 02:58:36.459039 18877 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35042 (request call id 13) took 959 ms (client timeout 673 ms). Trace:
I20260130 02:58:36.459602 21542 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 107f274cdda543038832f06a56ab9130 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20260130 02:58:36.459429 18877 rpcz_store.cc:269] 0130 02:58:35.499321 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:58:35.499467 (+ 146us) service_pool.cc:225] Handling call
0130 02:58:36.458957 (+959490us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260130 02:58:36.455860 18883 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3749baa287ed4f249329a74071781edd" candidate_term: 9 candidate_status { last_received { term: 8 index: 375 } } ignore_live_leader: false dest_uuid: "658299197a424f53a37f14fadf089ae4" is_pre_election: true
I20260130 02:58:36.461874 20613 leader_election.cc:304] T 00000000000000000000000000000000 P 3749baa287ed4f249329a74071781edd [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3749baa287ed4f249329a74071781edd; no voters: 107f274cdda543038832f06a56ab9130, 658299197a424f53a37f14fadf089ae4
I20260130 02:58:36.471383 21547 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 3749baa287ed4f249329a74071781edd [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20260130 02:58:36.482679 19463 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.100.65:42021, user_credentials={real_user=slave}} blocked reactor thread for 114166us
I20260130 02:58:36.577086 21249 catalog_manager.cc:1269] Loaded cluster ID: db1c0e7709154c10a8c9a0ea68ffe38b
I20260130 02:58:36.577374 21249 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260130 02:58:36.690764 21249 catalog_manager.cc:1514] Loading token signing keys...
I20260130 02:58:36.780181 21249 catalog_manager.cc:6038] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4: Loaded TSK: 0
I20260130 02:58:36.781219 21249 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260130 02:58:36.902274 18868 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50284:
table { table_name: "default.Table_baaf376733384fdbbac49ec8f2357432" } new_table_name: "default.Table_a99512d7e42f46509759ae14fba6fd9b" modify_external_catalogs: true
I20260130 02:58:36.903311 18869 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50284:
table { table_name: "default.Table_92836e7fcf3a48d8bac2160c2ff7c482" } modify_external_catalogs: true
I20260130 02:58:36.903968 18869 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50284:
table { table_name: "default.Table_92836e7fcf3a48d8bac2160c2ff7c482" } modify_external_catalogs: true
I20260130 02:58:36.904822 18866 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50284:
table { table_name: "default.Table_6a7faee4c75c4eb7862fb4c05afc17cb" } new_table_name: "default.Table_32f4a0f63b1046529836267ff3f551fa" modify_external_catalogs: true
I20260130 02:58:36.905625 18867 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50284:
name: "default.Table_6960856bdebc4d3d811d307f2e3ec7a6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260130 02:58:36.907605 18867 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6960856bdebc4d3d811d307f2e3ec7a6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260130 02:58:36.920043 18875 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50284:
name: "default.Table_c479426df54f421fb41cdc93d84ac324"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260130 02:58:36.921607 18875 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c479426df54f421fb41cdc93d84ac324 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260130 02:58:36.922896 19462 consensus_peers.cc:597] T ab6afa40408a4f458495abf20290e55b P d870164263bb44b6b780a557daea3656 -> Peer 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767): Couldn't send request to peer 78c2c46c7520492db35d51c244fb4f74. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ab6afa40408a4f458495abf20290e55b. This is attempt 6: this message will repeat every 5th retry.
I20260130 02:58:36.923224 18872 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50284:
name: "default.Table_9efd5f63d3824c00950bd6a4fceabe09"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260130 02:58:36.924757 18872 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9efd5f63d3824c00950bd6a4fceabe09 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260130 02:58:36.927130 18873 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50284:
name: "default.Table_12eed69fd8f343df810ae768b5761879"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260130 02:58:36.928508 18873 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_12eed69fd8f343df810ae768b5761879 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260130 02:58:36.930920 18874 master_service.cc:954] ReplaceTablet: received request to replace tablet 345eaaa5625c48d099280b7fbb2e514b from {username='slave'} at 127.0.0.1:50284
I20260130 02:58:36.931604 18870 master_service.cc:954] ReplaceTablet: received request to replace tablet 111287b5557d4abda1c5cd560fb0d019 from {username='slave'} at 127.0.0.1:50284
I20260130 02:58:36.995548 19505 tablet_service.cc:1552] Processing DeleteTablet for tablet ab6afa40408a4f458495abf20290e55b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:58:15 UTC) from {username='slave'} at 127.0.0.1:59068
I20260130 02:58:37.016160 21554 tablet_replica.cc:333] T ab6afa40408a4f458495abf20290e55b P d870164263bb44b6b780a557daea3656: stopping tablet replica
I20260130 02:58:37.045290 21554 raft_consensus.cc:2243] T ab6afa40408a4f458495abf20290e55b P d870164263bb44b6b780a557daea3656 [term 1 LEADER]: Raft consensus shutting down.
I20260130 02:58:37.046383 21554 raft_consensus.cc:2272] T ab6afa40408a4f458495abf20290e55b P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260130 02:58:37.048743 21554 ts_tablet_manager.cc:1916] T ab6afa40408a4f458495abf20290e55b P d870164263bb44b6b780a557daea3656: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260130 02:58:37.095302 21554 ts_tablet_manager.cc:1929] T ab6afa40408a4f458495abf20290e55b P d870164263bb44b6b780a557daea3656: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260130 02:58:37.106920 21554 log.cc:1199] T ab6afa40408a4f458495abf20290e55b P d870164263bb44b6b780a557daea3656: Deleting WAL directory at /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-2/wal/wals/ab6afa40408a4f458495abf20290e55b
I20260130 02:58:37.108131 21554 ts_tablet_manager.cc:1950] T ab6afa40408a4f458495abf20290e55b P d870164263bb44b6b780a557daea3656: Deleting consensus metadata
I20260130 02:58:37.117146 18860 catalog_manager.cc:4985] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): tablet ab6afa40408a4f458495abf20290e55b (table default.Table_748cb9c252084fb48ceac74812b82a64 [id=237bc2670f354027a817a6203f7e662d]) successfully deleted
I20260130 02:58:37.136611 18869 catalog_manager.cc:5941] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4: Sending DeleteTablet for 3 replicas of tablet d3ba2df0b15c44f882070570f91b254d
I20260130 02:58:37.140661 19505 tablet_service.cc:1552] Processing DeleteTablet for tablet d3ba2df0b15c44f882070570f91b254d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:58:36 UTC) from {username='slave'} at 127.0.0.1:59068
I20260130 02:58:37.140753 19358 tablet_service.cc:1552] Processing DeleteTablet for tablet d3ba2df0b15c44f882070570f91b254d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:58:36 UTC) from {username='slave'} at 127.0.0.1:60110
I20260130 02:58:37.141449 21554 tablet_replica.cc:333] T d3ba2df0b15c44f882070570f91b254d P d870164263bb44b6b780a557daea3656: stopping tablet replica
I20260130 02:58:37.142482 21554 raft_consensus.cc:2243] T d3ba2df0b15c44f882070570f91b254d P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260130 02:58:37.146857 21554 raft_consensus.cc:2272] T d3ba2df0b15c44f882070570f91b254d P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260130 02:58:37.146821 19202 tablet_service.cc:1552] Processing DeleteTablet for tablet d3ba2df0b15c44f882070570f91b254d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-30 02:58:36 UTC) from {username='slave'} at 127.0.0.1:42870
I20260130 02:58:37.149137 21554 ts_tablet_manager.cc:1916] T d3ba2df0b15c44f882070570f91b254d P d870164263bb44b6b780a557daea3656: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260130 02:58:37.170064 21565 tablet_replica.cc:333] T d3ba2df0b15c44f882070570f91b254d P 78c2c46c7520492db35d51c244fb4f74: stopping tablet replica
I20260130 02:58:37.171000 21565 raft_consensus.cc:2243] T d3ba2df0b15c44f882070570f91b254d P 78c2c46c7520492db35d51c244fb4f74 [term 1 LEADER]: Raft consensus shutting down.
I20260130 02:58:37.171990 21565 raft_consensus.cc:2272] T d3ba2df0b15c44f882070570f91b254d P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260130 02:58:37.182863 18833 master-stress-test.cc:579] Killed test thread 1/10
I20260130 02:58:37.183229 18833 master-stress-test.cc:577] Killing test thread 2/10
I20260130 02:58:37.184490 18833 master-stress-test.cc:579] Killed test thread 2/10
I20260130 02:58:37.184731 18833 master-stress-test.cc:577] Killing test thread 3/10
I20260130 02:58:37.172104 21566 tablet_replica.cc:333] T d3ba2df0b15c44f882070570f91b254d P 371a36d219164974abdf848841528152: stopping tablet replica
I20260130 02:58:37.188463 21566 raft_consensus.cc:2243] T d3ba2df0b15c44f882070570f91b254d P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260130 02:58:37.199204 21574 tablet.cc:1722] T dcba6b534f9d406dad4fb502be40921c P d870164263bb44b6b780a557daea3656: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260130 02:58:37.207716 18833 master-stress-test.cc:579] Killed test thread 3/10
I20260130 02:58:37.208006 18833 master-stress-test.cc:577] Killing test thread 4/10
I20260130 02:58:37.208242 18833 master-stress-test.cc:579] Killed test thread 4/10
I20260130 02:58:37.208431 18833 master-stress-test.cc:577] Killing test thread 5/10
I20260130 02:58:37.208659 18833 master-stress-test.cc:579] Killed test thread 5/10
I20260130 02:58:37.208851 18833 master-stress-test.cc:577] Killing test thread 6/10
I20260130 02:58:37.208876 18870 catalog_manager.cc:5941] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4: Sending DeleteTablet for 3 replicas of tablet 111287b5557d4abda1c5cd560fb0d019
I20260130 02:58:37.209261 18833 master-stress-test.cc:579] Killed test thread 6/10
I20260130 02:58:37.209471 18833 master-stress-test.cc:577] Killing test thread 7/10
I20260130 02:58:37.210160 18874 catalog_manager.cc:5941] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4: Sending DeleteTablet for 3 replicas of tablet 345eaaa5625c48d099280b7fbb2e514b
I20260130 02:58:37.210700 18833 master-stress-test.cc:579] Killed test thread 7/10
I20260130 02:58:37.210915 18833 master-stress-test.cc:577] Killing test thread 8/10
I20260130 02:58:37.211189 18833 master-stress-test.cc:579] Killed test thread 8/10
I20260130 02:58:37.211485 18833 master-stress-test.cc:577] Killing test thread 9/10
I20260130 02:58:37.211519 19358 tablet_service.cc:1552] Processing DeleteTablet for tablet 111287b5557d4abda1c5cd560fb0d019 with delete_type TABLET_DATA_DELETED (replaced by tablet 0eeee0ddd9fa4f8a8c694fb134026efd) from {username='slave'} at 127.0.0.1:60110
I20260130 02:58:37.211526 19505 tablet_service.cc:1552] Processing DeleteTablet for tablet 111287b5557d4abda1c5cd560fb0d019 with delete_type TABLET_DATA_DELETED (replaced by tablet 0eeee0ddd9fa4f8a8c694fb134026efd) from {username='slave'} at 127.0.0.1:59068
I20260130 02:58:37.212502 18874 catalog_manager.cc:6724] ReplaceTablet: tablet 345eaaa5625c48d099280b7fbb2e514b deleted and replaced by tablet aa87a26273ab48f3b60760902bafcbea
I20260130 02:58:37.210152 18870 catalog_manager.cc:6724] ReplaceTablet: tablet 111287b5557d4abda1c5cd560fb0d019 deleted and replaced by tablet 0eeee0ddd9fa4f8a8c694fb134026efd
I20260130 02:58:37.214077 19505 tablet_service.cc:1552] Processing DeleteTablet for tablet 345eaaa5625c48d099280b7fbb2e514b with delete_type TABLET_DATA_DELETED (replaced by tablet aa87a26273ab48f3b60760902bafcbea) from {username='slave'} at 127.0.0.1:59068
I20260130 02:58:37.214630 19358 tablet_service.cc:1552] Processing DeleteTablet for tablet 345eaaa5625c48d099280b7fbb2e514b with delete_type TABLET_DATA_DELETED (replaced by tablet aa87a26273ab48f3b60760902bafcbea) from {username='slave'} at 127.0.0.1:60110
I20260130 02:58:37.211030 19202 tablet_service.cc:1552] Processing DeleteTablet for tablet 111287b5557d4abda1c5cd560fb0d019 with delete_type TABLET_DATA_DELETED (replaced by tablet 0eeee0ddd9fa4f8a8c694fb134026efd) from {username='slave'} at 127.0.0.1:42870
I20260130 02:58:37.215274 19199 tablet_service.cc:1552] Processing DeleteTablet for tablet 345eaaa5625c48d099280b7fbb2e514b with delete_type TABLET_DATA_DELETED (replaced by tablet aa87a26273ab48f3b60760902bafcbea) from {username='slave'} at 127.0.0.1:42870
I20260130 02:58:37.215516 21566 raft_consensus.cc:2272] T d3ba2df0b15c44f882070570f91b254d P 371a36d219164974abdf848841528152 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260130 02:58:37.215821 21573 tablet.cc:1722] T dcba6b534f9d406dad4fb502be40921c P 371a36d219164974abdf848841528152: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260130 02:58:37.218411 21566 ts_tablet_manager.cc:1916] T d3ba2df0b15c44f882070570f91b254d P 371a36d219164974abdf848841528152: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260130 02:58:37.223278 21565 ts_tablet_manager.cc:1916] T d3ba2df0b15c44f882070570f91b254d P 78c2c46c7520492db35d51c244fb4f74: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260130 02:58:37.226759 18833 master-stress-test.cc:579] Killed test thread 9/10
I20260130 02:58:37.227033 18833 master-stress-test.cc:577] Killing test thread 10/10
I20260130 02:58:37.227346 18833 master-stress-test.cc:579] Killed test thread 10/10
I20260130 02:58:37.227684 18833 external_mini_cluster.cc:1658] Killing /tmp/dist-test-tasktGe2wI/build/tsan/bin/kudu with pid 19114
I20260130 02:58:37.231824 21554 ts_tablet_manager.cc:1929] T d3ba2df0b15c44f882070570f91b254d P d870164263bb44b6b780a557daea3656: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260130 02:58:37.232335 21554 log.cc:1199] T d3ba2df0b15c44f882070570f91b254d P d870164263bb44b6b780a557daea3656: Deleting WAL directory at /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-2/wal/wals/d3ba2df0b15c44f882070570f91b254d
I20260130 02:58:37.233162 21554 ts_tablet_manager.cc:1950] T d3ba2df0b15c44f882070570f91b254d P d870164263bb44b6b780a557daea3656: Deleting consensus metadata
I20260130 02:58:37.235890 18860 catalog_manager.cc:4985] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): tablet d3ba2df0b15c44f882070570f91b254d (table default.Table_92836e7fcf3a48d8bac2160c2ff7c482 [id=e3ae4b66fa0048668cb1e4d9da7d16fa]) successfully deleted
I20260130 02:58:37.237704 21554 tablet_replica.cc:333] T 111287b5557d4abda1c5cd560fb0d019 P d870164263bb44b6b780a557daea3656: stopping tablet replica
I20260130 02:58:37.247017 21554 raft_consensus.cc:2243] T 111287b5557d4abda1c5cd560fb0d019 P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260130 02:58:37.247712 21554 raft_consensus.cc:2272] T 111287b5557d4abda1c5cd560fb0d019 P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260130 02:58:37.252425 19575 heartbeater.cc:499] Master 127.18.100.126:38191 was elected leader, sending a full tablet report...
I20260130 02:58:37.281401 21554 ts_tablet_manager.cc:1916] T 111287b5557d4abda1c5cd560fb0d019 P d870164263bb44b6b780a557daea3656: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260130 02:58:37.337600 21565 ts_tablet_manager.cc:1929] T d3ba2df0b15c44f882070570f91b254d P 78c2c46c7520492db35d51c244fb4f74: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260130 02:58:37.351241 21565 log.cc:1199] T d3ba2df0b15c44f882070570f91b254d P 78c2c46c7520492db35d51c244fb4f74: Deleting WAL directory at /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-1/wal/wals/d3ba2df0b15c44f882070570f91b254d
I20260130 02:58:37.350895 21554 ts_tablet_manager.cc:1929] T 111287b5557d4abda1c5cd560fb0d019 P d870164263bb44b6b780a557daea3656: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260130 02:58:37.354825 21554 log.cc:1199] T 111287b5557d4abda1c5cd560fb0d019 P d870164263bb44b6b780a557daea3656: Deleting WAL directory at /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-2/wal/wals/111287b5557d4abda1c5cd560fb0d019
I20260130 02:58:37.355388 21565 ts_tablet_manager.cc:1950] T d3ba2df0b15c44f882070570f91b254d P 78c2c46c7520492db35d51c244fb4f74: Deleting consensus metadata
I20260130 02:58:37.355569 21554 ts_tablet_manager.cc:1950] T 111287b5557d4abda1c5cd560fb0d019 P d870164263bb44b6b780a557daea3656: Deleting consensus metadata
I20260130 02:58:37.358306 18860 catalog_manager.cc:4985] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767): tablet d3ba2df0b15c44f882070570f91b254d (table default.Table_92836e7fcf3a48d8bac2160c2ff7c482 [id=e3ae4b66fa0048668cb1e4d9da7d16fa]) successfully deleted
I20260130 02:58:37.363801 18860 catalog_manager.cc:4985] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): tablet 111287b5557d4abda1c5cd560fb0d019 (table default.Table_fec49c37fdd045e7801325ace326f367 [id=06f1da4d6e0e46e4a141914a3aa68c07]) successfully deleted
I20260130 02:58:37.365999 21554 tablet_replica.cc:333] T 345eaaa5625c48d099280b7fbb2e514b P d870164263bb44b6b780a557daea3656: stopping tablet replica
I20260130 02:58:37.372442 21554 raft_consensus.cc:2243] T 345eaaa5625c48d099280b7fbb2e514b P d870164263bb44b6b780a557daea3656 [term 1 LEADER]: Raft consensus shutting down.
I20260130 02:58:37.374290 21554 raft_consensus.cc:2272] T 345eaaa5625c48d099280b7fbb2e514b P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260130 02:58:37.374665 21565 tablet_replica.cc:333] T 111287b5557d4abda1c5cd560fb0d019 P 78c2c46c7520492db35d51c244fb4f74: stopping tablet replica
I20260130 02:58:37.379269 21565 raft_consensus.cc:2243] T 111287b5557d4abda1c5cd560fb0d019 P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260130 02:58:37.386431 21565 raft_consensus.cc:2272] T 111287b5557d4abda1c5cd560fb0d019 P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260130 02:58:37.389330 21565 ts_tablet_manager.cc:1916] T 111287b5557d4abda1c5cd560fb0d019 P 78c2c46c7520492db35d51c244fb4f74: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260130 02:58:37.380426 21554 ts_tablet_manager.cc:1916] T 345eaaa5625c48d099280b7fbb2e514b P d870164263bb44b6b780a557daea3656: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260130 02:58:37.425280 21554 ts_tablet_manager.cc:1929] T 345eaaa5625c48d099280b7fbb2e514b P d870164263bb44b6b780a557daea3656: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260130 02:58:37.425762 21554 log.cc:1199] T 345eaaa5625c48d099280b7fbb2e514b P d870164263bb44b6b780a557daea3656: Deleting WAL directory at /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-2/wal/wals/345eaaa5625c48d099280b7fbb2e514b
I20260130 02:58:37.426821 21554 ts_tablet_manager.cc:1950] T 345eaaa5625c48d099280b7fbb2e514b P d870164263bb44b6b780a557daea3656: Deleting consensus metadata
I20260130 02:58:37.429387 18860 catalog_manager.cc:4985] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571): tablet 345eaaa5625c48d099280b7fbb2e514b (table default.Table_c1886dd9a7934e7797e0e7e0b065d692 [id=7a2d5367224b492b844199088b01b149]) successfully deleted
I20260130 02:58:37.471882 21565 ts_tablet_manager.cc:1929] T 111287b5557d4abda1c5cd560fb0d019 P 78c2c46c7520492db35d51c244fb4f74: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260130 02:58:37.473379 21565 log.cc:1199] T 111287b5557d4abda1c5cd560fb0d019 P 78c2c46c7520492db35d51c244fb4f74: Deleting WAL directory at /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1769741859011789-18833-0/minicluster-data/ts-1/wal/wals/111287b5557d4abda1c5cd560fb0d019
I20260130 02:58:37.479509 19358 tablet_service.cc:1505] Processing CreateTablet for tablet c2c927e461f04fb48bfa9e3a685b077d (DEFAULT_TABLE table=default.Table_aa496bf2019748f29823a9a253b1f3ca [id=780188e2335f43fbb11de8212eb38d56]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:37.480675 19358 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c2c927e461f04fb48bfa9e3a685b077d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:37.488857 19351 tablet_service.cc:1505] Processing CreateTablet for tablet aedf72892b7248e2ace8772283c2b476 (DEFAULT_TABLE table=default.Table_203b3a960d3e4b5cb978b6168435f3a4 [id=ffe264c47e31462a8deaf81a8b96d34c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:37.488976 19505 tablet_service.cc:1505] Processing CreateTablet for tablet c2c927e461f04fb48bfa9e3a685b077d (DEFAULT_TABLE table=default.Table_aa496bf2019748f29823a9a253b1f3ca [id=780188e2335f43fbb11de8212eb38d56]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:37.488848 19360 tablet_service.cc:1505] Processing CreateTablet for tablet dc4bdd8e4f104d4a86c0af6ac6accda0 (DEFAULT_TABLE table=default.Table_ba19837ced4c45d79be506e93308449b [id=e732719fbe544e38a15b35017010c7b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:37.490044 19351 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aedf72892b7248e2ace8772283c2b476. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:37.490044 19505 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c2c927e461f04fb48bfa9e3a685b077d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:37.490509 19360 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dc4bdd8e4f104d4a86c0af6ac6accda0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:37.492369 19496 tablet_service.cc:1505] Processing CreateTablet for tablet aedf72892b7248e2ace8772283c2b476 (DEFAULT_TABLE table=default.Table_203b3a960d3e4b5cb978b6168435f3a4 [id=ffe264c47e31462a8deaf81a8b96d34c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:37.493366 19496 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aedf72892b7248e2ace8772283c2b476. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:37.519641 19501 tablet_service.cc:1505] Processing CreateTablet for tablet dc4bdd8e4f104d4a86c0af6ac6accda0 (DEFAULT_TABLE table=default.Table_ba19837ced4c45d79be506e93308449b [id=e732719fbe544e38a15b35017010c7b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:37.520681 19501 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dc4bdd8e4f104d4a86c0af6ac6accda0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:37.524379 19349 tablet_service.cc:1505] Processing CreateTablet for tablet 47f574674aa7414588605ca38eb7200a (DEFAULT_TABLE table=default.Table_d2a3438ed66741eab245ae6d6c5c2f37 [id=4fb5c26265e94083a8714be512fcd222]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:37.524813 21565 ts_tablet_manager.cc:1950] T 111287b5557d4abda1c5cd560fb0d019 P 78c2c46c7520492db35d51c244fb4f74: Deleting consensus metadata
I20260130 02:58:37.525520 19349 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 47f574674aa7414588605ca38eb7200a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:37.532512 18860 catalog_manager.cc:4985] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767): tablet 111287b5557d4abda1c5cd560fb0d019 (table default.Table_fec49c37fdd045e7801325ace326f367 [id=06f1da4d6e0e46e4a141914a3aa68c07]) successfully deleted
I20260130 02:58:37.539978 19427 heartbeater.cc:499] Master 127.18.100.126:38191 was elected leader, sending a full tablet report...
I20260130 02:58:37.542749 19503 tablet_service.cc:1505] Processing CreateTablet for tablet 47f574674aa7414588605ca38eb7200a (DEFAULT_TABLE table=default.Table_d2a3438ed66741eab245ae6d6c5c2f37 [id=4fb5c26265e94083a8714be512fcd222]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:37.550689 19503 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 47f574674aa7414588605ca38eb7200a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:37.582440 21584 tablet_bootstrap.cc:492] T dc4bdd8e4f104d4a86c0af6ac6accda0 P 78c2c46c7520492db35d51c244fb4f74: Bootstrap starting.
I20260130 02:58:37.607337 21565 tablet_replica.cc:333] T 345eaaa5625c48d099280b7fbb2e514b P 78c2c46c7520492db35d51c244fb4f74: stopping tablet replica
I20260130 02:58:37.608103 19349 tablet_service.cc:1505] Processing CreateTablet for tablet 3379641837df4b6ea360e8e27dfe4f8b (DEFAULT_TABLE table=default.Table_8f632ff84d624b72a10cb90fb9e78636 [id=3cf79022ea954f6397f0a0497e6e6a6c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:37.609163 19349 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3379641837df4b6ea360e8e27dfe4f8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:37.627045 21585 tablet_bootstrap.cc:492] T dc4bdd8e4f104d4a86c0af6ac6accda0 P d870164263bb44b6b780a557daea3656: Bootstrap starting.
I20260130 02:58:37.629310 19496 tablet_service.cc:1505] Processing CreateTablet for tablet 3379641837df4b6ea360e8e27dfe4f8b (DEFAULT_TABLE table=default.Table_8f632ff84d624b72a10cb90fb9e78636 [id=3cf79022ea954f6397f0a0497e6e6a6c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:37.630329 19496 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3379641837df4b6ea360e8e27dfe4f8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:37.646894 19360 tablet_service.cc:1505] Processing CreateTablet for tablet 6ad6bb8c986a433d94c745f2b47266ff (DEFAULT_TABLE table=default.Table_bc168ff8b5f34cacbd398117647ba25a [id=2a21ee52b7d246f1b44eda02bf5a411e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:37.648687 19501 tablet_service.cc:1505] Processing CreateTablet for tablet 6ad6bb8c986a433d94c745f2b47266ff (DEFAULT_TABLE table=default.Table_bc168ff8b5f34cacbd398117647ba25a [id=2a21ee52b7d246f1b44eda02bf5a411e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:58:37.650317 19501 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6ad6bb8c986a433d94c745f2b47266ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:37.651813 19360 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6ad6bb8c986a433d94c745f2b47266ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:37.661185 21584 tablet_bootstrap.cc:654] T dc4bdd8e4f104d4a86c0af6ac6accda0 P 78c2c46c7520492db35d51c244fb4f74: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:37.708613 21585 tablet_bootstrap.cc:654] T dc4bdd8e4f104d4a86c0af6ac6accda0 P d870164263bb44b6b780a557daea3656: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:37.721624 21565 raft_consensus.cc:2243] T 345eaaa5625c48d099280b7fbb2e514b P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260130 02:58:37.722267 21565 raft_consensus.cc:2272] T 345eaaa5625c48d099280b7fbb2e514b P 78c2c46c7520492db35d51c244fb4f74 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260130 02:58:37.724391 21565 ts_tablet_manager.cc:1916] T 345eaaa5625c48d099280b7fbb2e514b P 78c2c46c7520492db35d51c244fb4f74: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260130 02:58:37.727475 21584 tablet_bootstrap.cc:492] T dc4bdd8e4f104d4a86c0af6ac6accda0 P 78c2c46c7520492db35d51c244fb4f74: No bootstrap required, opened a new log
I20260130 02:58:37.727926 21584 ts_tablet_manager.cc:1403] T dc4bdd8e4f104d4a86c0af6ac6accda0 P 78c2c46c7520492db35d51c244fb4f74: Time spent bootstrapping tablet: real 0.146s user 0.011s sys 0.000s
I20260130 02:58:37.739041 21584 raft_consensus.cc:359] T dc4bdd8e4f104d4a86c0af6ac6accda0 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:37.739714 21584 raft_consensus.cc:385] T dc4bdd8e4f104d4a86c0af6ac6accda0 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:37.739988 21584 raft_consensus.cc:740] T dc4bdd8e4f104d4a86c0af6ac6accda0 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78c2c46c7520492db35d51c244fb4f74, State: Initialized, Role: FOLLOWER
I20260130 02:58:37.740720 21584 consensus_queue.cc:260] T dc4bdd8e4f104d4a86c0af6ac6accda0 P 78c2c46c7520492db35d51c244fb4f74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:37.749739 18870 catalog_manager.cc:5654] T c76f0dc7762c49c389fa6fef41f02141 P d870164263bb44b6b780a557daea3656 reported cstate change: term changed from 0 to 1, leader changed from <none> to 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66). New cstate: current_term: 1 leader_uuid: "78c2c46c7520492db35d51c244fb4f74" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } }
I20260130 02:58:37.758397 21584 ts_tablet_manager.cc:1434] T dc4bdd8e4f104d4a86c0af6ac6accda0 P 78c2c46c7520492db35d51c244fb4f74: Time spent starting tablet: real 0.028s user 0.002s sys 0.004s
I20260130 02:58:37.763417 18870 catalog_manager.cc:5746] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571) does not have the latest schema for tablet dc09a0f0cecf49738bab04596944c0f3 (table default.Table_32f4a0f63b1046529836267ff3f551fa [id=676ece9dc36b418a8b1400e8406d6b98]). Expected version 1 got 0
I20260130 02:58:37.767309 18870 catalog_manager.cc:5654] T aae68390dbbe4dc58ce3d3bd01725ae6 P d870164263bb44b6b780a557daea3656 reported cstate change: term changed from 0 to 1, leader changed from <none> to 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66). New cstate: current_term: 1 leader_uuid: "78c2c46c7520492db35d51c244fb4f74" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } }
I20260130 02:58:37.772746 21584 tablet_bootstrap.cc:492] T c2c927e461f04fb48bfa9e3a685b077d P 78c2c46c7520492db35d51c244fb4f74: Bootstrap starting.
I20260130 02:58:37.775471 18870 catalog_manager.cc:5654] T f7607df5efd543438fb2dff3667cc8c8 P d870164263bb44b6b780a557daea3656 reported cstate change: term changed from 0 to 1, leader changed from <none> to 371a36d219164974abdf848841528152 (127.18.100.65). New cstate: current_term: 1 leader_uuid: "371a36d219164974abdf848841528152" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } }
I20260130 02:58:37.781512 18870 catalog_manager.cc:5654] T 8b47653c7c3945a68e9bf4680bfc0496 P d870164263bb44b6b780a557daea3656 reported cstate change: term changed from 0 to 1, leader changed from <none> to d870164263bb44b6b780a557daea3656 (127.18.100.67). New cstate: current_term: 1 leader_uuid: "d870164263bb44b6b780a557daea3656" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } health_report { overall_health: HEALTHY } } }
I20260130 02:58:37.788736 18870 catalog_manager.cc:5746] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571) does not have the latest schema for tablet 8b47653c7c3945a68e9bf4680bfc0496 (table default.Table_1ae68ae3f0404843aca3c72f02678ac6 [id=8a9182f49e18489fafec8dca190c1b2f]). Expected version 1 got 0
I20260130 02:58:37.790441 18870 catalog_manager.cc:5654] T 306452e208ca4e719609f67345114a13 P d870164263bb44b6b780a557daea3656 reported cstate change: term changed from 0 to 1, leader changed from <none> to 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66). New cstate: current_term: 1 leader_uuid: "78c2c46c7520492db35d51c244fb4f74" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } }
I20260130 02:58:37.806025 21584 tablet_bootstrap.cc:654] T c2c927e461f04fb48bfa9e3a685b077d P 78c2c46c7520492db35d51c244fb4f74: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:37.793922 18870 catalog_manager.cc:5654] T 39bcc41f10ae4241b370529364454586 P d870164263bb44b6b780a557daea3656 reported cstate change: term changed from 0 to 1, leader changed from <none> to d870164263bb44b6b780a557daea3656 (127.18.100.67). New cstate: current_term: 1 leader_uuid: "d870164263bb44b6b780a557daea3656" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } health_report { overall_health: HEALTHY } } }
I20260130 02:58:37.813027 18870 catalog_manager.cc:5654] T 2b0edaad6877460cb7982efc62b00b66 P d870164263bb44b6b780a557daea3656 reported cstate change: term changed from 0 to 1, leader changed from <none> to d870164263bb44b6b780a557daea3656 (127.18.100.67). New cstate: current_term: 1 leader_uuid: "d870164263bb44b6b780a557daea3656" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } health_report { overall_health: HEALTHY } } }
I20260130 02:58:37.821022 21585 tablet_bootstrap.cc:492] T dc4bdd8e4f104d4a86c0af6ac6accda0 P d870164263bb44b6b780a557daea3656: No bootstrap required, opened a new log
I20260130 02:58:37.821514 21585 ts_tablet_manager.cc:1403] T dc4bdd8e4f104d4a86c0af6ac6accda0 P d870164263bb44b6b780a557daea3656: Time spent bootstrapping tablet: real 0.195s user 0.020s sys 0.009s
I20260130 02:58:37.824186 21585 raft_consensus.cc:359] T dc4bdd8e4f104d4a86c0af6ac6accda0 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:37.819365 18870 catalog_manager.cc:5746] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571) does not have the latest schema for tablet 2b0edaad6877460cb7982efc62b00b66 (table default.Table_d95381c590c94802bdc83acef0f0df96 [id=64a8e0e079f34806aa68795f3d548c5e]). Expected version 1 got 0
I20260130 02:58:37.831681 18870 catalog_manager.cc:5654] T 133e9bfe175f44aab1a53114abe67e4d P d870164263bb44b6b780a557daea3656 reported cstate change: term changed from 0 to 1, leader changed from <none> to d870164263bb44b6b780a557daea3656 (127.18.100.67). New cstate: current_term: 1 leader_uuid: "d870164263bb44b6b780a557daea3656" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } health_report { overall_health: HEALTHY } } }
I20260130 02:58:37.832755 21585 raft_consensus.cc:385] T dc4bdd8e4f104d4a86c0af6ac6accda0 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:37.833101 21585 raft_consensus.cc:740] T dc4bdd8e4f104d4a86c0af6ac6accda0 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d870164263bb44b6b780a557daea3656, State: Initialized, Role: FOLLOWER
I20260130 02:58:37.834218 21585 consensus_queue.cc:260] T dc4bdd8e4f104d4a86c0af6ac6accda0 P d870164263bb44b6b780a557daea3656 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
W20260130 02:58:37.848994 19463 connection.cc:537] server connection from 127.18.100.65:38175 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260130 02:58:37.858616 21584 tablet_bootstrap.cc:492] T c2c927e461f04fb48bfa9e3a685b077d P 78c2c46c7520492db35d51c244fb4f74: No bootstrap required, opened a new log
I20260130 02:58:37.859158 21584 ts_tablet_manager.cc:1403] T c2c927e461f04fb48bfa9e3a685b077d P 78c2c46c7520492db35d51c244fb4f74: Time spent bootstrapping tablet: real 0.087s user 0.011s sys 0.005s
I20260130 02:58:37.863938 21583 raft_consensus.cc:493] T dc4bdd8e4f104d4a86c0af6ac6accda0 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:37.864360 21583 raft_consensus.cc:515] T dc4bdd8e4f104d4a86c0af6ac6accda0 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:37.866191 21583 leader_election.cc:290] T dc4bdd8e4f104d4a86c0af6ac6accda0 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767)
I20260130 02:58:37.867650 19371 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc4bdd8e4f104d4a86c0af6ac6accda0" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74" is_pre_election: true
I20260130 02:58:37.868220 19371 raft_consensus.cc:2468] T dc4bdd8e4f104d4a86c0af6ac6accda0 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d870164263bb44b6b780a557daea3656 in term 0.
I20260130 02:58:37.868460 21585 ts_tablet_manager.cc:1434] T dc4bdd8e4f104d4a86c0af6ac6accda0 P d870164263bb44b6b780a557daea3656: Time spent starting tablet: real 0.047s user 0.006s sys 0.000s
I20260130 02:58:37.861624 21584 raft_consensus.cc:359] T c2c927e461f04fb48bfa9e3a685b077d P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:37.869464 19462 leader_election.cc:304] T dc4bdd8e4f104d4a86c0af6ac6accda0 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78c2c46c7520492db35d51c244fb4f74, d870164263bb44b6b780a557daea3656; no voters:
I20260130 02:58:37.874195 21583 raft_consensus.cc:2804] T dc4bdd8e4f104d4a86c0af6ac6accda0 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260130 02:58:37.874513 21583 raft_consensus.cc:493] T dc4bdd8e4f104d4a86c0af6ac6accda0 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260130 02:58:37.874886 21583 raft_consensus.cc:3060] T dc4bdd8e4f104d4a86c0af6ac6accda0 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:37.877027 21584 raft_consensus.cc:385] T c2c927e461f04fb48bfa9e3a685b077d P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:37.877317 21584 raft_consensus.cc:740] T c2c927e461f04fb48bfa9e3a685b077d P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78c2c46c7520492db35d51c244fb4f74, State: Initialized, Role: FOLLOWER
I20260130 02:58:37.878065 18870 catalog_manager.cc:5654] T 94355fc1338f479e8798070aff04fcba P d870164263bb44b6b780a557daea3656 reported cstate change: term changed from 0 to 2, leader changed from <none> to d870164263bb44b6b780a557daea3656 (127.18.100.67). New cstate: current_term: 2 leader_uuid: "d870164263bb44b6b780a557daea3656" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } health_report { overall_health: HEALTHY } } }
I20260130 02:58:37.879338 21585 tablet_bootstrap.cc:492] T aedf72892b7248e2ace8772283c2b476 P d870164263bb44b6b780a557daea3656: Bootstrap starting.
I20260130 02:58:37.879850 21584 consensus_queue.cc:260] T c2c927e461f04fb48bfa9e3a685b077d P 78c2c46c7520492db35d51c244fb4f74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:37.881824 21583 raft_consensus.cc:515] T dc4bdd8e4f104d4a86c0af6ac6accda0 P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:37.882431 18870 catalog_manager.cc:5654] T e34d5da0ad864b2eaea0d575c982e447 P d870164263bb44b6b780a557daea3656 reported cstate change: term changed from 0 to 1, leader changed from <none> to d870164263bb44b6b780a557daea3656 (127.18.100.67). New cstate: current_term: 1 leader_uuid: "d870164263bb44b6b780a557daea3656" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } health_report { overall_health: HEALTHY } } }
I20260130 02:58:37.903121 21583 leader_election.cc:290] T dc4bdd8e4f104d4a86c0af6ac6accda0 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 election: Requested vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767)
W20260130 02:58:37.904943 19463 connection.cc:537] client connection to 127.18.100.65:42021 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260130 02:58:37.905367 19463 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260130 02:58:37.906728 19371 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc4bdd8e4f104d4a86c0af6ac6accda0" candidate_uuid: "d870164263bb44b6b780a557daea3656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c2c46c7520492db35d51c244fb4f74"
I20260130 02:58:37.907241 19371 raft_consensus.cc:3060] T dc4bdd8e4f104d4a86c0af6ac6accda0 P 78c2c46c7520492db35d51c244fb4f74 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:58:37.910657 18833 external_mini_cluster.cc:1658] Killing /tmp/dist-test-tasktGe2wI/build/tsan/bin/kudu with pid 19279
W20260130 02:58:37.903911 18861 connection.cc:537] client connection to 127.18.100.65:42021 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260130 02:58:37.912396 18861 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107) [suppressed 1 similar messages]
I20260130 02:58:37.919178 18870 catalog_manager.cc:5654] T 1fc8f09d6329407c8078542b4123ef4d P d870164263bb44b6b780a557daea3656 reported cstate change: term changed from 0 to 1, leader changed from <none> to 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66). New cstate: current_term: 1 leader_uuid: "78c2c46c7520492db35d51c244fb4f74" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } }
I20260130 02:58:37.934734 18870 catalog_manager.cc:5654] T bc04be1284414a8699d3c4b88bf62813 P d870164263bb44b6b780a557daea3656 reported cstate change: term changed from 0 to 1, leader changed from <none> to 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66). New cstate: current_term: 1 leader_uuid: "78c2c46c7520492db35d51c244fb4f74" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } }
I20260130 02:58:37.942704 21585 tablet_bootstrap.cc:654] T aedf72892b7248e2ace8772283c2b476 P d870164263bb44b6b780a557daea3656: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:37.943702 18870 catalog_manager.cc:5654] T 3738385130c044e0a5a5803da74c567d P d870164263bb44b6b780a557daea3656 reported cstate change: term changed from 0 to 1, leader changed from <none> to 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66). New cstate: current_term: 1 leader_uuid: "78c2c46c7520492db35d51c244fb4f74" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } }
I20260130 02:58:37.951776 18870 catalog_manager.cc:5654] T 320dbe98a32a43b9899bbeed80b20f55 P d870164263bb44b6b780a557daea3656 reported cstate change: term changed from 0 to 1, leader changed from <none> to 371a36d219164974abdf848841528152 (127.18.100.65). New cstate: current_term: 1 leader_uuid: "371a36d219164974abdf848841528152" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } }
W20260130 02:58:37.950731 18861 catalog_manager.cc:4712] TS 371a36d219164974abdf848841528152 (127.18.100.65:42021): CreateTablet RPC failed for tablet dc4bdd8e4f104d4a86c0af6ac6accda0: Network error: Client connection negotiation failed: client connection to 127.18.100.65:42021: connect: Connection refused (error 111)
I20260130 02:58:37.956375 21593 raft_consensus.cc:493] T 00000000000000000000000000000000 P 3749baa287ed4f249329a74071781edd [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 658299197a424f53a37f14fadf089ae4)
I20260130 02:58:37.960852 21591 raft_consensus.cc:493] T 00000000000000000000000000000000 P 107f274cdda543038832f06a56ab9130 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 658299197a424f53a37f14fadf089ae4)
I20260130 02:58:37.961311 21591 raft_consensus.cc:515] T 00000000000000000000000000000000 P 107f274cdda543038832f06a56ab9130 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "658299197a424f53a37f14fadf089ae4" member_type: VOTER last_known_addr { host: "127.18.100.126" port: 38191 } } peers { permanent_uuid: "107f274cdda543038832f06a56ab9130" member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39711 } } peers { permanent_uuid: "3749baa287ed4f249329a74071781edd" member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34629 } }
I20260130 02:58:37.963065 21593 raft_consensus.cc:515] T 00000000000000000000000000000000 P 3749baa287ed4f249329a74071781edd [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "658299197a424f53a37f14fadf089ae4" member_type: VOTER last_known_addr { host: "127.18.100.126" port: 38191 } } peers { permanent_uuid: "107f274cdda543038832f06a56ab9130" member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39711 } } peers { permanent_uuid: "3749baa287ed4f249329a74071781edd" member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34629 } }
I20260130 02:58:37.966900 18883 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3749baa287ed4f249329a74071781edd" candidate_term: 9 candidate_status { last_received { term: 8 index: 385 } } ignore_live_leader: false dest_uuid: "658299197a424f53a37f14fadf089ae4" is_pre_election: true
I20260130 02:58:37.967123 18870 catalog_manager.cc:5654] T 657e9b0d8c05413f904129dce745710b P d870164263bb44b6b780a557daea3656 reported cstate change: term changed from 0 to 1, leader changed from <none> to d870164263bb44b6b780a557daea3656 (127.18.100.67). New cstate: current_term: 1 leader_uuid: "d870164263bb44b6b780a557daea3656" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } health_report { overall_health: HEALTHY } } }
I20260130 02:58:37.968240 18870 catalog_manager.cc:5746] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571) does not have the latest schema for tablet 657e9b0d8c05413f904129dce745710b (table default.Table_526677c8ef4144bd9de7a54a5e78e2be [id=b16908b75587459597a978c71756451c]). Expected version 1 got 0
W20260130 02:58:37.967548 19463 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.100.65:42021, user_credentials={real_user=slave}} blocked reactor thread for 62235us
I20260130 02:58:37.969208 18870 catalog_manager.cc:5654] T 0855d88806914a36b4bf6b49036f0ad0 P d870164263bb44b6b780a557daea3656 reported cstate change: term changed from 0 to 1, leader changed from <none> to d870164263bb44b6b780a557daea3656 (127.18.100.67). New cstate: current_term: 1 leader_uuid: "d870164263bb44b6b780a557daea3656" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } health_report { overall_health: HEALTHY } } }
I20260130 02:58:37.976948 21593 leader_election.cc:290] T 00000000000000000000000000000000 P 3749baa287ed4f249329a74071781edd [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 658299197a424f53a37f14fadf089ae4 (127.18.100.126:38191), 107f274cdda543038832f06a56ab9130 (127.18.100.125:39711)
I20260130 02:58:37.977967 18960 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3749baa287ed4f249329a74071781edd" candidate_term: 9 candidate_status { last_received { term: 8 index: 385 } } ignore_live_leader: false dest_uuid: "107f274cdda543038832f06a56ab9130" is_pre_election: true
I20260130 02:58:37.981317 20627 leader_election.cc:304] T 00000000000000000000000000000000 P 3749baa287ed4f249329a74071781edd [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3749baa287ed4f249329a74071781edd; no voters: 107f274cdda543038832f06a56ab9130, 658299197a424f53a37f14fadf089ae4
I20260130 02:58:37.986637 21593 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 3749baa287ed4f249329a74071781edd [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260130 02:58:37.991216 20807 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "107f274cdda543038832f06a56ab9130" candidate_term: 9 candidate_status { last_received { term: 8 index: 385 } } ignore_live_leader: false dest_uuid: "3749baa287ed4f249329a74071781edd" is_pre_election: true
I20260130 02:58:37.992893 21591 leader_election.cc:290] T 00000000000000000000000000000000 P 107f274cdda543038832f06a56ab9130 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 658299197a424f53a37f14fadf089ae4 (127.18.100.126:38191), 3749baa287ed4f249329a74071781edd (127.18.100.124:34629)
W20260130 02:58:38.000984 19463 consensus_peers.cc:597] T f9e5f01e69254d658c369f3e21cef523 P d870164263bb44b6b780a557daea3656 -> Peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Couldn't send request to peer 371a36d219164974abdf848841528152. Status: Network error: Client connection negotiation failed: client connection to 127.18.100.65:42021: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260130 02:58:38.001420 19463 consensus_peers.cc:597] T 9896ff9364744857968e87e80e776dd2 P d870164263bb44b6b780a557daea3656 -> Peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Couldn't send request to peer 371a36d219164974abdf848841528152. Status: Network error: Client connection negotiation failed: client connection to 127.18.100.65:42021: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260130 02:58:38.001786 19463 consensus_peers.cc:597] T 6d1f121f49cd47c2b8247daaf6874e01 P d870164263bb44b6b780a557daea3656 -> Peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Couldn't send request to peer 371a36d219164974abdf848841528152. Status: Network error: Client connection negotiation failed: client connection to 127.18.100.65:42021: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260130 02:58:38.002137 19463 consensus_peers.cc:597] T 8b47653c7c3945a68e9bf4680bfc0496 P d870164263bb44b6b780a557daea3656 -> Peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Couldn't send request to peer 371a36d219164974abdf848841528152. Status: Network error: Client connection negotiation failed: client connection to 127.18.100.65:42021: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260130 02:58:37.994916 18883 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "107f274cdda543038832f06a56ab9130" candidate_term: 9 candidate_status { last_received { term: 8 index: 385 } } ignore_live_leader: false dest_uuid: "658299197a424f53a37f14fadf089ae4" is_pre_election: true
W20260130 02:58:38.002470 19463 consensus_peers.cc:597] T 155ff0c35fc44cb292a28407e6ed937d P d870164263bb44b6b780a557daea3656 -> Peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Couldn't send request to peer 371a36d219164974abdf848841528152. Status: Network error: Client connection negotiation failed: client connection to 127.18.100.65:42021: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260130 02:58:38.002838 19463 consensus_peers.cc:597] T dd377fab8c544bf69121dc9e9a060b29 P d870164263bb44b6b780a557daea3656 -> Peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Couldn't send request to peer 371a36d219164974abdf848841528152. Status: Network error: Client connection negotiation failed: client connection to 127.18.100.65:42021: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260130 02:58:38.003156 19463 consensus_peers.cc:597] T eb7a755ab62b4552bb8588264c5f7b75 P d870164263bb44b6b780a557daea3656 -> Peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Couldn't send request to peer 371a36d219164974abdf848841528152. Status: Network error: Client connection negotiation failed: client connection to 127.18.100.65:42021: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260130 02:58:38.003471 19463 consensus_peers.cc:597] T 1129f72158ec4bae89da3c94cdb70839 P d870164263bb44b6b780a557daea3656 -> Peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Couldn't send request to peer 371a36d219164974abdf848841528152. Status: Network error: Client connection negotiation failed: client connection to 127.18.100.65:42021: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260130 02:58:38.003798 19463 consensus_peers.cc:597] T 94355fc1338f479e8798070aff04fcba P d870164263bb44b6b780a557daea3656 -> Peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Couldn't send request to peer 371a36d219164974abdf848841528152. Status: Network error: Client connection negotiation failed: client connection to 127.18.100.65:42021: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260130 02:58:38.003758 18934 leader_election.cc:304] T 00000000000000000000000000000000 P 107f274cdda543038832f06a56ab9130 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 107f274cdda543038832f06a56ab9130; no voters: 3749baa287ed4f249329a74071781edd, 658299197a424f53a37f14fadf089ae4
I20260130 02:58:38.006253 21585 tablet_bootstrap.cc:492] T aedf72892b7248e2ace8772283c2b476 P d870164263bb44b6b780a557daea3656: No bootstrap required, opened a new log
I20260130 02:58:38.006706 21591 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 107f274cdda543038832f06a56ab9130 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260130 02:58:38.007269 21585 ts_tablet_manager.cc:1403] T aedf72892b7248e2ace8772283c2b476 P d870164263bb44b6b780a557daea3656: Time spent bootstrapping tablet: real 0.128s user 0.003s sys 0.009s
I20260130 02:58:38.010113 21585 raft_consensus.cc:359] T aedf72892b7248e2ace8772283c2b476 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:37.970547 18870 catalog_manager.cc:5654] T 13f0ace4d9de43abab08eca32cdb30c1 P d870164263bb44b6b780a557daea3656 reported cstate change: term changed from 0 to 1, leader changed from <none> to 371a36d219164974abdf848841528152 (127.18.100.65). New cstate: current_term: 1 leader_uuid: "371a36d219164974abdf848841528152" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } }
I20260130 02:58:38.017956 18870 catalog_manager.cc:5654] T cd30d7adcc4545848e3501a205a10293 P d870164263bb44b6b780a557daea3656 reported cstate change: term changed from 0 to 1, leader changed from <none> to 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66). New cstate: current_term: 1 leader_uuid: "78c2c46c7520492db35d51c244fb4f74" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } }
I20260130 02:58:38.019583 18870 catalog_manager.cc:5654] T ca06f07bb98f45a7bf63aebc28348e08 P d870164263bb44b6b780a557daea3656 reported cstate change: term changed from 0 to 2, leader changed from <none> to 371a36d219164974abdf848841528152 (127.18.100.65). New cstate: current_term: 2 leader_uuid: "371a36d219164974abdf848841528152" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } }
W20260130 02:58:38.019229 19463 consensus_peers.cc:597] T 30cc52136bb3422895d95c0fed0c6798 P d870164263bb44b6b780a557daea3656 -> Peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Couldn't send request to peer 371a36d219164974abdf848841528152. Status: Network error: Client connection negotiation failed: client connection to 127.18.100.65:42021: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260130 02:58:38.020536 21585 raft_consensus.cc:385] T aedf72892b7248e2ace8772283c2b476 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20260130 02:58:38.020615 19463 consensus_peers.cc:597] T 133e9bfe175f44aab1a53114abe67e4d P d870164263bb44b6b780a557daea3656 -> Peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Couldn't send request to peer 371a36d219164974abdf848841528152. Status: Network error: Client connection negotiation failed: client connection to 127.18.100.65:42021: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260130 02:58:38.020958 21585 raft_consensus.cc:740] T aedf72892b7248e2ace8772283c2b476 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d870164263bb44b6b780a557daea3656, State: Initialized, Role: FOLLOWER
W20260130 02:58:38.021028 19463 consensus_peers.cc:597] T 6e12a8eeb3204325a9a8b7779f72ab29 P d870164263bb44b6b780a557daea3656 -> Peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Couldn't send request to peer 371a36d219164974abdf848841528152. Status: Network error: Client connection negotiation failed: client connection to 127.18.100.65:42021: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260130 02:58:38.021476 19463 consensus_peers.cc:597] T 2c79a3aecab844698e70834999e1caac P d870164263bb44b6b780a557daea3656 -> Peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Couldn't send request to peer 371a36d219164974abdf848841528152. Status: Network error: Client connection negotiation failed: client connection to 127.18.100.65:42021: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260130 02:58:38.021793 19463 consensus_peers.cc:597] T 3aa76993beb94b12bc706a76db26a03b P d870164263bb44b6b780a557daea3656 -> Peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Couldn't send request to peer 371a36d219164974abdf848841528152. Status: Network error: Client connection negotiation failed: client connection to 127.18.100.65:42021: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260130 02:58:38.022152 19463 consensus_peers.cc:597] T e026dee22493424692d00f14848ab57c P d870164263bb44b6b780a557daea3656 -> Peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Couldn't send request to peer 371a36d219164974abdf848841528152. Status: Network error: Client connection negotiation failed: client connection to 127.18.100.65:42021: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260130 02:58:38.022470 19463 consensus_peers.cc:597] T 475b435b2967418f927fa172d2ff04ea P d870164263bb44b6b780a557daea3656 -> Peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Couldn't send request to peer 371a36d219164974abdf848841528152. Status: Network error: Client connection negotiation failed: client connection to 127.18.100.65:42021: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260130 02:58:38.022883 19463 consensus_peers.cc:597] T 2b0edaad6877460cb7982efc62b00b66 P d870164263bb44b6b780a557daea3656 -> Peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Couldn't send request to peer 371a36d219164974abdf848841528152. Status: Network error: Client connection negotiation failed: client connection to 127.18.100.65:42021: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260130 02:58:38.023502 19463 consensus_peers.cc:597] T 0d14fb95ace9447f810ccbed4d5befdf P d870164263bb44b6b780a557daea3656 -> Peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Couldn't send request to peer 371a36d219164974abdf848841528152. Status: Network error: Client connection negotiation failed: client connection to 127.18.100.65:42021: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260130 02:58:38.023860 19463 consensus_peers.cc:597] T e34d5da0ad864b2eaea0d575c982e447 P d870164263bb44b6b780a557daea3656 -> Peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Couldn't send request to peer 371a36d219164974abdf848841528152. Status: Network error: Client connection negotiation failed: client connection to 127.18.100.65:42021: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260130 02:58:38.024179 19463 consensus_peers.cc:597] T cfd12a72726846e587cff9d6a4d61bd6 P d870164263bb44b6b780a557daea3656 -> Peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Couldn't send request to peer 371a36d219164974abdf848841528152. Status: Network error: Client connection negotiation failed: client connection to 127.18.100.65:42021: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260130 02:58:38.021970 21585 consensus_queue.cc:260] T aedf72892b7248e2ace8772283c2b476 P d870164263bb44b6b780a557daea3656 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
W20260130 02:58:38.024498 19463 consensus_peers.cc:597] T 657e9b0d8c05413f904129dce745710b P d870164263bb44b6b780a557daea3656 -> Peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Couldn't send request to peer 371a36d219164974abdf848841528152. Status: Network error: Client connection negotiation failed: client connection to 127.18.100.65:42021: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260130 02:58:38.025396 19463 consensus_peers.cc:597] T 00ddbfe6218c44d788fbf65245a31dc6 P d870164263bb44b6b780a557daea3656 -> Peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Couldn't send request to peer 371a36d219164974abdf848841528152. Status: Network error: Client connection negotiation failed: client connection to 127.18.100.65:42021: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260130 02:58:38.027397 21585 ts_tablet_manager.cc:1434] T aedf72892b7248e2ace8772283c2b476 P d870164263bb44b6b780a557daea3656: Time spent starting tablet: real 0.019s user 0.006s sys 0.000s
I20260130 02:58:38.022986 18870 catalog_manager.cc:5654] T dfbbe9045e7c4aa5b520218687e7e4c1 P d870164263bb44b6b780a557daea3656 reported cstate change: term changed from 0 to 1, leader changed from <none> to d870164263bb44b6b780a557daea3656 (127.18.100.67). New cstate: current_term: 1 leader_uuid: "d870164263bb44b6b780a557daea3656" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } health_report { overall_health: HEALTHY } } }
I20260130 02:58:38.039490 18870 catalog_manager.cc:5654] T a3258852aed24ed582a79f77860e6f0d P d870164263bb44b6b780a557daea3656 reported cstate change: term changed from 0 to 1, leader changed from <none> to d870164263bb44b6b780a557daea3656 (127.18.100.67). New cstate: current_term: 1 leader_uuid: "d870164263bb44b6b780a557daea3656" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } health_report { overall_health: HEALTHY } } }
I20260130 02:58:38.051069 21585 tablet_bootstrap.cc:492] T 3379641837df4b6ea360e8e27dfe4f8b P d870164263bb44b6b780a557daea3656: Bootstrap starting.
I20260130 02:58:38.052011 18870 catalog_manager.cc:5654] T 49efcfced0b74c48962393149ac2a500 P d870164263bb44b6b780a557daea3656 reported cstate change: term changed from 0 to 1, leader changed from <none> to 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66). New cstate: current_term: 1 leader_uuid: "78c2c46c7520492db35d51c244fb4f74" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } }
W20260130 02:58:38.025780 19463 leader_election.cc:336] T dc4bdd8e4f104d4a86c0af6ac6accda0 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Network error: Client connection negotiation failed: client connection to 127.18.100.65:42021: connect: Connection refused (error 111)
I20260130 02:58:38.054951 18870 catalog_manager.cc:5746] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571) does not have the latest schema for tablet 49efcfced0b74c48962393149ac2a500 (table default.Table_2e81f0a0cbc449a397141b717db17025 [id=a9c919f57c0c437cb35a6921103e45a6]). Expected version 1 got 0
W20260130 02:58:38.055075 19463 consensus_peers.cc:597] T dfbbe9045e7c4aa5b520218687e7e4c1 P d870164263bb44b6b780a557daea3656 -> Peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Couldn't send request to peer 371a36d219164974abdf848841528152. Status: Network error: Client connection negotiation failed: client connection to 127.18.100.65:42021: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260130 02:58:38.055454 19463 leader_election.cc:336] T dc4bdd8e4f104d4a86c0af6ac6accda0 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Network error: Client connection negotiation failed: client connection to 127.18.100.65:42021: connect: Connection refused (error 111)
W20260130 02:58:38.055863 19463 consensus_peers.cc:597] T 91f954f3ef4244c9a575a7421a7e75eb P d870164263bb44b6b780a557daea3656 -> Peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Couldn't send request to peer 371a36d219164974abdf848841528152. Status: Network error: Client connection negotiation failed: client connection to 127.18.100.65:42021: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260130 02:58:38.056206 19463 consensus_peers.cc:597] T 03025d13f0e545cfa6288677a2715343 P d870164263bb44b6b780a557daea3656 -> Peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Couldn't send request to peer 371a36d219164974abdf848841528152. Status: Network error: Client connection negotiation failed: client connection to 127.18.100.65:42021: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260130 02:58:38.061929 19463 consensus_peers.cc:597] T f41f4cff7dd4489f948700a45e9a7057 P d870164263bb44b6b780a557daea3656 -> Peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Couldn't send request to peer 371a36d219164974abdf848841528152. Status: Network error: Client connection negotiation failed: client connection to 127.18.100.65:42021: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260130 02:58:38.062314 19463 consensus_peers.cc:597] T 39bcc41f10ae4241b370529364454586 P d870164263bb44b6b780a557daea3656 -> Peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Couldn't send request to peer 371a36d219164974abdf848841528152. Status: Network error: Client connection negotiation failed: client connection to 127.18.100.65:42021: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260130 02:58:38.069597 19463 consensus_peers.cc:597] T af0f90071a334016aa17fae2e3a97fc8 P d870164263bb44b6b780a557daea3656 -> Peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Couldn't send request to peer 371a36d219164974abdf848841528152. Status: Network error: Client connection negotiation failed: client connection to 127.18.100.65:42021: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260130 02:58:38.056907 18870 catalog_manager.cc:5654] T 6f5a90b75a0f4851b5db672c41253f7b P d870164263bb44b6b780a557daea3656 reported cstate change: term changed from 0 to 1, leader changed from <none> to 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66). New cstate: current_term: 1 leader_uuid: "78c2c46c7520492db35d51c244fb4f74" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } }
I20260130 02:58:38.072520 18870 catalog_manager.cc:5654] T eb7a755ab62b4552bb8588264c5f7b75 P d870164263bb44b6b780a557daea3656 reported cstate change: term changed from 0 to 1, leader changed from <none> to d870164263bb44b6b780a557daea3656 (127.18.100.67). New cstate: current_term: 1 leader_uuid: "d870164263bb44b6b780a557daea3656" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } health_report { overall_health: HEALTHY } } }
I20260130 02:58:38.074144 18870 catalog_manager.cc:5654] T 12c3f3ec523040e7b7cd356e3da57e0d P d870164263bb44b6b780a557daea3656 reported cstate change: term changed from 0 to 1, leader changed from <none> to 371a36d219164974abdf848841528152 (127.18.100.65). New cstate: current_term: 1 leader_uuid: "371a36d219164974abdf848841528152" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } }
W20260130 02:58:38.076874 19463 consensus_peers.cc:597] T f2d01bfdc9c4494386bd02bed655fe63 P d870164263bb44b6b780a557daea3656 -> Peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Couldn't send request to peer 371a36d219164974abdf848841528152. Status: Network error: Client connection negotiation failed: client connection to 127.18.100.65:42021: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260130 02:58:38.077251 19463 consensus_peers.cc:597] T 7caf7cbad5054396b5617094e4d536a0 P d870164263bb44b6b780a557daea3656 -> Peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Couldn't send request to peer 371a36d219164974abdf848841528152. Status: Network error: Client connection negotiation failed: client connection to 127.18.100.65:42021: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260130 02:58:38.077732 21585 tablet_bootstrap.cc:654] T 3379641837df4b6ea360e8e27dfe4f8b P d870164263bb44b6b780a557daea3656: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:38.090843 18870 catalog_manager.cc:5654] T cfd12a72726846e587cff9d6a4d61bd6 P d870164263bb44b6b780a557daea3656 reported cstate change: term changed from 0 to 1, leader changed from <none> to d870164263bb44b6b780a557daea3656 (127.18.100.67). New cstate: current_term: 1 leader_uuid: "d870164263bb44b6b780a557daea3656" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } health_report { overall_health: HEALTHY } } }
W20260130 02:58:38.090194 19463 consensus_peers.cc:597] T e691c238199b4914bcc3fd248f31420c P d870164263bb44b6b780a557daea3656 -> Peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Couldn't send request to peer 371a36d219164974abdf848841528152. Status: Network error: Client connection negotiation failed: client connection to 127.18.100.65:42021: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260130 02:58:38.097620 18870 catalog_manager.cc:5654] T e691c238199b4914bcc3fd248f31420c P d870164263bb44b6b780a557daea3656 reported cstate change: term changed from 0 to 1, leader changed from <none> to d870164263bb44b6b780a557daea3656 (127.18.100.67). New cstate: current_term: 1 leader_uuid: "d870164263bb44b6b780a557daea3656" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } health_report { overall_health: HEALTHY } } }
I20260130 02:58:38.100639 18870 catalog_manager.cc:5654] T 920f52b975ab480fbf42656171222f09 P d870164263bb44b6b780a557daea3656 reported cstate change: term changed from 0 to 2, leader changed from <none> to 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66). New cstate: current_term: 2 leader_uuid: "78c2c46c7520492db35d51c244fb4f74" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } }
I20260130 02:58:38.104465 18870 catalog_manager.cc:5654] T 6d1f121f49cd47c2b8247daaf6874e01 P d870164263bb44b6b780a557daea3656 reported cstate change: term changed from 0 to 1, leader changed from <none> to d870164263bb44b6b780a557daea3656 (127.18.100.67). New cstate: current_term: 1 leader_uuid: "d870164263bb44b6b780a557daea3656" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } health_report { overall_health: HEALTHY } } }
I20260130 02:58:38.107371 18870 catalog_manager.cc:5654] T 2b8034a92b2043e5b57cf02f1ee6296d P d870164263bb44b6b780a557daea3656 reported cstate change: term changed from 0 to 1, leader changed from <none> to d870164263bb44b6b780a557daea3656 (127.18.100.67). New cstate: current_term: 1 leader_uuid: "d870164263bb44b6b780a557daea3656" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } health_report { overall_health: HEALTHY } } }
I20260130 02:58:38.116636 21585 tablet_bootstrap.cc:492] T 3379641837df4b6ea360e8e27dfe4f8b P d870164263bb44b6b780a557daea3656: No bootstrap required, opened a new log
I20260130 02:58:38.117312 21585 ts_tablet_manager.cc:1403] T 3379641837df4b6ea360e8e27dfe4f8b P d870164263bb44b6b780a557daea3656: Time spent bootstrapping tablet: real 0.066s user 0.011s sys 0.000s
I20260130 02:58:38.119876 21585 raft_consensus.cc:359] T 3379641837df4b6ea360e8e27dfe4f8b P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
W20260130 02:58:38.120222 19463 consensus_peers.cc:597] T 66a7487a7ab343a4abca5312afe78862 P d870164263bb44b6b780a557daea3656 -> Peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Couldn't send request to peer 371a36d219164974abdf848841528152. Status: Network error: Client connection negotiation failed: client connection to 127.18.100.65:42021: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260130 02:58:38.131749 21585 raft_consensus.cc:385] T 3379641837df4b6ea360e8e27dfe4f8b P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:38.121176 18870 catalog_manager.cc:5654] T 66a7487a7ab343a4abca5312afe78862 P d870164263bb44b6b780a557daea3656 reported cstate change: term changed from 0 to 1, leader changed from <none> to d870164263bb44b6b780a557daea3656 (127.18.100.67). New cstate: current_term: 1 leader_uuid: "d870164263bb44b6b780a557daea3656" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } health_report { overall_health: HEALTHY } } }
I20260130 02:58:38.132899 18870 catalog_manager.cc:5654] T 00ddbfe6218c44d788fbf65245a31dc6 P d870164263bb44b6b780a557daea3656 reported cstate change: term changed from 0 to 1, leader changed from <none> to d870164263bb44b6b780a557daea3656 (127.18.100.67). New cstate: current_term: 1 leader_uuid: "d870164263bb44b6b780a557daea3656" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } health_report { overall_health: HEALTHY } } }
I20260130 02:58:38.133877 21585 raft_consensus.cc:740] T 3379641837df4b6ea360e8e27dfe4f8b P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d870164263bb44b6b780a557daea3656, State: Initialized, Role: FOLLOWER
I20260130 02:58:38.134708 21585 consensus_queue.cc:260] T 3379641837df4b6ea360e8e27dfe4f8b P d870164263bb44b6b780a557daea3656 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:38.136014 18870 catalog_manager.cc:5654] T 838e49e5e8874db393bdc15a1c92f87e P d870164263bb44b6b780a557daea3656 reported cstate change: term changed from 0 to 1, leader changed from <none> to 371a36d219164974abdf848841528152 (127.18.100.65). New cstate: current_term: 1 leader_uuid: "371a36d219164974abdf848841528152" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } }
I20260130 02:58:38.139541 21546 raft_consensus.cc:493] T aedf72892b7248e2ace8772283c2b476 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:38.138078 21585 ts_tablet_manager.cc:1434] T 3379641837df4b6ea360e8e27dfe4f8b P d870164263bb44b6b780a557daea3656: Time spent starting tablet: real 0.020s user 0.003s sys 0.003s
I20260130 02:58:38.139997 21546 raft_consensus.cc:515] T aedf72892b7248e2ace8772283c2b476 P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:38.141782 21546 leader_election.cc:290] T aedf72892b7248e2ace8772283c2b476 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767)
I20260130 02:58:38.142182 18870 catalog_manager.cc:5746] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571) does not have the latest schema for tablet 838e49e5e8874db393bdc15a1c92f87e (table default.Table_da3d581b15a94840898777f74611994e [id=b7e241ced53e4160ac6fa1f65ef3674e]). Expected version 1 got 0
W20260130 02:58:38.144811 19463 consensus_peers.cc:597] T 3b02480d9c6f4328af83f7940716d698 P d870164263bb44b6b780a557daea3656 -> Peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Couldn't send request to peer 371a36d219164974abdf848841528152. Status: Network error: Client connection negotiation failed: client connection to 127.18.100.65:42021: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260130 02:58:38.148607 19463 consensus_peers.cc:597] T 2b8034a92b2043e5b57cf02f1ee6296d P d870164263bb44b6b780a557daea3656 -> Peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Couldn't send request to peer 371a36d219164974abdf848841528152. Status: Network error: Client connection negotiation failed: client connection to 127.18.100.65:42021: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260130 02:58:38.149348 18870 catalog_manager.cc:5654] T 30cc52136bb3422895d95c0fed0c6798 P d870164263bb44b6b780a557daea3656 reported cstate change: term changed from 0 to 1, leader changed from <none> to d870164263bb44b6b780a557daea3656 (127.18.100.67). New cstate: current_term: 1 leader_uuid: "d870164263bb44b6b780a557daea3656" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } health_report { overall_health: HEALTHY } } }
W20260130 02:58:38.151479 19463 leader_election.cc:336] T aedf72892b7248e2ace8772283c2b476 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Network error: Client connection negotiation failed: client connection to 127.18.100.65:42021: connect: Connection refused (error 111)
I20260130 02:58:38.152621 18870 catalog_manager.cc:5654] T d480bf18140245e3870a62582d5a8318 P d870164263bb44b6b780a557daea3656 reported cstate change: term changed from 0 to 1, leader changed from <none> to 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66). New cstate: current_term: 1 leader_uuid: "78c2c46c7520492db35d51c244fb4f74" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } }
I20260130 02:58:38.156677 18870 catalog_manager.cc:5654] T 155ff0c35fc44cb292a28407e6ed937d P d870164263bb44b6b780a557daea3656 reported cstate change: term changed from 0 to 1, leader changed from <none> to d870164263bb44b6b780a557daea3656 (127.18.100.67). New cstate: current_term: 1 leader_uuid: "d870164263bb44b6b780a557daea3656" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } health_report { overall_health: HEALTHY } } }
I20260130 02:58:38.155059 21585 tablet_bootstrap.cc:492] T c2c927e461f04fb48bfa9e3a685b077d P d870164263bb44b6b780a557daea3656: Bootstrap starting.
W20260130 02:58:38.174775 19463 consensus_peers.cc:597] T 341041cb0be2456e88403367c0814704 P d870164263bb44b6b780a557daea3656 -> Peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Couldn't send request to peer 371a36d219164974abdf848841528152. Status: Network error: Client connection negotiation failed: client connection to 127.18.100.65:42021: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260130 02:58:38.175995 18870 catalog_manager.cc:5654] T 532e867fd5944d468d7672d2a5c92f3c P d870164263bb44b6b780a557daea3656 reported cstate change: term changed from 0 to 1, leader changed from <none> to 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66). New cstate: current_term: 1 leader_uuid: "78c2c46c7520492db35d51c244fb4f74" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } }
I20260130 02:58:38.178979 18870 catalog_manager.cc:5654] T 9896ff9364744857968e87e80e776dd2 P d870164263bb44b6b780a557daea3656 reported cstate change: term changed from 0 to 1, leader changed from <none> to d870164263bb44b6b780a557daea3656 (127.18.100.67). New cstate: current_term: 1 leader_uuid: "d870164263bb44b6b780a557daea3656" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } health_report { overall_health: HEALTHY } } }
I20260130 02:58:38.183698 21585 tablet_bootstrap.cc:654] T c2c927e461f04fb48bfa9e3a685b077d P d870164263bb44b6b780a557daea3656: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:38.184610 18870 catalog_manager.cc:5654] T 0d14fb95ace9447f810ccbed4d5befdf P d870164263bb44b6b780a557daea3656 reported cstate change: term changed from 0 to 1, leader changed from <none> to d870164263bb44b6b780a557daea3656 (127.18.100.67). New cstate: current_term: 1 leader_uuid: "d870164263bb44b6b780a557daea3656" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } health_report { overall_health: HEALTHY } } }
I20260130 02:58:38.194844 18870 catalog_manager.cc:5654] T dd377fab8c544bf69121dc9e9a060b29 P d870164263bb44b6b780a557daea3656 reported cstate change: term changed from 0 to 1, leader changed from <none> to d870164263bb44b6b780a557daea3656 (127.18.100.67). New cstate: current_term: 1 leader_uuid: "d870164263bb44b6b780a557daea3656" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } health_report { overall_health: HEALTHY } } }
I20260130 02:58:38.202988 18870 catalog_manager.cc:5654] T 475b435b2967418f927fa172d2ff04ea P d870164263bb44b6b780a557daea3656 reported cstate change: term changed from 0 to 1, leader changed from <none> to d870164263bb44b6b780a557daea3656 (127.18.100.67). New cstate: current_term: 1 leader_uuid: "d870164263bb44b6b780a557daea3656" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } health_report { overall_health: HEALTHY } } }
I20260130 02:58:38.206761 18870 catalog_manager.cc:5746] TS d870164263bb44b6b780a557daea3656 (127.18.100.67:39571) does not have the latest schema for tablet 475b435b2967418f927fa172d2ff04ea (table default.Table_dee034aec0b24982b8dc2a3fa1b326f1 [id=66033e75b3814a8584311cf7e4a07ad3]). Expected version 1 got 0
I20260130 02:58:38.223168 21585 tablet_bootstrap.cc:492] T c2c927e461f04fb48bfa9e3a685b077d P d870164263bb44b6b780a557daea3656: No bootstrap required, opened a new log
I20260130 02:58:38.223666 21585 ts_tablet_manager.cc:1403] T c2c927e461f04fb48bfa9e3a685b077d P d870164263bb44b6b780a557daea3656: Time spent bootstrapping tablet: real 0.069s user 0.007s sys 0.005s
I20260130 02:58:38.226217 21585 raft_consensus.cc:359] T c2c927e461f04fb48bfa9e3a685b077d P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:38.230589 21585 raft_consensus.cc:385] T c2c927e461f04fb48bfa9e3a685b077d P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:38.234338 21585 raft_consensus.cc:740] T c2c927e461f04fb48bfa9e3a685b077d P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d870164263bb44b6b780a557daea3656, State: Initialized, Role: FOLLOWER
I20260130 02:58:38.236334 21585 consensus_queue.cc:260] T c2c927e461f04fb48bfa9e3a685b077d P d870164263bb44b6b780a557daea3656 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
W20260130 02:58:38.242902 19463 consensus_peers.cc:597] T a3258852aed24ed582a79f77860e6f0d P d870164263bb44b6b780a557daea3656 -> Peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Couldn't send request to peer 371a36d219164974abdf848841528152. Status: Network error: Client connection negotiation failed: client connection to 127.18.100.65:42021: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260130 02:58:38.245090 21585 ts_tablet_manager.cc:1434] T c2c927e461f04fb48bfa9e3a685b077d P d870164263bb44b6b780a557daea3656: Time spent starting tablet: real 0.021s user 0.005s sys 0.000s
I20260130 02:58:38.249137 21583 raft_consensus.cc:493] T 3379641837df4b6ea360e8e27dfe4f8b P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:58:38.250615 21583 raft_consensus.cc:515] T 3379641837df4b6ea360e8e27dfe4f8b P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:38.252470 21583 leader_election.cc:290] T 3379641837df4b6ea360e8e27dfe4f8b P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 371a36d219164974abdf848841528152 (127.18.100.65:42021), 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767)
W20260130 02:58:38.256685 19463 leader_election.cc:336] T 3379641837df4b6ea360e8e27dfe4f8b P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Network error: Client connection negotiation failed: client connection to 127.18.100.65:42021: connect: Connection refused (error 111)
I20260130 02:58:38.272212 21585 tablet_bootstrap.cc:492] T 47f574674aa7414588605ca38eb7200a P d870164263bb44b6b780a557daea3656: Bootstrap starting.
I20260130 02:58:38.297354 21585 tablet_bootstrap.cc:654] T 47f574674aa7414588605ca38eb7200a P d870164263bb44b6b780a557daea3656: Neither blocks nor log segments found. Creating new log.
W20260130 02:58:38.375721 19463 consensus_peers.cc:597] T 0855d88806914a36b4bf6b49036f0ad0 P d870164263bb44b6b780a557daea3656 -> Peer 371a36d219164974abdf848841528152 (127.18.100.65:42021): Couldn't send request to peer 371a36d219164974abdf848841528152. Status: Network error: Client connection negotiation failed: client connection to 127.18.100.65:42021: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260130 02:58:38.376163 19575 heartbeater.cc:499] Master 127.18.100.126:38191 was elected leader, sending a full tablet report...
I20260130 02:58:38.415064 21585 tablet_bootstrap.cc:492] T 47f574674aa7414588605ca38eb7200a P d870164263bb44b6b780a557daea3656: No bootstrap required, opened a new log
I20260130 02:58:38.423339 21585 ts_tablet_manager.cc:1403] T 47f574674aa7414588605ca38eb7200a P d870164263bb44b6b780a557daea3656: Time spent bootstrapping tablet: real 0.151s user 0.013s sys 0.002s
I20260130 02:58:38.426118 21585 raft_consensus.cc:359] T 47f574674aa7414588605ca38eb7200a P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:38.427004 21585 raft_consensus.cc:385] T 47f574674aa7414588605ca38eb7200a P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:38.427333 21585 raft_consensus.cc:740] T 47f574674aa7414588605ca38eb7200a P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d870164263bb44b6b780a557daea3656, State: Initialized, Role: FOLLOWER
I20260130 02:58:38.428124 21585 consensus_queue.cc:260] T 47f574674aa7414588605ca38eb7200a P d870164263bb44b6b780a557daea3656 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:38.430630 21585 ts_tablet_manager.cc:1434] T 47f574674aa7414588605ca38eb7200a P d870164263bb44b6b780a557daea3656: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260130 02:58:38.433902 21585 tablet_bootstrap.cc:492] T 6ad6bb8c986a433d94c745f2b47266ff P d870164263bb44b6b780a557daea3656: Bootstrap starting.
I20260130 02:58:38.446887 21585 tablet_bootstrap.cc:654] T 6ad6bb8c986a433d94c745f2b47266ff P d870164263bb44b6b780a557daea3656: Neither blocks nor log segments found. Creating new log.
I20260130 02:58:38.473109 21585 tablet_bootstrap.cc:492] T 6ad6bb8c986a433d94c745f2b47266ff P d870164263bb44b6b780a557daea3656: No bootstrap required, opened a new log
I20260130 02:58:38.473652 21585 ts_tablet_manager.cc:1403] T 6ad6bb8c986a433d94c745f2b47266ff P d870164263bb44b6b780a557daea3656: Time spent bootstrapping tablet: real 0.040s user 0.009s sys 0.003s
I20260130 02:58:38.476190 21585 raft_consensus.cc:359] T 6ad6bb8c986a433d94c745f2b47266ff P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:38.479799 21585 raft_consensus.cc:385] T 6ad6bb8c986a433d94c745f2b47266ff P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:58:38.480139 21585 raft_consensus.cc:740] T 6ad6bb8c986a433d94c745f2b47266ff P d870164263bb44b6b780a557daea3656 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d870164263bb44b6b780a557daea3656, State: Initialized, Role: FOLLOWER
I20260130 02:58:38.480923 21585 consensus_queue.cc:260] T 6ad6bb8c986a433d94c745f2b47266ff P d870164263bb44b6b780a557daea3656 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371a36d219164974abdf848841528152" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 42021 } } peers { permanent_uuid: "78c2c46c7520492db35d51c244fb4f74" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 40767 } } peers { permanent_uuid: "d870164263bb44b6b780a557daea3656" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 39571 } }
I20260130 02:58:38.486291 21585 ts_tablet_manager.cc:1434] T 6ad6bb8c986a433d94c745f2b47266ff P d870164263bb44b6b780a557daea3656: Time spent starting tablet: real 0.012s user 0.005s sys 0.000s
W20260130 02:58:38.534459 20627 connection.cc:537] server connection from 127.18.100.66:39475 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260130 02:58:38.538317 18936 connection.cc:537] server connection from 127.18.100.66:49417 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260130 02:58:38.539630 19462 connection.cc:537] client connection to 127.18.100.66:40767 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260130 02:58:38.550132 21596 negotiation.cc:337] Failed RPC negotiation. Trace:
0130 02:58:38.548708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.100.66:40767 (local address 127.0.0.1:38656)
0130 02:58:38.549323 (+ 615us) negotiation.cc:107] Waiting for socket to connect
0130 02:58:38.549501 (+ 178us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.100.66:40767: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":58}
W20260130 02:58:38.551790 21598 negotiation.cc:337] Failed RPC negotiation. Trace:
0130 02:58:38.544371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.100.66:40767 (local address 127.18.100.67:60837)
0130 02:58:38.545004 (+ 633us) negotiation.cc:107] Waiting for socket to connect
0130 02:58:38.545065 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0130 02:58:38.545500 (+ 435us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0130 02:58:38.550874 (+ 5374us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.100.66:40767: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":68}
W20260130 02:58:38.552254 19462 leader_election.cc:336] T dc4bdd8e4f104d4a86c0af6ac6accda0 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767): Network error: Client connection negotiation failed: client connection to 127.18.100.66:40767: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260130 02:58:38.552685 19462 leader_election.cc:304] T dc4bdd8e4f104d4a86c0af6ac6accda0 P d870164263bb44b6b780a557daea3656 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d870164263bb44b6b780a557daea3656; no voters: 371a36d219164974abdf848841528152, 78c2c46c7520492db35d51c244fb4f74
I20260130 02:58:38.553560 21513 raft_consensus.cc:2749] T dc4bdd8e4f104d4a86c0af6ac6accda0 P d870164263bb44b6b780a557daea3656 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
W20260130 02:58:38.554425 19462 consensus_peers.cc:597] T 0d14fb95ace9447f810ccbed4d5befdf P d870164263bb44b6b780a557daea3656 -> Peer 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767): Couldn't send request to peer 78c2c46c7520492db35d51c244fb4f74. Status: Network error: Client connection negotiation failed: client connection to 127.18.100.66:40767: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20260130 02:58:38.558363 19462 consensus_peers.cc:597] T eb7a755ab62b4552bb8588264c5f7b75 P d870164263bb44b6b780a557daea3656 -> Peer 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767): Couldn't send request to peer 78c2c46c7520492db35d51c244fb4f74. Status: Network error: Client connection negotiation failed: client connection to 127.18.100.66:40767: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20260130 02:58:38.562800 19462 consensus_peers.cc:597] T a3258852aed24ed582a79f77860e6f0d P d870164263bb44b6b780a557daea3656 -> Peer 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767): Couldn't send request to peer 78c2c46c7520492db35d51c244fb4f74. Status: Network error: Client connection negotiation failed: client connection to 127.18.100.66:40767: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20260130 02:58:38.563235 19462 consensus_peers.cc:597] T 2b8034a92b2043e5b57cf02f1ee6296d P d870164263bb44b6b780a557daea3656 -> Peer 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767): Couldn't send request to peer 78c2c46c7520492db35d51c244fb4f74. Status: Network error: Client connection negotiation failed: client connection to 127.18.100.66:40767: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20260130 02:58:38.563624 19462 consensus_peers.cc:597] T 3aa76993beb94b12bc706a76db26a03b P d870164263bb44b6b780a557daea3656 -> Peer 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767): Couldn't send request to peer 78c2c46c7520492db35d51c244fb4f74. Status: Network error: Client connection negotiation failed: client connection to 127.18.100.66:40767: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20260130 02:58:38.564025 19462 consensus_peers.cc:597] T 7caf7cbad5054396b5617094e4d536a0 P d870164263bb44b6b780a557daea3656 -> Peer 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767): Couldn't send request to peer 78c2c46c7520492db35d51c244fb4f74. Status: Network error: Client connection negotiation failed: client connection to 127.18.100.66:40767: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20260130 02:58:38.564404 19462 consensus_peers.cc:597] T 30cc52136bb3422895d95c0fed0c6798 P d870164263bb44b6b780a557daea3656 -> Peer 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767): Couldn't send request to peer 78c2c46c7520492db35d51c244fb4f74. Status: Network error: Client connection negotiation failed: client connection to 127.18.100.66:40767: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20260130 02:58:38.564769 19462 consensus_peers.cc:597] T 6d1f121f49cd47c2b8247daaf6874e01 P d870164263bb44b6b780a557daea3656 -> Peer 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767): Couldn't send request to peer 78c2c46c7520492db35d51c244fb4f74. Status: Network error: Client connection negotiation failed: client connection to 127.18.100.66:40767: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20260130 02:58:38.565155 19462 consensus_peers.cc:597] T e691c238199b4914bcc3fd248f31420c P d870164263bb44b6b780a557daea3656 -> Peer 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767): Couldn't send request to peer 78c2c46c7520492db35d51c244fb4f74. Status: Network error: Client connection negotiation failed: client connection to 127.18.100.66:40767: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
I20260130 02:58:38.565732 18833 external_mini_cluster.cc:1658] Killing /tmp/dist-test-tasktGe2wI/build/tsan/bin/kudu with pid 19436
I20260130 02:58:38.580971 18874 catalog_manager.cc:5746] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767) does not have the latest schema for tablet 8b47653c7c3945a68e9bf4680bfc0496 (table default.Table_1ae68ae3f0404843aca3c72f02678ac6 [id=8a9182f49e18489fafec8dca190c1b2f]). Expected version 1 got 0
I20260130 02:58:38.582159 18874 catalog_manager.cc:5746] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767) does not have the latest schema for tablet dc09a0f0cecf49738bab04596944c0f3 (table default.Table_32f4a0f63b1046529836267ff3f551fa [id=676ece9dc36b418a8b1400e8406d6b98]). Expected version 1 got 0
I20260130 02:58:38.591464 18874 catalog_manager.cc:5746] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767) does not have the latest schema for tablet 2b0edaad6877460cb7982efc62b00b66 (table default.Table_d95381c590c94802bdc83acef0f0df96 [id=64a8e0e079f34806aa68795f3d548c5e]). Expected version 1 got 0
I20260130 02:58:38.594275 18874 catalog_manager.cc:5746] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767) does not have the latest schema for tablet dcba6b534f9d406dad4fb502be40921c (table default.Table_a99512d7e42f46509759ae14fba6fd9b [id=dea97f19a50141878f97820194afa59e]). Expected version 1 got 0
I20260130 02:58:38.597402 18874 catalog_manager.cc:5746] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767) does not have the latest schema for tablet 657e9b0d8c05413f904129dce745710b (table default.Table_526677c8ef4144bd9de7a54a5e78e2be [id=b16908b75587459597a978c71756451c]). Expected version 1 got 0
I20260130 02:58:38.609438 18874 catalog_manager.cc:5746] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767) does not have the latest schema for tablet 49efcfced0b74c48962393149ac2a500 (table default.Table_2e81f0a0cbc449a397141b717db17025 [id=a9c919f57c0c437cb35a6921103e45a6]). Expected version 1 got 0
I20260130 02:58:38.616588 18874 catalog_manager.cc:5746] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767) does not have the latest schema for tablet 838e49e5e8874db393bdc15a1c92f87e (table default.Table_da3d581b15a94840898777f74611994e [id=b7e241ced53e4160ac6fa1f65ef3674e]). Expected version 1 got 0
I20260130 02:58:38.640112 18874 catalog_manager.cc:5746] TS 78c2c46c7520492db35d51c244fb4f74 (127.18.100.66:40767) does not have the latest schema for tablet 475b435b2967418f927fa172d2ff04ea (table default.Table_dee034aec0b24982b8dc2a3fa1b326f1 [id=66033e75b3814a8584311cf7e4a07ad3]). Expected version 1 got 0
I20260130 02:58:38.641168 18870 catalog_manager.cc:6199] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4: default.Table_aaf139c1a55d4a1783298ca12ae0d5c0 [id=d128767e303145a885eec3c5f714a27f] alter complete (version 4)
I20260130 02:58:38.678653 18870 catalog_manager.cc:6199] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4: default.Table_d0b4516d24384e5386c1debaf2543adb [id=f1342f9497b74e7a9ecad1de8082578d] alter complete (version 1)
I20260130 02:58:38.697171 18874 catalog_manager.cc:6199] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4: default.Table_c4f4904b6fac4b9880979cf0314e71c1 [id=c456c6418c93499a9d08f652720854ca] alter complete (version 1)
I20260130 02:58:38.717442 18874 catalog_manager.cc:6199] T 00000000000000000000000000000000 P 658299197a424f53a37f14fadf089ae4: default.Table_805e07e3546e447d8c8b8029477013cc [id=1515db389f284e2f9ed1a3cd0061d0f1] alter complete (version 7)
I20260130 02:58:38.759893 18833 external_mini_cluster.cc:1658] Killing /tmp/dist-test-tasktGe2wI/build/tsan/bin/kudu with pid 18842
W20260130 02:58:38.760004 21612 negotiation.cc:337] Failed RPC negotiation. Trace:
0130 02:58:38.757505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.100.67:39571 (local address 127.0.0.1:41006)
0130 02:58:38.759267 (+ 1762us) negotiation.cc:107] Waiting for socket to connect
0130 02:58:38.759432 (+ 165us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.100.67:39571: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":1225}
W20260130 02:58:38.831418 18936 connection.cc:537] server connection from 127.0.0.1:48658 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260130 02:58:38.831395 20613 connection.cc:537] server connection from 127.0.0.1:33758 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260130 02:58:38.835261 18833 external_mini_cluster.cc:1658] Killing /tmp/dist-test-tasktGe2wI/build/tsan/bin/kudu with pid 18912
I20260130 02:58:38.869777 18833 external_mini_cluster.cc:1658] Killing /tmp/dist-test-tasktGe2wI/build/tsan/bin/kudu with pid 19872
2026-01-30T02:58:38Z chronyd exiting
I20260130 02:58:38.922928 18833 master-stress-test.cc:585] Tables created: 141
I20260130 02:58:38.923130 18833 master-stress-test.cc:586] Tables altered: 56
I20260130 02:58:38.923272 18833 master-stress-test.cc:587] Tables deleted: 45
I20260130 02:58:38.923410 18833 master-stress-test.cc:588] Tablets replaced: 30
I20260130 02:58:38.923557 18833 master-stress-test.cc:589] Masters restarted: 1
[ OK ] CatalogManagerConfigurations/MasterStressTest.Test/0 (59845 ms)
[ RUN ] CatalogManagerConfigurations/MasterStressTest.Test/3
2026-01-30T02:58:39Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-01-30T02:58:39Z Disabled control of system clock
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1769741859011789-18833-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1769741859011789-18833-0/minicluster-data/security.properties
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/tmp/dist-test-tasktGe2wI/thirdparty/src/hive-3.1.2/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/tmp/dist-test-tasktGe2wI/thirdparty/src/hadoop-3.4.1/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Metastore connection URL: jdbc:derby:/tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1769741859011789-18833-0/minicluster-data/metadb;create=true
Metastore Connection Driver : org.apache.derby.jdbc.EmbeddedDriver
Metastore connection User: APP
Starting metastore schema initialization to 3.1.0
Initialization script hive-schema-3.1.0.derby.sql
Initialization script completed
schemaTool completed
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1769741859011789-18833-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1769741859011789-18833-0/minicluster-data/security.properties
2026-01-30 02:58:45: Starting Hive Metastore Server
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1769741859011789-18833-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1769741859011789-18833-0/minicluster-data/security.properties
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/tmp/dist-test-tasktGe2wI/thirdparty/src/hive-3.1.2/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/tmp/dist-test-tasktGe2wI/thirdparty/src/hadoop-3.4.1/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Starting hive metastore on port 0
I20260130 02:58:54.814379 18833 external_mini_cluster.cc:1366] Running /tmp/dist-test-tasktGe2wI/build/tsan/bin/kudu
/tmp/dist-test-tasktGe2wI/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1769741859011789-18833-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1769741859011789-18833-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1769741859011789-18833-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1769741859011789-18833-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.100.126:42077
--webserver_interface=127.18.100.126
--webserver_port=0
--builtin_ntp_servers=127.18.100.84:33971
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.100.126:42077,127.18.100.125:39157,127.18.100.124:34987
--hive_metastore_uris=thrift://127.0.0.1:42185
--hive_metastore_notification_log_poll_period_seconds=1
--enable_metadata_cleanup_for_deleted_tables_and_tablets=true
--catalog_manager_bg_task_wait_ms=10
--metadata_for_deleted_table_and_tablet_reserved_secs=0
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20260130 02:58:55.110044 21854 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260130 02:58:55.110828 21854 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260130 02:58:55.111507 21854 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260130 02:58:55.142921 21854 flags.cc:425] Enabled experimental flag: --enable_metadata_cleanup_for_deleted_tables_and_tablets=true
W20260130 02:58:55.143180 21854 flags.cc:425] Enabled experimental flag: --metadata_for_deleted_table_and_tablet_reserved_secs=0
W20260130 02:58:55.143383 21854 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260130 02:58:55.143579 21854 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260130 02:58:55.143780 21854 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260130 02:58:55.143977 21854 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260130 02:58:55.179754 21854 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.100.84:33971
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1769741859011789-18833-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1769741859011789-18833-0/minicluster-data/master-0/wal
--hive_metastore_uris=thrift://127.0.0.1:42185
--catalog_manager_bg_task_wait_ms=10
--enable_metadata_cleanup_for_deleted_tables_and_tablets=true
--metadata_for_deleted_table_and_tablet_reserved_secs=0
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.18.100.126:42077,127.18.100.125:39157,127.18.100.124:34987
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.100.126:42077
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1769741859011789-18833-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.100.126
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1769741859011789-18833-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type FASTDEBUG
built by None at 30 Jan 2026 02:43:21 UTC on 5fd53c4cbb9d
build id 10257
TSAN enabled
I20260130 02:58:55.181078 21854 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260130 02:58:55.182691 21854 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260130 02:58:55.193981 21860 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260130 02:58:56.596590 21859 debug-util.cc:398] Leaking SignalData structure 0x7b0800028120 after lost signal to thread 21854
W20260130 02:58:56.740568 21859 kernel_stack_watchdog.cc:198] Thread 21854 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:642 for 400ms:
Kernel stack:
(could not read kernel stack)
User stack:
<Timed out: thread did not respond: maybe it is blocking signals>
W20260130 02:58:56.740722 21854 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.547s user 0.590s sys 0.956s
W20260130 02:58:55.195629 21861 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260130 02:58:56.741232 21854 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.547s user 0.591s sys 0.956s
W20260130 02:58:56.741847 21862 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1545 milliseconds
I20260130 02:58:56.743155 21854 server_base.cc:1055] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
W20260130 02:58:56.743152 21863 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260130 02:58:56.747020 21854 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260130 02:58:56.749581 21854 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260130 02:58:56.750957 21854 hybrid_clock.cc:648] HybridClock initialized: now 1769741936750892 us; error 67 us; skew 500 ppm
I20260130 02:58:56.751813 21854 server_base.cc:860] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260130 02:58:56.757901 21854 webserver.cc:492] Webserver started at http://127.18.100.126:44889/ using document root <none> and password file <none>
I20260130 02:58:56.758880 21854 fs_manager.cc:362] Metadata directory not provided
I20260130 02:58:56.759104 21854 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260130 02:58:56.759537 21854 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260130 02:58:56.763872 21854 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1769741859011789-18833-0/minicluster-data/master-0/data/instance:
uuid: "22dedcf69e00438999d3a5fedde57324"
format_stamp: "Formatted at 2026-01-30 02:58:56 on dist-test-slave-mn07"
I20260130 02:58:56.764930 21854 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1769741859011789-18833-0/minicluster-data/master-0/wal/instance:
uuid: "22dedcf69e00438999d3a5fedde57324"
format_stamp: "Formatted at 2026-01-30 02:58:56 on dist-test-slave-mn07"
I20260130 02:58:56.772464 21854 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.004s sys 0.004s
I20260130 02:58:56.778209 21870 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260130 02:58:56.779346 21854 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260130 02:58:56.779690 21854 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1769741859011789-18833-0/minicluster-data/master-0/data,/tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1769741859011789-18833-0/minicluster-data/master-0/wal
uuid: "22dedcf69e00438999d3a5fedde57324"
format_stamp: "Formatted at 2026-01-30 02:58:56 on dist-test-slave-mn07"
I20260130 02:58:56.780031 21854 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1769741859011789-18833-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1769741859011789-18833-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1769741859011789-18833-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260130 02:58:56.843456 21854 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260130 02:58:56.844925 21854 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260130 02:58:56.845348 21854 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260130 02:58:56.912744 21854 rpc_server.cc:307] RPC server started. Bound to: 127.18.100.126:42077
I20260130 02:58:56.912906 21921 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.100.126:42077 every 8 connection(s)
I20260130 02:58:56.915390 21854 server_base.cc:1192] Dumped server information to /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1769741859011789-18833-0/minicluster-data/master-0/data/info.pb
I20260130 02:58:56.921149 21922 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:56.923097 18833 external_mini_cluster.cc:1428] Started /tmp/dist-test-tasktGe2wI/build/tsan/bin/kudu as pid 21854
I20260130 02:58:56.923472 18833 external_mini_cluster.cc:1442] Reading /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1769741859011789-18833-0/minicluster-data/master-0/wal/instance
I20260130 02:58:56.925891 18833 external_mini_cluster.cc:1366] Running /tmp/dist-test-tasktGe2wI/build/tsan/bin/kudu
/tmp/dist-test-tasktGe2wI/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1769741859011789-18833-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1769741859011789-18833-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1769741859011789-18833-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1769741859011789-18833-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.100.125:39157
--webserver_interface=127.18.100.125
--webserver_port=0
--builtin_ntp_servers=127.18.100.84:33971
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.100.126:42077,127.18.100.125:39157,127.18.100.124:34987
--hive_metastore_uris=thrift://127.0.0.1:42185
--hive_metastore_notification_log_poll_period_seconds=1
--enable_metadata_cleanup_for_deleted_tables_and_tablets=true
--catalog_manager_bg_task_wait_ms=10
--metadata_for_deleted_table_and_tablet_reserved_secs=0
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20260130 02:58:56.932061 21922 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.100.126" port: 42077 } has no permanent_uuid. Determining permanent_uuid...
I20260130 02:58:56.960572 21922 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39157 } has no permanent_uuid. Determining permanent_uuid...
W20260130 02:58:56.962888 21874 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.18.100.125:39157: connect: Connection refused (error 111)
W20260130 02:58:56.966898 21922 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.18.100.125:39157: Network error: Client connection negotiation failed: client connection to 127.18.100.125:39157: connect: Connection refused (error 111)
I20260130 02:58:57.016396 21922 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39157 } attempt: 1
W20260130 02:58:57.020274 21922 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.18.100.125:39157: Network error: Client connection negotiation failed: client connection to 127.18.100.125:39157: connect: Connection refused (error 111)
I20260130 02:58:57.088716 21922 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39157 } attempt: 2
W20260130 02:58:57.093495 21922 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.18.100.125:39157: Network error: Client connection negotiation failed: client connection to 127.18.100.125:39157: connect: Connection refused (error 111)
I20260130 02:58:57.185024 21922 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39157 } attempt: 3
W20260130 02:58:57.189759 21922 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.18.100.125:39157: Network error: Client connection negotiation failed: client connection to 127.18.100.125:39157: connect: Connection refused (error 111)
W20260130 02:58:57.232424 21923 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260130 02:58:57.233022 21923 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260130 02:58:57.233467 21923 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260130 02:58:57.265425 21923 flags.cc:425] Enabled experimental flag: --enable_metadata_cleanup_for_deleted_tables_and_tablets=true
W20260130 02:58:57.265723 21923 flags.cc:425] Enabled experimental flag: --metadata_for_deleted_table_and_tablet_reserved_secs=0
W20260130 02:58:57.265980 21923 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260130 02:58:57.266222 21923 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260130 02:58:57.266469 21923 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260130 02:58:57.266716 21923 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260130 02:58:57.302942 21923 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.100.84:33971
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1769741859011789-18833-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1769741859011789-18833-0/minicluster-data/master-1/wal
--hive_metastore_uris=thrift://127.0.0.1:42185
--catalog_manager_bg_task_wait_ms=10
--enable_metadata_cleanup_for_deleted_tables_and_tablets=true
--metadata_for_deleted_table_and_tablet_reserved_secs=0
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.18.100.126:42077,127.18.100.125:39157,127.18.100.124:34987
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.100.125:39157
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1769741859011789-18833-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.18.100.125
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1769741859011789-18833-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type FASTDEBUG
built by None at 30 Jan 2026 02:43:21 UTC on 5fd53c4cbb9d
build id 10257
TSAN enabled
I20260130 02:58:57.304286 21923 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260130 02:58:57.305884 21923 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260130 02:58:57.317696 21932 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260130 02:58:57.333228 21922 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39157 } attempt: 4
W20260130 02:58:57.339948 21922 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.18.100.125:39157: Network error: Client connection negotiation failed: client connection to 127.18.100.125:39157: connect: Connection refused (error 111)
I20260130 02:58:57.639540 21922 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39157 } attempt: 5
W20260130 02:58:57.646776 21922 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.18.100.125:39157: Network error: Client connection negotiation failed: client connection to 127.18.100.125:39157: connect: Connection refused (error 111)
I20260130 02:58:58.194382 21922 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39157 } attempt: 6
W20260130 02:58:58.219434 21922 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.18.100.125:39157: Network error: Client connection negotiation failed: client connection to 127.18.100.125:39157: connect: Connection refused (error 111)
W20260130 02:58:58.692984 21934 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1372 milliseconds
W20260130 02:58:57.318120 21933 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260130 02:58:58.693667 21935 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260130 02:58:58.696491 21923 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.379s user 0.541s sys 0.826s
W20260130 02:58:58.696857 21923 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.380s user 0.541s sys 0.826s
I20260130 02:58:58.697129 21923 server_base.cc:1055] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20260130 02:58:58.699651 21923 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260130 02:58:58.702297 21923 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260130 02:58:58.703648 21923 hybrid_clock.cc:648] HybridClock initialized: now 1769741938703622 us; error 54 us; skew 500 ppm
I20260130 02:58:58.704512 21923 server_base.cc:860] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260130 02:58:58.711872 21923 webserver.cc:492] Webserver started at http://127.18.100.125:38355/ using document root <none> and password file <none>
I20260130 02:58:58.712864 21923 fs_manager.cc:362] Metadata directory not provided
I20260130 02:58:58.713090 21923 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260130 02:58:58.713522 21923 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260130 02:58:58.717938 21923 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1769741859011789-18833-0/minicluster-data/master-1/data/instance:
uuid: "b01057b6c5ff4fc4b07253801881fe13"
format_stamp: "Formatted at 2026-01-30 02:58:58 on dist-test-slave-mn07"
I20260130 02:58:58.719112 21923 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1769741859011789-18833-0/minicluster-data/master-1/wal/instance:
uuid: "b01057b6c5ff4fc4b07253801881fe13"
format_stamp: "Formatted at 2026-01-30 02:58:58 on dist-test-slave-mn07"
I20260130 02:58:58.727180 21923 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.001s sys 0.006s
I20260130 02:58:58.733342 21944 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260130 02:58:58.734612 21923 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.003s
I20260130 02:58:58.734951 21923 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1769741859011789-18833-0/minicluster-data/master-1/data,/tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1769741859011789-18833-0/minicluster-data/master-1/wal
uuid: "b01057b6c5ff4fc4b07253801881fe13"
format_stamp: "Formatted at 2026-01-30 02:58:58 on dist-test-slave-mn07"
I20260130 02:58:58.735297 21923 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1769741859011789-18833-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1769741859011789-18833-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1769741859011789-18833-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260130 02:58:58.797139 21923 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260130 02:58:58.798748 21923 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260130 02:58:58.799189 21923 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260130 02:58:58.867321 21923 rpc_server.cc:307] RPC server started. Bound to: 127.18.100.125:39157
I20260130 02:58:58.867393 21995 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.100.125:39157 every 8 connection(s)
I20260130 02:58:58.869941 21923 server_base.cc:1192] Dumped server information to /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1769741859011789-18833-0/minicluster-data/master-1/data/info.pb
I20260130 02:58:58.873009 18833 external_mini_cluster.cc:1428] Started /tmp/dist-test-tasktGe2wI/build/tsan/bin/kudu as pid 21923
I20260130 02:58:58.873405 18833 external_mini_cluster.cc:1442] Reading /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1769741859011789-18833-0/minicluster-data/master-1/wal/instance
I20260130 02:58:58.875598 21996 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:58:58.875994 18833 external_mini_cluster.cc:1366] Running /tmp/dist-test-tasktGe2wI/build/tsan/bin/kudu
/tmp/dist-test-tasktGe2wI/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1769741859011789-18833-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1769741859011789-18833-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1769741859011789-18833-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1769741859011789-18833-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.100.124:34987
--webserver_interface=127.18.100.124
--webserver_port=0
--builtin_ntp_servers=127.18.100.84:33971
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.100.126:42077,127.18.100.125:39157,127.18.100.124:34987
--hive_metastore_uris=thrift://127.0.0.1:42185
--hive_metastore_notification_log_poll_period_seconds=1
--enable_metadata_cleanup_for_deleted_tables_and_tablets=true
--catalog_manager_bg_task_wait_ms=10
--metadata_for_deleted_table_and_tablet_reserved_secs=0
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20260130 02:58:58.885962 21996 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.100.126" port: 42077 } has no permanent_uuid. Determining permanent_uuid...
I20260130 02:58:58.906157 21996 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39157 } has no permanent_uuid. Determining permanent_uuid...
I20260130 02:58:58.922152 21996 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34987 } has no permanent_uuid. Determining permanent_uuid...
W20260130 02:58:58.924830 21946 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.18.100.124:34987: connect: Connection refused (error 111)
W20260130 02:58:58.929224 21996 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.18.100.124:34987: Network error: Client connection negotiation failed: client connection to 127.18.100.124:34987: connect: Connection refused (error 111)
I20260130 02:58:58.978786 21996 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34987 } attempt: 1
W20260130 02:58:58.983161 21996 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.18.100.124:34987: Network error: Client connection negotiation failed: client connection to 127.18.100.124:34987: connect: Connection refused (error 111)
I20260130 02:58:59.051586 21996 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34987 } attempt: 2
W20260130 02:58:59.056015 21996 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.18.100.124:34987: Network error: Client connection negotiation failed: client connection to 127.18.100.124:34987: connect: Connection refused (error 111)
I20260130 02:58:59.147516 21996 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34987 } attempt: 3
W20260130 02:58:59.152735 21996 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.18.100.124:34987: Network error: Client connection negotiation failed: client connection to 127.18.100.124:34987: connect: Connection refused (error 111)
W20260130 02:58:59.199103 21997 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260130 02:58:59.199687 21997 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260130 02:58:59.200136 21997 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260130 02:58:59.231518 21997 flags.cc:425] Enabled experimental flag: --enable_metadata_cleanup_for_deleted_tables_and_tablets=true
W20260130 02:58:59.231791 21997 flags.cc:425] Enabled experimental flag: --metadata_for_deleted_table_and_tablet_reserved_secs=0
W20260130 02:58:59.232057 21997 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260130 02:58:59.232308 21997 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260130 02:58:59.232630 21997 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260130 02:58:59.232956 21997 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260130 02:58:59.271090 21997 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.100.84:33971
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1769741859011789-18833-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1769741859011789-18833-0/minicluster-data/master-2/wal
--hive_metastore_uris=thrift://127.0.0.1:42185
--catalog_manager_bg_task_wait_ms=10
--enable_metadata_cleanup_for_deleted_tables_and_tablets=true
--metadata_for_deleted_table_and_tablet_reserved_secs=0
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.18.100.126:42077,127.18.100.125:39157,127.18.100.124:34987
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.100.124:34987
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1769741859011789-18833-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.18.100.124
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1769741859011789-18833-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type FASTDEBUG
built by None at 30 Jan 2026 02:43:21 UTC on 5fd53c4cbb9d
build id 10257
TSAN enabled
I20260130 02:58:59.272486 21997 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260130 02:58:59.274114 21997 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260130 02:58:59.286404 22007 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260130 02:58:59.292040 21922 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39157 } attempt: 7
I20260130 02:58:59.296252 21996 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34987 } attempt: 4
W20260130 02:58:59.302595 21996 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.18.100.124:34987: Network error: Client connection negotiation failed: client connection to 127.18.100.124:34987: connect: Connection refused (error 111)
I20260130 02:58:59.319983 21922 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34987 } has no permanent_uuid. Determining permanent_uuid...
W20260130 02:58:59.343596 21922 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.18.100.124:34987: Network error: Client connection negotiation failed: client connection to 127.18.100.124:34987: connect: Connection refused (error 111)
I20260130 02:58:59.402098 21922 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34987 } attempt: 1
W20260130 02:58:59.408018 21922 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.18.100.124:34987: Network error: Client connection negotiation failed: client connection to 127.18.100.124:34987: connect: Connection refused (error 111)
I20260130 02:58:59.489491 21922 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34987 } attempt: 2
W20260130 02:58:59.495018 21922 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.18.100.124:34987: Network error: Client connection negotiation failed: client connection to 127.18.100.124:34987: connect: Connection refused (error 111)
I20260130 02:58:59.580492 21922 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34987 } attempt: 3
W20260130 02:58:59.587785 21922 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.18.100.124:34987: Network error: Client connection negotiation failed: client connection to 127.18.100.124:34987: connect: Connection refused (error 111)
I20260130 02:58:59.602144 21996 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34987 } attempt: 5
W20260130 02:58:59.608542 21996 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.18.100.124:34987: Network error: Client connection negotiation failed: client connection to 127.18.100.124:34987: connect: Connection refused (error 111)
I20260130 02:58:59.728267 21922 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34987 } attempt: 4
W20260130 02:58:59.734333 21922 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.18.100.124:34987: Network error: Client connection negotiation failed: client connection to 127.18.100.124:34987: connect: Connection refused (error 111)
I20260130 02:59:00.017829 21922 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34987 } attempt: 5
W20260130 02:59:00.023633 21922 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.18.100.124:34987: Network error: Client connection negotiation failed: client connection to 127.18.100.124:34987: connect: Connection refused (error 111)
W20260130 02:59:00.117197 21918 debug-util.cc:398] Leaking SignalData structure 0x7b080006f060 after lost signal to thread 21855
W20260130 02:59:00.118031 21918 debug-util.cc:398] Leaking SignalData structure 0x7b080006f400 after lost signal to thread 21921
I20260130 02:59:00.156035 21996 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34987 } attempt: 6
W20260130 02:59:00.186043 21996 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.18.100.124:34987: Network error: Client connection negotiation failed: client connection to 127.18.100.124:34987: connect: Connection refused (error 111)
I20260130 02:59:00.576079 21922 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34987 } attempt: 6
W20260130 02:59:00.605408 21922 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.18.100.124:34987: Network error: Client connection negotiation failed: client connection to 127.18.100.124:34987: connect: Connection refused (error 111)
W20260130 02:59:00.688138 22006 debug-util.cc:398] Leaking SignalData structure 0x7b0800028120 after lost signal to thread 21997
W20260130 02:59:00.845407 22006 kernel_stack_watchdog.cc:198] Thread 21997 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:642 for 400ms:
Kernel stack:
(could not read kernel stack)
User stack:
<Timed out: thread did not respond: maybe it is blocking signals>
W20260130 02:58:59.286692 22008 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260130 02:59:00.846660 22009 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1560 milliseconds
W20260130 02:59:00.847076 21997 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.562s user 0.585s sys 0.936s
W20260130 02:59:00.847472 21997 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.562s user 0.585s sys 0.936s
I20260130 02:59:00.847880 21997 server_base.cc:1055] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
W20260130 02:59:00.847945 22010 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260130 02:59:00.851763 21997 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260130 02:59:00.854437 21997 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260130 02:59:00.855931 21997 hybrid_clock.cc:648] HybridClock initialized: now 1769741940855904 us; error 39 us; skew 500 ppm
I20260130 02:59:00.856793 21997 server_base.cc:860] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260130 02:59:00.863214 21997 webserver.cc:492] Webserver started at http://127.18.100.124:34049/ using document root <none> and password file <none>
I20260130 02:59:00.864158 21997 fs_manager.cc:362] Metadata directory not provided
I20260130 02:59:00.864389 21997 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260130 02:59:00.864843 21997 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260130 02:59:00.869280 21997 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1769741859011789-18833-0/minicluster-data/master-2/data/instance:
uuid: "342be83f20a648fd849c7d23454fba08"
format_stamp: "Formatted at 2026-01-30 02:59:00 on dist-test-slave-mn07"
I20260130 02:59:00.870522 21997 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1769741859011789-18833-0/minicluster-data/master-2/wal/instance:
uuid: "342be83f20a648fd849c7d23454fba08"
format_stamp: "Formatted at 2026-01-30 02:59:00 on dist-test-slave-mn07"
I20260130 02:59:00.879014 21997 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.002s sys 0.005s
I20260130 02:59:00.885005 22025 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260130 02:59:00.886219 21997 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.005s sys 0.001s
I20260130 02:59:00.886540 21997 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1769741859011789-18833-0/minicluster-data/master-2/data,/tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1769741859011789-18833-0/minicluster-data/master-2/wal
uuid: "342be83f20a648fd849c7d23454fba08"
format_stamp: "Formatted at 2026-01-30 02:59:00 on dist-test-slave-mn07"
I20260130 02:59:00.886948 21997 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1769741859011789-18833-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1769741859011789-18833-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1769741859011789-18833-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260130 02:59:00.936300 21997 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260130 02:59:00.937777 21997 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260130 02:59:00.938215 21997 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260130 02:59:01.017587 21997 rpc_server.cc:307] RPC server started. Bound to: 127.18.100.124:34987
I20260130 02:59:01.017727 22076 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.100.124:34987 every 8 connection(s)
I20260130 02:59:01.020258 21997 server_base.cc:1192] Dumped server information to /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1769741859011789-18833-0/minicluster-data/master-2/data/info.pb
I20260130 02:59:01.027078 22077 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:59:01.030740 18833 external_mini_cluster.cc:1428] Started /tmp/dist-test-tasktGe2wI/build/tsan/bin/kudu as pid 21997
I20260130 02:59:01.031327 18833 external_mini_cluster.cc:1442] Reading /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1769741859011789-18833-0/minicluster-data/master-2/wal/instance
I20260130 02:59:01.042440 22077 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.100.126" port: 42077 } has no permanent_uuid. Determining permanent_uuid...
I20260130 02:59:01.078140 22077 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39157 } has no permanent_uuid. Determining permanent_uuid...
I20260130 02:59:01.089910 22077 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34987 } has no permanent_uuid. Determining permanent_uuid...
I20260130 02:59:01.108068 22077 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 342be83f20a648fd849c7d23454fba08: Bootstrap starting.
I20260130 02:59:01.113988 22077 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 342be83f20a648fd849c7d23454fba08: Neither blocks nor log segments found. Creating new log.
I20260130 02:59:01.116513 22077 log.cc:826] T 00000000000000000000000000000000 P 342be83f20a648fd849c7d23454fba08: Log is configured to *not* fsync() on all Append() calls
I20260130 02:59:01.121373 22077 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 342be83f20a648fd849c7d23454fba08: No bootstrap required, opened a new log
I20260130 02:59:01.141340 22077 raft_consensus.cc:359] T 00000000000000000000000000000000 P 342be83f20a648fd849c7d23454fba08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22dedcf69e00438999d3a5fedde57324" member_type: VOTER last_known_addr { host: "127.18.100.126" port: 42077 } } peers { permanent_uuid: "b01057b6c5ff4fc4b07253801881fe13" member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39157 } } peers { permanent_uuid: "342be83f20a648fd849c7d23454fba08" member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34987 } }
I20260130 02:59:01.142027 22077 raft_consensus.cc:385] T 00000000000000000000000000000000 P 342be83f20a648fd849c7d23454fba08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:59:01.142261 22077 raft_consensus.cc:740] T 00000000000000000000000000000000 P 342be83f20a648fd849c7d23454fba08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 342be83f20a648fd849c7d23454fba08, State: Initialized, Role: FOLLOWER
I20260130 02:59:01.142961 22077 consensus_queue.cc:260] T 00000000000000000000000000000000 P 342be83f20a648fd849c7d23454fba08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22dedcf69e00438999d3a5fedde57324" member_type: VOTER last_known_addr { host: "127.18.100.126" port: 42077 } } peers { permanent_uuid: "b01057b6c5ff4fc4b07253801881fe13" member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39157 } } peers { permanent_uuid: "342be83f20a648fd849c7d23454fba08" member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34987 } }
I20260130 02:59:01.145540 22085 sys_catalog.cc:455] T 00000000000000000000000000000000 P 342be83f20a648fd849c7d23454fba08 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22dedcf69e00438999d3a5fedde57324" member_type: VOTER last_known_addr { host: "127.18.100.126" port: 42077 } } peers { permanent_uuid: "b01057b6c5ff4fc4b07253801881fe13" member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39157 } } peers { permanent_uuid: "342be83f20a648fd849c7d23454fba08" member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34987 } } }
I20260130 02:59:01.146440 22085 sys_catalog.cc:458] T 00000000000000000000000000000000 P 342be83f20a648fd849c7d23454fba08 [sys.catalog]: This master's current role is: FOLLOWER
I20260130 02:59:01.147706 22077 sys_catalog.cc:565] T 00000000000000000000000000000000 P 342be83f20a648fd849c7d23454fba08 [sys.catalog]: configured and running, proceeding with master startup.
W20260130 02:59:01.193593 22031 tablet.cc:2378] T 00000000000000000000000000000000 P 342be83f20a648fd849c7d23454fba08: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260130 02:59:01.193601 22099 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 342be83f20a648fd849c7d23454fba08: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260130 02:59:01.196189 22099 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260130 02:59:01.214200 22099 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 342be83f20a648fd849c7d23454fba08: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260130 02:59:01.214532 22099 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260130 02:59:01.218328 22085 raft_consensus.cc:493] T 00000000000000000000000000000000 P 342be83f20a648fd849c7d23454fba08 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:59:01.218870 22085 raft_consensus.cc:515] T 00000000000000000000000000000000 P 342be83f20a648fd849c7d23454fba08 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22dedcf69e00438999d3a5fedde57324" member_type: VOTER last_known_addr { host: "127.18.100.126" port: 42077 } } peers { permanent_uuid: "b01057b6c5ff4fc4b07253801881fe13" member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39157 } } peers { permanent_uuid: "342be83f20a648fd849c7d23454fba08" member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34987 } }
I20260130 02:59:01.221841 22085 leader_election.cc:290] T 00000000000000000000000000000000 P 342be83f20a648fd849c7d23454fba08 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 22dedcf69e00438999d3a5fedde57324 (127.18.100.126:42077), b01057b6c5ff4fc4b07253801881fe13 (127.18.100.125:39157)
I20260130 02:59:01.182603 21997 master_runner.cc:186] Error getting master registration for 127.18.100.126:42077: OK, instance_id { permanent_uuid: "22dedcf69e00438999d3a5fedde57324" instance_seqno: 1769741936894714 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260130 02:59:01.232520 21997 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260130 02:59:01.232990 22099 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 342be83f20a648fd849c7d23454fba08: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260130 02:59:01.234865 22099 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260130 02:59:01.246491 21996 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34987 } attempt: 7
W20260130 02:59:01.251407 22099 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 342be83f20a648fd849c7d23454fba08: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260130 02:59:01.251713 22099 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260130 02:59:01.266067 22099 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 342be83f20a648fd849c7d23454fba08: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260130 02:59:01.266402 22099 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260130 02:59:01.280736 22099 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 342be83f20a648fd849c7d23454fba08: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260130 02:59:01.281078 22099 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260130 02:59:01.283893 21996 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b01057b6c5ff4fc4b07253801881fe13: Bootstrap starting.
W20260130 02:59:01.295614 22099 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 342be83f20a648fd849c7d23454fba08: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260130 02:59:01.295964 22099 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260130 02:59:01.306002 21996 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b01057b6c5ff4fc4b07253801881fe13: Neither blocks nor log segments found. Creating new log.
W20260130 02:59:01.310276 22099 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 342be83f20a648fd849c7d23454fba08: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260130 02:59:01.310604 22099 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260130 02:59:01.310343 21996 log.cc:826] T 00000000000000000000000000000000 P b01057b6c5ff4fc4b07253801881fe13: Log is configured to *not* fsync() on all Append() calls
W20260130 02:59:01.324995 22099 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 342be83f20a648fd849c7d23454fba08: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260130 02:59:01.325321 22099 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260130 02:59:01.336676 21996 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b01057b6c5ff4fc4b07253801881fe13: No bootstrap required, opened a new log
W20260130 02:59:01.339720 22099 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 342be83f20a648fd849c7d23454fba08: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260130 02:59:01.340039 22099 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260130 02:59:01.354341 22099 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 342be83f20a648fd849c7d23454fba08: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260130 02:59:01.354676 22099 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260130 02:59:01.369038 22099 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 342be83f20a648fd849c7d23454fba08: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260130 02:59:01.369372 22099 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260130 02:59:01.382632 21996 raft_consensus.cc:359] T 00000000000000000000000000000000 P b01057b6c5ff4fc4b07253801881fe13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22dedcf69e00438999d3a5fedde57324" member_type: VOTER last_known_addr { host: "127.18.100.126" port: 42077 } } peers { permanent_uuid: "b01057b6c5ff4fc4b07253801881fe13" member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39157 } } peers { permanent_uuid: "342be83f20a648fd849c7d23454fba08" member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34987 } }
I20260130 02:59:01.383684 21996 raft_consensus.cc:385] T 00000000000000000000000000000000 P b01057b6c5ff4fc4b07253801881fe13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20260130 02:59:01.383967 22099 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 342be83f20a648fd849c7d23454fba08: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260130 02:59:01.384042 21996 raft_consensus.cc:740] T 00000000000000000000000000000000 P b01057b6c5ff4fc4b07253801881fe13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b01057b6c5ff4fc4b07253801881fe13, State: Initialized, Role: FOLLOWER
W20260130 02:59:01.384284 22099 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260130 02:59:01.385262 21996 consensus_queue.cc:260] T 00000000000000000000000000000000 P b01057b6c5ff4fc4b07253801881fe13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22dedcf69e00438999d3a5fedde57324" member_type: VOTER last_known_addr { host: "127.18.100.126" port: 42077 } } peers { permanent_uuid: "b01057b6c5ff4fc4b07253801881fe13" member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39157 } } peers { permanent_uuid: "342be83f20a648fd849c7d23454fba08" member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34987 } }
I20260130 02:59:01.393519 22104 sys_catalog.cc:455] T 00000000000000000000000000000000 P b01057b6c5ff4fc4b07253801881fe13 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22dedcf69e00438999d3a5fedde57324" member_type: VOTER last_known_addr { host: "127.18.100.126" port: 42077 } } peers { permanent_uuid: "b01057b6c5ff4fc4b07253801881fe13" member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39157 } } peers { permanent_uuid: "342be83f20a648fd849c7d23454fba08" member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34987 } } }
W20260130 02:59:01.399677 22099 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 342be83f20a648fd849c7d23454fba08: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260130 02:59:01.400487 22099 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260130 02:59:01.403774 22104 sys_catalog.cc:458] T 00000000000000000000000000000000 P b01057b6c5ff4fc4b07253801881fe13 [sys.catalog]: This master's current role is: FOLLOWER
I20260130 02:59:01.403556 21971 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "342be83f20a648fd849c7d23454fba08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b01057b6c5ff4fc4b07253801881fe13" is_pre_election: true
I20260130 02:59:01.403401 21996 sys_catalog.cc:565] T 00000000000000000000000000000000 P b01057b6c5ff4fc4b07253801881fe13 [sys.catalog]: configured and running, proceeding with master startup.
I20260130 02:59:01.405588 21971 raft_consensus.cc:2468] T 00000000000000000000000000000000 P b01057b6c5ff4fc4b07253801881fe13 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 342be83f20a648fd849c7d23454fba08 in term 0.
I20260130 02:59:01.407164 22029 leader_election.cc:304] T 00000000000000000000000000000000 P 342be83f20a648fd849c7d23454fba08 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 342be83f20a648fd849c7d23454fba08, b01057b6c5ff4fc4b07253801881fe13; no voters:
I20260130 02:59:01.408131 22085 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 342be83f20a648fd849c7d23454fba08 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260130 02:59:01.408461 22085 raft_consensus.cc:493] T 00000000000000000000000000000000 P 342be83f20a648fd849c7d23454fba08 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260130 02:59:01.408749 22085 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 342be83f20a648fd849c7d23454fba08 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:59:01.416196 22085 raft_consensus.cc:515] T 00000000000000000000000000000000 P 342be83f20a648fd849c7d23454fba08 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22dedcf69e00438999d3a5fedde57324" member_type: VOTER last_known_addr { host: "127.18.100.126" port: 42077 } } peers { permanent_uuid: "b01057b6c5ff4fc4b07253801881fe13" member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39157 } } peers { permanent_uuid: "342be83f20a648fd849c7d23454fba08" member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34987 } }
I20260130 02:59:01.422709 22085 leader_election.cc:290] T 00000000000000000000000000000000 P 342be83f20a648fd849c7d23454fba08 [CANDIDATE]: Term 1 election: Requested vote from peers 22dedcf69e00438999d3a5fedde57324 (127.18.100.126:42077), b01057b6c5ff4fc4b07253801881fe13 (127.18.100.125:39157)
I20260130 02:59:01.425642 21971 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "342be83f20a648fd849c7d23454fba08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b01057b6c5ff4fc4b07253801881fe13"
I20260130 02:59:01.426988 21971 raft_consensus.cc:3060] T 00000000000000000000000000000000 P b01057b6c5ff4fc4b07253801881fe13 [term 0 FOLLOWER]: Advancing to term 1
W20260130 02:59:01.446997 22099 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 342be83f20a648fd849c7d23454fba08: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260130 02:59:01.448491 22099 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260130 02:59:01.452725 21971 raft_consensus.cc:2468] T 00000000000000000000000000000000 P b01057b6c5ff4fc4b07253801881fe13 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 342be83f20a648fd849c7d23454fba08 in term 1.
I20260130 02:59:01.455519 22029 leader_election.cc:304] T 00000000000000000000000000000000 P 342be83f20a648fd849c7d23454fba08 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 342be83f20a648fd849c7d23454fba08, b01057b6c5ff4fc4b07253801881fe13; no voters:
I20260130 02:59:01.456336 22085 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 342be83f20a648fd849c7d23454fba08 [term 1 FOLLOWER]: Leader election won for term 1
I20260130 02:59:01.471905 22085 raft_consensus.cc:697] T 00000000000000000000000000000000 P 342be83f20a648fd849c7d23454fba08 [term 1 LEADER]: Becoming Leader. State: Replica: 342be83f20a648fd849c7d23454fba08, State: Running, Role: LEADER
I20260130 02:59:01.473425 22085 consensus_queue.cc:237] T 00000000000000000000000000000000 P 342be83f20a648fd849c7d23454fba08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22dedcf69e00438999d3a5fedde57324" member_type: VOTER last_known_addr { host: "127.18.100.126" port: 42077 } } peers { permanent_uuid: "b01057b6c5ff4fc4b07253801881fe13" member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39157 } } peers { permanent_uuid: "342be83f20a648fd849c7d23454fba08" member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34987 } }
I20260130 02:59:01.493278 22108 sys_catalog.cc:455] T 00000000000000000000000000000000 P 342be83f20a648fd849c7d23454fba08 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 342be83f20a648fd849c7d23454fba08. Latest consensus state: current_term: 1 leader_uuid: "342be83f20a648fd849c7d23454fba08" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22dedcf69e00438999d3a5fedde57324" member_type: VOTER last_known_addr { host: "127.18.100.126" port: 42077 } } peers { permanent_uuid: "b01057b6c5ff4fc4b07253801881fe13" member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39157 } } peers { permanent_uuid: "342be83f20a648fd849c7d23454fba08" member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34987 } } }
I20260130 02:59:01.494141 22108 sys_catalog.cc:458] T 00000000000000000000000000000000 P 342be83f20a648fd849c7d23454fba08 [sys.catalog]: This master's current role is: LEADER
W20260130 02:59:01.499686 22099 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 342be83f20a648fd849c7d23454fba08: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260130 02:59:01.503394 22099 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260130 02:59:01.505132 22114 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260130 02:59:01.521314 22114 catalog_manager.cc:1494] Initializing Kudu cluster ID...
W20260130 02:59:01.557155 21950 tablet.cc:2378] T 00000000000000000000000000000000 P b01057b6c5ff4fc4b07253801881fe13: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260130 02:59:01.557298 21971 raft_consensus.cc:1275] T 00000000000000000000000000000000 P b01057b6c5ff4fc4b07253801881fe13 [term 1 FOLLOWER]: Refusing update from remote peer 342be83f20a648fd849c7d23454fba08: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260130 02:59:01.558878 22108 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 342be83f20a648fd849c7d23454fba08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b01057b6c5ff4fc4b07253801881fe13" member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260130 02:59:01.624454 22104 sys_catalog.cc:455] T 00000000000000000000000000000000 P b01057b6c5ff4fc4b07253801881fe13 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 342be83f20a648fd849c7d23454fba08. Latest consensus state: current_term: 1 leader_uuid: "342be83f20a648fd849c7d23454fba08" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22dedcf69e00438999d3a5fedde57324" member_type: VOTER last_known_addr { host: "127.18.100.126" port: 42077 } } peers { permanent_uuid: "b01057b6c5ff4fc4b07253801881fe13" member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39157 } } peers { permanent_uuid: "342be83f20a648fd849c7d23454fba08" member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34987 } } }
I20260130 02:59:01.625150 22104 sys_catalog.cc:458] T 00000000000000000000000000000000 P b01057b6c5ff4fc4b07253801881fe13 [sys.catalog]: This master's current role is: FOLLOWER
I20260130 02:59:01.628749 22119 mvcc.cc:204] Tried to move back new op lower bound from 7248862992603222016 to 7248862992288210944. Current Snapshot: MvccSnapshot[applied={T|T < 7248862992603222016}]
I20260130 02:59:01.633045 22124 mvcc.cc:204] Tried to move back new op lower bound from 7248862992603222016 to 7248862992288210944. Current Snapshot: MvccSnapshot[applied={T|T < 7248862992603222016}]
I20260130 02:59:01.638806 21922 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34987 } attempt: 7
I20260130 02:59:01.590973 21923 master_runner.cc:186] Error getting master registration for 127.18.100.126:42077: OK, instance_id { permanent_uuid: "22dedcf69e00438999d3a5fedde57324" instance_seqno: 1769741936894714 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260130 02:59:01.663517 22104 sys_catalog.cc:455] T 00000000000000000000000000000000 P b01057b6c5ff4fc4b07253801881fe13 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "342be83f20a648fd849c7d23454fba08" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22dedcf69e00438999d3a5fedde57324" member_type: VOTER last_known_addr { host: "127.18.100.126" port: 42077 } } peers { permanent_uuid: "b01057b6c5ff4fc4b07253801881fe13" member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39157 } } peers { permanent_uuid: "342be83f20a648fd849c7d23454fba08" member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34987 } } }
I20260130 02:59:01.664314 22104 sys_catalog.cc:458] T 00000000000000000000000000000000 P b01057b6c5ff4fc4b07253801881fe13 [sys.catalog]: This master's current role is: FOLLOWER
I20260130 02:59:01.666781 22085 sys_catalog.cc:455] T 00000000000000000000000000000000 P 342be83f20a648fd849c7d23454fba08 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "342be83f20a648fd849c7d23454fba08" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22dedcf69e00438999d3a5fedde57324" member_type: VOTER last_known_addr { host: "127.18.100.126" port: 42077 } } peers { permanent_uuid: "b01057b6c5ff4fc4b07253801881fe13" member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39157 } } peers { permanent_uuid: "342be83f20a648fd849c7d23454fba08" member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34987 } } }
I20260130 02:59:01.667498 22085 sys_catalog.cc:458] T 00000000000000000000000000000000 P 342be83f20a648fd849c7d23454fba08 [sys.catalog]: This master's current role is: LEADER
I20260130 02:59:01.670800 22114 catalog_manager.cc:1357] Generated new cluster ID: 40eb42871f464e95bd641ceacb567979
I20260130 02:59:01.671109 22114 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
W20260130 02:59:01.662441 22123 catalog_manager.cc:1568] T 00000000000000000000000000000000 P b01057b6c5ff4fc4b07253801881fe13: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260130 02:59:01.673491 22123 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260130 02:59:01.674001 21923 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260130 02:59:01.689379 22123 catalog_manager.cc:1269] Loaded cluster ID: 40eb42871f464e95bd641ceacb567979
I20260130 02:59:01.689747 22123 catalog_manager.cc:1562] T 00000000000000000000000000000000 P b01057b6c5ff4fc4b07253801881fe13: loading cluster ID for follower catalog manager: success
I20260130 02:59:01.692559 21922 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324: Bootstrap starting.
W20260130 02:59:01.694679 22123 catalog_manager.cc:1586] T 00000000000000000000000000000000 P b01057b6c5ff4fc4b07253801881fe13: acquiring CA information for follower catalog manager: Not found: root CA entry not found
W20260130 02:59:01.702225 22123 catalog_manager.cc:883] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry
I20260130 02:59:01.711035 21922 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324: Neither blocks nor log segments found. Creating new log.
I20260130 02:59:01.713481 21922 log.cc:826] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324: Log is configured to *not* fsync() on all Append() calls
I20260130 02:59:01.719885 22114 catalog_manager.cc:1380] Generated new certificate authority record
W20260130 02:59:01.720963 22123 catalog_manager.cc:1586] T 00000000000000000000000000000000 P b01057b6c5ff4fc4b07253801881fe13: acquiring CA information for follower catalog manager: Not found: root CA entry not found
W20260130 02:59:01.722163 22123 catalog_manager.cc:883] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry
I20260130 02:59:01.726079 22114 catalog_manager.cc:1514] Loading token signing keys...
I20260130 02:59:01.734706 21922 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324: No bootstrap required, opened a new log
I20260130 02:59:01.744860 22123 catalog_manager.cc:1584] T 00000000000000000000000000000000 P b01057b6c5ff4fc4b07253801881fe13: acquiring CA information for follower catalog manager: success
W20260130 02:59:01.750197 22028 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.18.100.126:42077 timed out after 0.527s (SENT)
W20260130 02:59:01.762338 22123 catalog_manager.cc:883] Not found: no TSK found in the system table: failed to prepare follower catalog manager, will retry
W20260130 02:59:01.780788 22123 catalog_manager.cc:883] Not found: no TSK found in the system table: failed to prepare follower catalog manager, will retry
I20260130 02:59:01.786788 22114 catalog_manager.cc:6027] T 00000000000000000000000000000000 P 342be83f20a648fd849c7d23454fba08: Generated new TSK 0
I20260130 02:59:01.791321 22114 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260130 02:59:01.799872 21922 raft_consensus.cc:359] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22dedcf69e00438999d3a5fedde57324" member_type: VOTER last_known_addr { host: "127.18.100.126" port: 42077 } } peers { permanent_uuid: "b01057b6c5ff4fc4b07253801881fe13" member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39157 } } peers { permanent_uuid: "342be83f20a648fd849c7d23454fba08" member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34987 } }
I20260130 02:59:01.802241 22123 catalog_manager.cc:1612] T 00000000000000000000000000000000 P b01057b6c5ff4fc4b07253801881fe13: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260130 02:59:01.803304 21922 raft_consensus.cc:385] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:59:01.803710 21922 raft_consensus.cc:740] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22dedcf69e00438999d3a5fedde57324, State: Initialized, Role: FOLLOWER
I20260130 02:59:01.804863 21922 consensus_queue.cc:260] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22dedcf69e00438999d3a5fedde57324" member_type: VOTER last_known_addr { host: "127.18.100.126" port: 42077 } } peers { permanent_uuid: "b01057b6c5ff4fc4b07253801881fe13" member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39157 } } peers { permanent_uuid: "342be83f20a648fd849c7d23454fba08" member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34987 } }
I20260130 02:59:01.810151 22114 catalog_manager.cc:1537] Loading latest processed Hive Metastore notification log event ID...
I20260130 02:59:01.822465 22132 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22dedcf69e00438999d3a5fedde57324" member_type: VOTER last_known_addr { host: "127.18.100.126" port: 42077 } } peers { permanent_uuid: "b01057b6c5ff4fc4b07253801881fe13" member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39157 } } peers { permanent_uuid: "342be83f20a648fd849c7d23454fba08" member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34987 } } }
I20260130 02:59:01.826503 22132 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [sys.catalog]: This master's current role is: FOLLOWER
I20260130 02:59:01.826333 21897 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "342be83f20a648fd849c7d23454fba08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22dedcf69e00438999d3a5fedde57324" is_pre_election: true
I20260130 02:59:01.826242 21922 sys_catalog.cc:565] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [sys.catalog]: configured and running, proceeding with master startup.
I20260130 02:59:01.827261 21897 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 342be83f20a648fd849c7d23454fba08 in term 0.
W20260130 02:59:01.827812 21897 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39896 (request call id 2) took 603 ms (client timeout 527 ms). Trace:
W20260130 02:59:01.828147 21897 rpcz_store.cc:269] 0130 02:59:01.223972 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:59:01.224112 (+ 140us) service_pool.cc:225] Handling call
0130 02:59:01.827731 (+603619us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260130 02:59:01.829684 21895 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:59:01.834825 21894 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "342be83f20a648fd849c7d23454fba08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22dedcf69e00438999d3a5fedde57324" is_pre_election: true
I20260130 02:59:01.836256 21896 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "342be83f20a648fd849c7d23454fba08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22dedcf69e00438999d3a5fedde57324"
W20260130 02:59:01.837169 21896 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39896 (request call id 3) took 417 ms (client timeout 525 ms). Trace:
W20260130 02:59:01.837487 21896 rpcz_store.cc:269] 0130 02:59:01.420105 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:59:01.420252 (+ 147us) service_pool.cc:225] Handling call
0130 02:59:01.837113 (+416861us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260130 02:59:01.842832 21895 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [term 1 FOLLOWER]: Refusing update from remote peer 342be83f20a648fd849c7d23454fba08: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260130 02:59:01.844543 22108 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 342be83f20a648fd849c7d23454fba08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22dedcf69e00438999d3a5fedde57324" member_type: VOTER last_known_addr { host: "127.18.100.126" port: 42077 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260130 02:59:01.845420 22114 catalog_manager.cc:5876] Last processed Hive Metastore notification event ID: -1
W20260130 02:59:01.883093 21876 tablet.cc:2378] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260130 02:59:01.883710 18833 external_mini_cluster.cc:1366] Running /tmp/dist-test-tasktGe2wI/build/tsan/bin/kudu
/tmp/dist-test-tasktGe2wI/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1769741859011789-18833-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1769741859011789-18833-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1769741859011789-18833-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1769741859011789-18833-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.100.65:0
--local_ip_for_outbound_sockets=127.18.100.65
--webserver_interface=127.18.100.65
--webserver_port=0
--tserver_master_addrs=127.18.100.126:42077,127.18.100.125:39157,127.18.100.124:34987
--builtin_ntp_servers=127.18.100.84:33971
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20260130 02:59:01.952733 22132 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 342be83f20a648fd849c7d23454fba08. Latest consensus state: current_term: 1 leader_uuid: "342be83f20a648fd849c7d23454fba08" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22dedcf69e00438999d3a5fedde57324" member_type: VOTER last_known_addr { host: "127.18.100.126" port: 42077 } } peers { permanent_uuid: "b01057b6c5ff4fc4b07253801881fe13" member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39157 } } peers { permanent_uuid: "342be83f20a648fd849c7d23454fba08" member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34987 } } }
I20260130 02:59:01.953449 22132 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [sys.catalog]: This master's current role is: FOLLOWER
I20260130 02:59:01.955472 22144 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "342be83f20a648fd849c7d23454fba08" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22dedcf69e00438999d3a5fedde57324" member_type: VOTER last_known_addr { host: "127.18.100.126" port: 42077 } } peers { permanent_uuid: "b01057b6c5ff4fc4b07253801881fe13" member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39157 } } peers { permanent_uuid: "342be83f20a648fd849c7d23454fba08" member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34987 } } }
I20260130 02:59:01.956240 22144 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [sys.catalog]: This master's current role is: FOLLOWER
I20260130 02:59:02.009032 22153 catalog_manager.cc:805] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260130 02:59:02.051992 22153 catalog_manager.cc:1269] Loaded cluster ID: 40eb42871f464e95bd641ceacb567979
I20260130 02:59:02.052277 22153 catalog_manager.cc:1562] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324: loading cluster ID for follower catalog manager: success
I20260130 02:59:02.071010 22153 catalog_manager.cc:1584] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324: acquiring CA information for follower catalog manager: success
I20260130 02:59:02.077299 22153 catalog_manager.cc:1612] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260130 02:59:02.349730 22144 raft_consensus.cc:493] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 342be83f20a648fd849c7d23454fba08)
I20260130 02:59:02.350260 22144 raft_consensus.cc:515] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22dedcf69e00438999d3a5fedde57324" member_type: VOTER last_known_addr { host: "127.18.100.126" port: 42077 } } peers { permanent_uuid: "b01057b6c5ff4fc4b07253801881fe13" member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39157 } } peers { permanent_uuid: "342be83f20a648fd849c7d23454fba08" member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34987 } }
I20260130 02:59:02.352577 22144 leader_election.cc:290] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b01057b6c5ff4fc4b07253801881fe13 (127.18.100.125:39157), 342be83f20a648fd849c7d23454fba08 (127.18.100.124:34987)
I20260130 02:59:02.355115 21971 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "22dedcf69e00438999d3a5fedde57324" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b01057b6c5ff4fc4b07253801881fe13" is_pre_election: true
I20260130 02:59:02.356469 22052 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "22dedcf69e00438999d3a5fedde57324" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "342be83f20a648fd849c7d23454fba08" is_pre_election: true
I20260130 02:59:02.358995 21874 leader_election.cc:304] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22dedcf69e00438999d3a5fedde57324; no voters: 342be83f20a648fd849c7d23454fba08, b01057b6c5ff4fc4b07253801881fe13
I20260130 02:59:02.360196 22144 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260130 02:59:02.484716 22108 sys_catalog.cc:455] T 00000000000000000000000000000000 P 342be83f20a648fd849c7d23454fba08 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "342be83f20a648fd849c7d23454fba08" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22dedcf69e00438999d3a5fedde57324" member_type: VOTER last_known_addr { host: "127.18.100.126" port: 42077 } } peers { permanent_uuid: "b01057b6c5ff4fc4b07253801881fe13" member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39157 } } peers { permanent_uuid: "342be83f20a648fd849c7d23454fba08" member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34987 } } }
I20260130 02:59:02.485468 22108 sys_catalog.cc:458] T 00000000000000000000000000000000 P 342be83f20a648fd849c7d23454fba08 [sys.catalog]: This master's current role is: LEADER
W20260130 02:59:02.543773 22140 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260130 02:59:02.544507 22140 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260130 02:59:02.545619 22140 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260130 02:59:02.604400 22140 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260130 02:59:02.606184 22140 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.100.65
I20260130 02:59:02.672140 22140 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.100.84:33971
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1769741859011789-18833-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1769741859011789-18833-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.100.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1769741859011789-18833-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.100.65
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.18.100.126:42077,127.18.100.125:39157,127.18.100.124:34987
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.100.65
--log_dir=/tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1769741859011789-18833-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type FASTDEBUG
built by None at 30 Jan 2026 02:43:21 UTC on 5fd53c4cbb9d
build id 10257
TSAN enabled
I20260130 02:59:02.674268 22140 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260130 02:59:02.676617 22140 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260130 02:59:02.711460 22164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260130 02:59:03.239360 22167 raft_consensus.cc:493] T 00000000000000000000000000000000 P b01057b6c5ff4fc4b07253801881fe13 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 342be83f20a648fd849c7d23454fba08)
I20260130 02:59:03.248942 22167 raft_consensus.cc:515] T 00000000000000000000000000000000 P b01057b6c5ff4fc4b07253801881fe13 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22dedcf69e00438999d3a5fedde57324" member_type: VOTER last_known_addr { host: "127.18.100.126" port: 42077 } } peers { permanent_uuid: "b01057b6c5ff4fc4b07253801881fe13" member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39157 } } peers { permanent_uuid: "342be83f20a648fd849c7d23454fba08" member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34987 } }
I20260130 02:59:03.254011 21895 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b01057b6c5ff4fc4b07253801881fe13" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "22dedcf69e00438999d3a5fedde57324" is_pre_election: true
I20260130 02:59:03.257084 22167 leader_election.cc:290] T 00000000000000000000000000000000 P b01057b6c5ff4fc4b07253801881fe13 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 22dedcf69e00438999d3a5fedde57324 (127.18.100.126:42077), 342be83f20a648fd849c7d23454fba08 (127.18.100.124:34987)
I20260130 02:59:03.257922 22052 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b01057b6c5ff4fc4b07253801881fe13" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "342be83f20a648fd849c7d23454fba08" is_pre_election: true
I20260130 02:59:03.261699 21946 leader_election.cc:304] T 00000000000000000000000000000000 P b01057b6c5ff4fc4b07253801881fe13 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b01057b6c5ff4fc4b07253801881fe13; no voters: 22dedcf69e00438999d3a5fedde57324, 342be83f20a648fd849c7d23454fba08
I20260130 02:59:03.263825 22167 raft_consensus.cc:2749] T 00000000000000000000000000000000 P b01057b6c5ff4fc4b07253801881fe13 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260130 02:59:03.440804 22168 raft_consensus.cc:493] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 342be83f20a648fd849c7d23454fba08)
I20260130 02:59:03.441609 22168 raft_consensus.cc:515] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22dedcf69e00438999d3a5fedde57324" member_type: VOTER last_known_addr { host: "127.18.100.126" port: 42077 } } peers { permanent_uuid: "b01057b6c5ff4fc4b07253801881fe13" member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39157 } } peers { permanent_uuid: "342be83f20a648fd849c7d23454fba08" member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34987 } }
I20260130 02:59:03.445086 22168 leader_election.cc:290] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b01057b6c5ff4fc4b07253801881fe13 (127.18.100.125:39157), 342be83f20a648fd849c7d23454fba08 (127.18.100.124:34987)
I20260130 02:59:03.444996 21971 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "22dedcf69e00438999d3a5fedde57324" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b01057b6c5ff4fc4b07253801881fe13" is_pre_election: true
I20260130 02:59:03.447361 22052 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "22dedcf69e00438999d3a5fedde57324" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "342be83f20a648fd849c7d23454fba08" is_pre_election: true
I20260130 02:59:03.449977 21872 leader_election.cc:304] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22dedcf69e00438999d3a5fedde57324; no voters: 342be83f20a648fd849c7d23454fba08, b01057b6c5ff4fc4b07253801881fe13
I20260130 02:59:03.451407 22168 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260130 02:59:03.807983 22169 raft_consensus.cc:493] T 00000000000000000000000000000000 P b01057b6c5ff4fc4b07253801881fe13 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 342be83f20a648fd849c7d23454fba08)
I20260130 02:59:03.815065 22169 raft_consensus.cc:515] T 00000000000000000000000000000000 P b01057b6c5ff4fc4b07253801881fe13 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22dedcf69e00438999d3a5fedde57324" member_type: VOTER last_known_addr { host: "127.18.100.126" port: 42077 } } peers { permanent_uuid: "b01057b6c5ff4fc4b07253801881fe13" member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39157 } } peers { permanent_uuid: "342be83f20a648fd849c7d23454fba08" member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34987 } }
I20260130 02:59:03.820389 21895 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b01057b6c5ff4fc4b07253801881fe13" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "22dedcf69e00438999d3a5fedde57324" is_pre_election: true
I20260130 02:59:03.823086 22169 leader_election.cc:290] T 00000000000000000000000000000000 P b01057b6c5ff4fc4b07253801881fe13 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 22dedcf69e00438999d3a5fedde57324 (127.18.100.126:42077), 342be83f20a648fd849c7d23454fba08 (127.18.100.124:34987)
I20260130 02:59:03.824167 22052 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b01057b6c5ff4fc4b07253801881fe13" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "342be83f20a648fd849c7d23454fba08" is_pre_election: true
I20260130 02:59:03.825699 21946 leader_election.cc:304] T 00000000000000000000000000000000 P b01057b6c5ff4fc4b07253801881fe13 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b01057b6c5ff4fc4b07253801881fe13; no voters: 22dedcf69e00438999d3a5fedde57324, 342be83f20a648fd849c7d23454fba08
I20260130 02:59:03.826469 22169 raft_consensus.cc:2749] T 00000000000000000000000000000000 P b01057b6c5ff4fc4b07253801881fe13 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260130 02:59:03.968792 22170 raft_consensus.cc:493] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 342be83f20a648fd849c7d23454fba08)
I20260130 02:59:03.969482 22170 raft_consensus.cc:515] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22dedcf69e00438999d3a5fedde57324" member_type: VOTER last_known_addr { host: "127.18.100.126" port: 42077 } } peers { permanent_uuid: "b01057b6c5ff4fc4b07253801881fe13" member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39157 } } peers { permanent_uuid: "342be83f20a648fd849c7d23454fba08" member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34987 } }
I20260130 02:59:03.974817 21971 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "22dedcf69e00438999d3a5fedde57324" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b01057b6c5ff4fc4b07253801881fe13" is_pre_election: true
I20260130 02:59:03.977844 22052 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "22dedcf69e00438999d3a5fedde57324" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "342be83f20a648fd849c7d23454fba08" is_pre_election: true
I20260130 02:59:03.979965 21872 leader_election.cc:304] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22dedcf69e00438999d3a5fedde57324; no voters: 342be83f20a648fd849c7d23454fba08, b01057b6c5ff4fc4b07253801881fe13
I20260130 02:59:03.985302 22170 leader_election.cc:290] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b01057b6c5ff4fc4b07253801881fe13 (127.18.100.125:39157), 342be83f20a648fd849c7d23454fba08 (127.18.100.124:34987)
I20260130 02:59:03.985559 22171 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260130 02:59:04.100431 22162 debug-util.cc:398] Leaking SignalData structure 0x7b0800034c40 after lost signal to thread 22140
W20260130 02:59:04.401597 22162 kernel_stack_watchdog.cc:198] Thread 22140 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:642 for 390ms:
Kernel stack:
(could not read kernel stack)
User stack:
<Timed out: thread did not respond: maybe it is blocking signals>
W20260130 02:59:02.712049 22163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260130 02:59:04.401932 22140 thread.cc:641] GCE (cloud detector) Time spent creating pthread: real 1.694s user 0.407s sys 0.612s
W20260130 02:59:04.402331 22140 thread.cc:608] GCE (cloud detector) Time spent starting thread: real 1.694s user 0.407s sys 0.612s
W20260130 02:59:04.418565 22173 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260130 02:59:04.420089 22140 server_base.cc:1060] running on GCE node
I20260130 02:59:04.421516 22140 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260130 02:59:04.439186 22140 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260130 02:59:04.440758 22140 hybrid_clock.cc:648] HybridClock initialized: now 1769741944440706 us; error 56 us; skew 500 ppm
I20260130 02:59:04.441892 22140 server_base.cc:860] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260130 02:59:04.466653 22140 webserver.cc:492] Webserver started at http://127.18.100.65:38953/ using document root <none> and password file <none>
I20260130 02:59:04.468065 22140 fs_manager.cc:362] Metadata directory not provided
I20260130 02:59:04.468370 22140 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260130 02:59:04.468978 22140 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260130 02:59:04.476361 22140 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1769741859011789-18833-0/minicluster-data/ts-0/data/instance:
uuid: "111215cc989f4203a6e1517f2291d66f"
format_stamp: "Formatted at 2026-01-30 02:59:04 on dist-test-slave-mn07"
I20260130 02:59:04.477898 22140 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1769741859011789-18833-0/minicluster-data/ts-0/wal/instance:
uuid: "111215cc989f4203a6e1517f2291d66f"
format_stamp: "Formatted at 2026-01-30 02:59:04 on dist-test-slave-mn07"
I20260130 02:59:04.495175 22140 fs_manager.cc:696] Time spent creating directory manager: real 0.016s user 0.007s sys 0.006s
I20260130 02:59:04.509295 22178 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260130 02:59:04.510766 22140 fs_manager.cc:730] Time spent opening block manager: real 0.011s user 0.002s sys 0.008s
I20260130 02:59:04.511186 22140 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1769741859011789-18833-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1769741859011789-18833-0/minicluster-data/ts-0/wal
uuid: "111215cc989f4203a6e1517f2291d66f"
format_stamp: "Formatted at 2026-01-30 02:59:04 on dist-test-slave-mn07"
I20260130 02:59:04.511742 22140 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1769741859011789-18833-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1769741859011789-18833-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1769741859011789-18833-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260130 02:59:04.633064 22140 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260130 02:59:04.636016 22140 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260130 02:59:04.636605 22140 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260130 02:59:04.647368 22140 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260130 02:59:04.660395 22140 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260130 02:59:04.660727 22140 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260130 02:59:04.661049 22140 ts_tablet_manager.cc:616] Registered 0 tablets
I20260130 02:59:04.661271 22140 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260130 02:59:04.972111 22211 raft_consensus.cc:493] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 342be83f20a648fd849c7d23454fba08)
I20260130 02:59:04.972816 22211 raft_consensus.cc:515] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22dedcf69e00438999d3a5fedde57324" member_type: VOTER last_known_addr { host: "127.18.100.126" port: 42077 } } peers { permanent_uuid: "b01057b6c5ff4fc4b07253801881fe13" member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39157 } } peers { permanent_uuid: "342be83f20a648fd849c7d23454fba08" member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34987 } }
I20260130 02:59:04.981357 21971 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "22dedcf69e00438999d3a5fedde57324" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b01057b6c5ff4fc4b07253801881fe13" is_pre_election: true
I20260130 02:59:04.982712 22052 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "22dedcf69e00438999d3a5fedde57324" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "342be83f20a648fd849c7d23454fba08" is_pre_election: true
I20260130 02:59:04.983248 22211 leader_election.cc:290] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b01057b6c5ff4fc4b07253801881fe13 (127.18.100.125:39157), 342be83f20a648fd849c7d23454fba08 (127.18.100.124:34987)
I20260130 02:59:04.984164 21872 leader_election.cc:304] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22dedcf69e00438999d3a5fedde57324; no voters: 342be83f20a648fd849c7d23454fba08, b01057b6c5ff4fc4b07253801881fe13
I20260130 02:59:04.984862 22211 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260130 02:59:05.342279 22140 rpc_server.cc:307] RPC server started. Bound to: 127.18.100.65:39751
I20260130 02:59:05.342432 22292 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.100.65:39751 every 8 connection(s)
I20260130 02:59:05.345767 22140 server_base.cc:1192] Dumped server information to /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1769741859011789-18833-0/minicluster-data/ts-0/data/info.pb
I20260130 02:59:05.354658 18833 external_mini_cluster.cc:1428] Started /tmp/dist-test-tasktGe2wI/build/tsan/bin/kudu as pid 22140
I20260130 02:59:05.355159 18833 external_mini_cluster.cc:1442] Reading /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1769741859011789-18833-0/minicluster-data/ts-0/wal/instance
I20260130 02:59:05.390465 18833 external_mini_cluster.cc:1366] Running /tmp/dist-test-tasktGe2wI/build/tsan/bin/kudu
/tmp/dist-test-tasktGe2wI/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1769741859011789-18833-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1769741859011789-18833-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1769741859011789-18833-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1769741859011789-18833-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.100.66:0
--local_ip_for_outbound_sockets=127.18.100.66
--webserver_interface=127.18.100.66
--webserver_port=0
--tserver_master_addrs=127.18.100.126:42077,127.18.100.125:39157,127.18.100.124:34987
--builtin_ntp_servers=127.18.100.84:33971
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20260130 02:59:05.466666 22293 heartbeater.cc:344] Connected to a master server at 127.18.100.125:39157
I20260130 02:59:05.467296 22293 heartbeater.cc:461] Registering TS with master...
I20260130 02:59:05.468758 22293 heartbeater.cc:507] Master 127.18.100.125:39157 requested a full tablet report, sending...
I20260130 02:59:05.476567 21961 ts_manager.cc:194] Registered new tserver with Master: 111215cc989f4203a6e1517f2291d66f (127.18.100.65:39751)
I20260130 02:59:05.499835 22294 heartbeater.cc:344] Connected to a master server at 127.18.100.124:34987
I20260130 02:59:05.500226 22294 heartbeater.cc:461] Registering TS with master...
I20260130 02:59:05.501144 22294 heartbeater.cc:507] Master 127.18.100.124:34987 requested a full tablet report, sending...
I20260130 02:59:05.503759 22042 ts_manager.cc:194] Registered new tserver with Master: 111215cc989f4203a6e1517f2291d66f (127.18.100.65:39751)
I20260130 02:59:05.504645 22298 heartbeater.cc:344] Connected to a master server at 127.18.100.126:42077
I20260130 02:59:05.504997 22298 heartbeater.cc:461] Registering TS with master...
I20260130 02:59:05.505844 22298 heartbeater.cc:507] Master 127.18.100.126:42077 requested a full tablet report, sending...
I20260130 02:59:05.506413 22042 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.18.100.65:50571
I20260130 02:59:05.509610 21887 ts_manager.cc:194] Registered new tserver with Master: 111215cc989f4203a6e1517f2291d66f (127.18.100.65:39751)
W20260130 02:59:05.796037 22297 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260130 02:59:05.796504 22297 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260130 02:59:05.797029 22297 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260130 02:59:05.829728 22297 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260130 02:59:05.830681 22297 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.100.66
I20260130 02:59:05.866178 22297 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.100.84:33971
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1769741859011789-18833-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1769741859011789-18833-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.100.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1769741859011789-18833-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.100.66
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.18.100.126:42077,127.18.100.125:39157,127.18.100.124:34987
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.100.66
--log_dir=/tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1769741859011789-18833-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type FASTDEBUG
built by None at 30 Jan 2026 02:43:21 UTC on 5fd53c4cbb9d
build id 10257
TSAN enabled
I20260130 02:59:05.867527 22297 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260130 02:59:05.869077 22297 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260130 02:59:05.883960 22310 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260130 02:59:06.010545 22294 heartbeater.cc:499] Master 127.18.100.124:34987 was elected leader, sending a full tablet report...
I20260130 02:59:06.517338 22315 raft_consensus.cc:493] T 00000000000000000000000000000000 P b01057b6c5ff4fc4b07253801881fe13 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 342be83f20a648fd849c7d23454fba08)
I20260130 02:59:06.517899 22315 raft_consensus.cc:515] T 00000000000000000000000000000000 P b01057b6c5ff4fc4b07253801881fe13 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22dedcf69e00438999d3a5fedde57324" member_type: VOTER last_known_addr { host: "127.18.100.126" port: 42077 } } peers { permanent_uuid: "b01057b6c5ff4fc4b07253801881fe13" member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39157 } } peers { permanent_uuid: "342be83f20a648fd849c7d23454fba08" member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34987 } }
I20260130 02:59:06.519953 22315 leader_election.cc:290] T 00000000000000000000000000000000 P b01057b6c5ff4fc4b07253801881fe13 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 22dedcf69e00438999d3a5fedde57324 (127.18.100.126:42077), 342be83f20a648fd849c7d23454fba08 (127.18.100.124:34987)
I20260130 02:59:06.520619 21895 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b01057b6c5ff4fc4b07253801881fe13" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "22dedcf69e00438999d3a5fedde57324" is_pre_election: true
I20260130 02:59:06.521013 22052 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b01057b6c5ff4fc4b07253801881fe13" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "342be83f20a648fd849c7d23454fba08" is_pre_election: true
I20260130 02:59:06.522511 21946 leader_election.cc:304] T 00000000000000000000000000000000 P b01057b6c5ff4fc4b07253801881fe13 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b01057b6c5ff4fc4b07253801881fe13; no voters: 22dedcf69e00438999d3a5fedde57324, 342be83f20a648fd849c7d23454fba08
I20260130 02:59:06.523355 22315 raft_consensus.cc:2749] T 00000000000000000000000000000000 P b01057b6c5ff4fc4b07253801881fe13 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260130 02:59:05.887550 22309 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260130 02:59:05.888809 22297 server_base.cc:1060] running on GCE node
W20260130 02:59:05.890105 22312 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260130 02:59:06.998208 22297 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260130 02:59:07.000519 22297 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260130 02:59:07.001899 22297 hybrid_clock.cc:648] HybridClock initialized: now 1769741947001865 us; error 41 us; skew 500 ppm
I20260130 02:59:07.002779 22297 server_base.cc:860] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260130 02:59:07.009272 22297 webserver.cc:492] Webserver started at http://127.18.100.66:40717/ using document root <none> and password file <none>
I20260130 02:59:07.010346 22297 fs_manager.cc:362] Metadata directory not provided
I20260130 02:59:07.010540 22297 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260130 02:59:07.011153 22297 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260130 02:59:07.015611 22297 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1769741859011789-18833-0/minicluster-data/ts-1/data/instance:
uuid: "dd32a35972b347b89ceaaad2455aef5a"
format_stamp: "Formatted at 2026-01-30 02:59:07 on dist-test-slave-mn07"
I20260130 02:59:07.016711 22297 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1769741859011789-18833-0/minicluster-data/ts-1/wal/instance:
uuid: "dd32a35972b347b89ceaaad2455aef5a"
format_stamp: "Formatted at 2026-01-30 02:59:07 on dist-test-slave-mn07"
I20260130 02:59:07.025218 22297 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.010s sys 0.000s
I20260130 02:59:07.031649 22320 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260130 02:59:07.032763 22297 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20260130 02:59:07.033077 22297 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1769741859011789-18833-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1769741859011789-18833-0/minicluster-data/ts-1/wal
uuid: "dd32a35972b347b89ceaaad2455aef5a"
format_stamp: "Formatted at 2026-01-30 02:59:07 on dist-test-slave-mn07"
I20260130 02:59:07.033406 22297 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1769741859011789-18833-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1769741859011789-18833-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1769741859011789-18833-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260130 02:59:07.081736 22297 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260130 02:59:07.083459 22297 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260130 02:59:07.083871 22297 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260130 02:59:07.086796 22297 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260130 02:59:07.091032 22297 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260130 02:59:07.091251 22297 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260130 02:59:07.091451 22297 ts_tablet_manager.cc:616] Registered 0 tablets
I20260130 02:59:07.091585 22297 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260130 02:59:07.126431 22349 raft_consensus.cc:493] T 00000000000000000000000000000000 P b01057b6c5ff4fc4b07253801881fe13 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 342be83f20a648fd849c7d23454fba08)
I20260130 02:59:07.126967 22349 raft_consensus.cc:515] T 00000000000000000000000000000000 P b01057b6c5ff4fc4b07253801881fe13 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22dedcf69e00438999d3a5fedde57324" member_type: VOTER last_known_addr { host: "127.18.100.126" port: 42077 } } peers { permanent_uuid: "b01057b6c5ff4fc4b07253801881fe13" member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39157 } } peers { permanent_uuid: "342be83f20a648fd849c7d23454fba08" member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34987 } }
I20260130 02:59:07.129714 22349 leader_election.cc:290] T 00000000000000000000000000000000 P b01057b6c5ff4fc4b07253801881fe13 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 22dedcf69e00438999d3a5fedde57324 (127.18.100.126:42077), 342be83f20a648fd849c7d23454fba08 (127.18.100.124:34987)
I20260130 02:59:07.129920 21895 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b01057b6c5ff4fc4b07253801881fe13" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "22dedcf69e00438999d3a5fedde57324" is_pre_election: true
I20260130 02:59:07.130105 22052 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b01057b6c5ff4fc4b07253801881fe13" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "342be83f20a648fd849c7d23454fba08" is_pre_election: true
I20260130 02:59:07.131759 21946 leader_election.cc:304] T 00000000000000000000000000000000 P b01057b6c5ff4fc4b07253801881fe13 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b01057b6c5ff4fc4b07253801881fe13; no voters: 22dedcf69e00438999d3a5fedde57324, 342be83f20a648fd849c7d23454fba08
I20260130 02:59:07.132562 22349 raft_consensus.cc:2749] T 00000000000000000000000000000000 P b01057b6c5ff4fc4b07253801881fe13 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260130 02:59:07.169482 22386 raft_consensus.cc:493] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 342be83f20a648fd849c7d23454fba08)
I20260130 02:59:07.170190 22386 raft_consensus.cc:515] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22dedcf69e00438999d3a5fedde57324" member_type: VOTER last_known_addr { host: "127.18.100.126" port: 42077 } } peers { permanent_uuid: "b01057b6c5ff4fc4b07253801881fe13" member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39157 } } peers { permanent_uuid: "342be83f20a648fd849c7d23454fba08" member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34987 } }
I20260130 02:59:07.173671 21971 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "22dedcf69e00438999d3a5fedde57324" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b01057b6c5ff4fc4b07253801881fe13" is_pre_election: true
I20260130 02:59:07.174403 22386 leader_election.cc:290] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b01057b6c5ff4fc4b07253801881fe13 (127.18.100.125:39157), 342be83f20a648fd849c7d23454fba08 (127.18.100.124:34987)
I20260130 02:59:07.175173 22052 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "22dedcf69e00438999d3a5fedde57324" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "342be83f20a648fd849c7d23454fba08" is_pre_election: true
I20260130 02:59:07.176575 21872 leader_election.cc:304] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22dedcf69e00438999d3a5fedde57324; no voters: 342be83f20a648fd849c7d23454fba08, b01057b6c5ff4fc4b07253801881fe13
I20260130 02:59:07.177354 22386 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260130 02:59:07.233814 22297 rpc_server.cc:307] RPC server started. Bound to: 127.18.100.66:39781
I20260130 02:59:07.233843 22436 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.100.66:39781 every 8 connection(s)
I20260130 02:59:07.236462 22297 server_base.cc:1192] Dumped server information to /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1769741859011789-18833-0/minicluster-data/ts-1/data/info.pb
I20260130 02:59:07.244120 18833 external_mini_cluster.cc:1428] Started /tmp/dist-test-tasktGe2wI/build/tsan/bin/kudu as pid 22297
I20260130 02:59:07.244648 18833 external_mini_cluster.cc:1442] Reading /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1769741859011789-18833-0/minicluster-data/ts-1/wal/instance
I20260130 02:59:07.265885 18833 external_mini_cluster.cc:1366] Running /tmp/dist-test-tasktGe2wI/build/tsan/bin/kudu
/tmp/dist-test-tasktGe2wI/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1769741859011789-18833-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1769741859011789-18833-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1769741859011789-18833-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1769741859011789-18833-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.100.67:0
--local_ip_for_outbound_sockets=127.18.100.67
--webserver_interface=127.18.100.67
--webserver_port=0
--tserver_master_addrs=127.18.100.126:42077,127.18.100.125:39157,127.18.100.124:34987
--builtin_ntp_servers=127.18.100.84:33971
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20260130 02:59:07.308151 22439 heartbeater.cc:344] Connected to a master server at 127.18.100.126:42077
I20260130 02:59:07.308589 22437 heartbeater.cc:344] Connected to a master server at 127.18.100.125:39157
I20260130 02:59:07.308698 22439 heartbeater.cc:461] Registering TS with master...
I20260130 02:59:07.308892 22438 heartbeater.cc:344] Connected to a master server at 127.18.100.124:34987
I20260130 02:59:07.308969 22437 heartbeater.cc:461] Registering TS with master...
I20260130 02:59:07.309214 22438 heartbeater.cc:461] Registering TS with master...
I20260130 02:59:07.310195 22439 heartbeater.cc:507] Master 127.18.100.126:42077 requested a full tablet report, sending...
I20260130 02:59:07.310253 22438 heartbeater.cc:507] Master 127.18.100.124:34987 requested a full tablet report, sending...
I20260130 02:59:07.310760 22437 heartbeater.cc:507] Master 127.18.100.125:39157 requested a full tablet report, sending...
I20260130 02:59:07.313025 22042 ts_manager.cc:194] Registered new tserver with Master: dd32a35972b347b89ceaaad2455aef5a (127.18.100.66:39781)
I20260130 02:59:07.313433 21887 ts_manager.cc:194] Registered new tserver with Master: dd32a35972b347b89ceaaad2455aef5a (127.18.100.66:39781)
I20260130 02:59:07.313796 21961 ts_manager.cc:194] Registered new tserver with Master: dd32a35972b347b89ceaaad2455aef5a (127.18.100.66:39781)
I20260130 02:59:07.314810 22042 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.18.100.66:58625
W20260130 02:59:07.611224 22447 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260130 02:59:07.611697 22447 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260130 02:59:07.612282 22447 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260130 02:59:07.644882 22447 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260130 02:59:07.645823 22447 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.100.67
I20260130 02:59:07.676077 22452 raft_consensus.cc:493] T 00000000000000000000000000000000 P b01057b6c5ff4fc4b07253801881fe13 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 342be83f20a648fd849c7d23454fba08)
I20260130 02:59:07.676654 22452 raft_consensus.cc:515] T 00000000000000000000000000000000 P b01057b6c5ff4fc4b07253801881fe13 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22dedcf69e00438999d3a5fedde57324" member_type: VOTER last_known_addr { host: "127.18.100.126" port: 42077 } } peers { permanent_uuid: "b01057b6c5ff4fc4b07253801881fe13" member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39157 } } peers { permanent_uuid: "342be83f20a648fd849c7d23454fba08" member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34987 } }
I20260130 02:59:07.678436 22452 leader_election.cc:290] T 00000000000000000000000000000000 P b01057b6c5ff4fc4b07253801881fe13 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 22dedcf69e00438999d3a5fedde57324 (127.18.100.126:42077), 342be83f20a648fd849c7d23454fba08 (127.18.100.124:34987)
I20260130 02:59:07.679294 21895 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b01057b6c5ff4fc4b07253801881fe13" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "22dedcf69e00438999d3a5fedde57324" is_pre_election: true
I20260130 02:59:07.679540 22052 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b01057b6c5ff4fc4b07253801881fe13" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "342be83f20a648fd849c7d23454fba08" is_pre_election: true
I20260130 02:59:07.681032 21946 leader_election.cc:304] T 00000000000000000000000000000000 P b01057b6c5ff4fc4b07253801881fe13 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b01057b6c5ff4fc4b07253801881fe13; no voters: 22dedcf69e00438999d3a5fedde57324, 342be83f20a648fd849c7d23454fba08
I20260130 02:59:07.681629 22452 raft_consensus.cc:2749] T 00000000000000000000000000000000 P b01057b6c5ff4fc4b07253801881fe13 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260130 02:59:07.686079 22447 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.100.84:33971
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1769741859011789-18833-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1769741859011789-18833-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.100.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1769741859011789-18833-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.100.67
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.18.100.126:42077,127.18.100.125:39157,127.18.100.124:34987
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.100.67
--log_dir=/tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1769741859011789-18833-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type FASTDEBUG
built by None at 30 Jan 2026 02:43:21 UTC on 5fd53c4cbb9d
build id 10257
TSAN enabled
I20260130 02:59:07.687424 22447 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260130 02:59:07.689014 22447 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260130 02:59:07.701328 22454 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260130 02:59:07.777230 22460 raft_consensus.cc:493] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 342be83f20a648fd849c7d23454fba08)
I20260130 02:59:07.777657 22460 raft_consensus.cc:515] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22dedcf69e00438999d3a5fedde57324" member_type: VOTER last_known_addr { host: "127.18.100.126" port: 42077 } } peers { permanent_uuid: "b01057b6c5ff4fc4b07253801881fe13" member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39157 } } peers { permanent_uuid: "342be83f20a648fd849c7d23454fba08" member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34987 } }
I20260130 02:59:07.779177 22460 leader_election.cc:290] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b01057b6c5ff4fc4b07253801881fe13 (127.18.100.125:39157), 342be83f20a648fd849c7d23454fba08 (127.18.100.124:34987)
I20260130 02:59:07.780304 22052 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "22dedcf69e00438999d3a5fedde57324" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "342be83f20a648fd849c7d23454fba08" is_pre_election: true
I20260130 02:59:07.780521 21971 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "22dedcf69e00438999d3a5fedde57324" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b01057b6c5ff4fc4b07253801881fe13" is_pre_election: true
I20260130 02:59:07.782614 21874 leader_election.cc:304] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22dedcf69e00438999d3a5fedde57324; no voters: 342be83f20a648fd849c7d23454fba08, b01057b6c5ff4fc4b07253801881fe13
I20260130 02:59:07.783496 22460 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260130 02:59:07.818292 22438 heartbeater.cc:499] Master 127.18.100.124:34987 was elected leader, sending a full tablet report...
I20260130 02:59:08.363657 22461 raft_consensus.cc:493] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 342be83f20a648fd849c7d23454fba08)
I20260130 02:59:08.364235 22461 raft_consensus.cc:515] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22dedcf69e00438999d3a5fedde57324" member_type: VOTER last_known_addr { host: "127.18.100.126" port: 42077 } } peers { permanent_uuid: "b01057b6c5ff4fc4b07253801881fe13" member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39157 } } peers { permanent_uuid: "342be83f20a648fd849c7d23454fba08" member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34987 } }
I20260130 02:59:08.367835 21971 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "22dedcf69e00438999d3a5fedde57324" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b01057b6c5ff4fc4b07253801881fe13" is_pre_election: true
I20260130 02:59:08.368269 22052 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "22dedcf69e00438999d3a5fedde57324" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "342be83f20a648fd849c7d23454fba08" is_pre_election: true
I20260130 02:59:08.369550 22461 leader_election.cc:290] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b01057b6c5ff4fc4b07253801881fe13 (127.18.100.125:39157), 342be83f20a648fd849c7d23454fba08 (127.18.100.124:34987)
I20260130 02:59:08.370874 21874 leader_election.cc:304] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22dedcf69e00438999d3a5fedde57324; no voters: 342be83f20a648fd849c7d23454fba08, b01057b6c5ff4fc4b07253801881fe13
I20260130 02:59:08.371848 22461 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260130 02:59:08.619935 22462 raft_consensus.cc:493] T 00000000000000000000000000000000 P b01057b6c5ff4fc4b07253801881fe13 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 342be83f20a648fd849c7d23454fba08)
I20260130 02:59:08.620682 22462 raft_consensus.cc:515] T 00000000000000000000000000000000 P b01057b6c5ff4fc4b07253801881fe13 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22dedcf69e00438999d3a5fedde57324" member_type: VOTER last_known_addr { host: "127.18.100.126" port: 42077 } } peers { permanent_uuid: "b01057b6c5ff4fc4b07253801881fe13" member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39157 } } peers { permanent_uuid: "342be83f20a648fd849c7d23454fba08" member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34987 } }
I20260130 02:59:08.622989 22462 leader_election.cc:290] T 00000000000000000000000000000000 P b01057b6c5ff4fc4b07253801881fe13 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 22dedcf69e00438999d3a5fedde57324 (127.18.100.126:42077), 342be83f20a648fd849c7d23454fba08 (127.18.100.124:34987)
I20260130 02:59:08.624228 22052 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b01057b6c5ff4fc4b07253801881fe13" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "342be83f20a648fd849c7d23454fba08" is_pre_election: true
I20260130 02:59:08.624405 21895 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b01057b6c5ff4fc4b07253801881fe13" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "22dedcf69e00438999d3a5fedde57324" is_pre_election: true
I20260130 02:59:08.626214 21947 leader_election.cc:304] T 00000000000000000000000000000000 P b01057b6c5ff4fc4b07253801881fe13 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b01057b6c5ff4fc4b07253801881fe13; no voters: 22dedcf69e00438999d3a5fedde57324, 342be83f20a648fd849c7d23454fba08
I20260130 02:59:08.627077 22462 raft_consensus.cc:2749] T 00000000000000000000000000000000 P b01057b6c5ff4fc4b07253801881fe13 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260130 02:59:07.703029 22455 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260130 02:59:08.875509 22457 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260130 02:59:08.878079 22456 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection timed out after 1176 milliseconds
I20260130 02:59:08.878211 22447 server_base.cc:1055] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20260130 02:59:08.879469 22447 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260130 02:59:08.881733 22447 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260130 02:59:08.883136 22447 hybrid_clock.cc:648] HybridClock initialized: now 1769741948883104 us; error 41 us; skew 500 ppm
I20260130 02:59:08.883927 22447 server_base.cc:860] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260130 02:59:08.890496 22447 webserver.cc:492] Webserver started at http://127.18.100.67:44847/ using document root <none> and password file <none>
I20260130 02:59:08.891455 22447 fs_manager.cc:362] Metadata directory not provided
I20260130 02:59:08.891680 22447 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260130 02:59:08.892182 22447 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260130 02:59:08.896775 22447 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1769741859011789-18833-0/minicluster-data/ts-2/data/instance:
uuid: "789dbb4a27204ffbb68d7ae40f8f5e4a"
format_stamp: "Formatted at 2026-01-30 02:59:08 on dist-test-slave-mn07"
I20260130 02:59:08.897985 22447 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1769741859011789-18833-0/minicluster-data/ts-2/wal/instance:
uuid: "789dbb4a27204ffbb68d7ae40f8f5e4a"
format_stamp: "Formatted at 2026-01-30 02:59:08 on dist-test-slave-mn07"
I20260130 02:59:08.905409 22447 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.006s sys 0.000s
I20260130 02:59:08.911221 22467 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260130 02:59:08.912437 22447 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20260130 02:59:08.912758 22447 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1769741859011789-18833-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1769741859011789-18833-0/minicluster-data/ts-2/wal
uuid: "789dbb4a27204ffbb68d7ae40f8f5e4a"
format_stamp: "Formatted at 2026-01-30 02:59:08 on dist-test-slave-mn07"
I20260130 02:59:08.913091 22447 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1769741859011789-18833-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1769741859011789-18833-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1769741859011789-18833-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260130 02:59:08.960183 22447 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260130 02:59:08.961717 22447 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260130 02:59:08.962164 22447 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260130 02:59:08.964975 22447 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260130 02:59:08.969358 22447 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260130 02:59:08.969584 22447 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260130 02:59:08.969866 22447 ts_tablet_manager.cc:616] Registered 0 tablets
I20260130 02:59:08.970083 22447 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260130 02:59:09.109844 22447 rpc_server.cc:307] RPC server started. Bound to: 127.18.100.67:41825
I20260130 02:59:09.109938 22579 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.100.67:41825 every 8 connection(s)
I20260130 02:59:09.112360 22447 server_base.cc:1192] Dumped server information to /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1769741859011789-18833-0/minicluster-data/ts-2/data/info.pb
I20260130 02:59:09.114341 18833 external_mini_cluster.cc:1428] Started /tmp/dist-test-tasktGe2wI/build/tsan/bin/kudu as pid 22447
I20260130 02:59:09.114897 18833 external_mini_cluster.cc:1442] Reading /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1769741859011789-18833-0/minicluster-data/ts-2/wal/instance
I20260130 02:59:09.153790 22581 heartbeater.cc:344] Connected to a master server at 127.18.100.124:34987
I20260130 02:59:09.154331 22581 heartbeater.cc:461] Registering TS with master...
I20260130 02:59:09.154536 22580 heartbeater.cc:344] Connected to a master server at 127.18.100.125:39157
I20260130 02:59:09.154911 22582 heartbeater.cc:344] Connected to a master server at 127.18.100.126:42077
I20260130 02:59:09.154928 22580 heartbeater.cc:461] Registering TS with master...
I20260130 02:59:09.155256 22582 heartbeater.cc:461] Registering TS with master...
I20260130 02:59:09.155793 22581 heartbeater.cc:507] Master 127.18.100.124:34987 requested a full tablet report, sending...
I20260130 02:59:09.156049 22582 heartbeater.cc:507] Master 127.18.100.126:42077 requested a full tablet report, sending...
I20260130 02:59:09.156112 22580 heartbeater.cc:507] Master 127.18.100.125:39157 requested a full tablet report, sending...
I20260130 02:59:09.158857 21961 ts_manager.cc:194] Registered new tserver with Master: 789dbb4a27204ffbb68d7ae40f8f5e4a (127.18.100.67:41825)
I20260130 02:59:09.159010 21887 ts_manager.cc:194] Registered new tserver with Master: 789dbb4a27204ffbb68d7ae40f8f5e4a (127.18.100.67:41825)
I20260130 02:59:09.159137 22042 ts_manager.cc:194] Registered new tserver with Master: 789dbb4a27204ffbb68d7ae40f8f5e4a (127.18.100.67:41825)
I20260130 02:59:09.160801 22042 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.18.100.67:42867
I20260130 02:59:09.166466 18833 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20260130 02:59:09.182130 22594 raft_consensus.cc:493] T 00000000000000000000000000000000 P b01057b6c5ff4fc4b07253801881fe13 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 342be83f20a648fd849c7d23454fba08)
I20260130 02:59:09.182814 22594 raft_consensus.cc:515] T 00000000000000000000000000000000 P b01057b6c5ff4fc4b07253801881fe13 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22dedcf69e00438999d3a5fedde57324" member_type: VOTER last_known_addr { host: "127.18.100.126" port: 42077 } } peers { permanent_uuid: "b01057b6c5ff4fc4b07253801881fe13" member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39157 } } peers { permanent_uuid: "342be83f20a648fd849c7d23454fba08" member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34987 } }
I20260130 02:59:09.184885 22594 leader_election.cc:290] T 00000000000000000000000000000000 P b01057b6c5ff4fc4b07253801881fe13 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 22dedcf69e00438999d3a5fedde57324 (127.18.100.126:42077), 342be83f20a648fd849c7d23454fba08 (127.18.100.124:34987)
I20260130 02:59:09.186045 22052 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b01057b6c5ff4fc4b07253801881fe13" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "342be83f20a648fd849c7d23454fba08" is_pre_election: true
I20260130 02:59:09.186406 21895 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b01057b6c5ff4fc4b07253801881fe13" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "22dedcf69e00438999d3a5fedde57324" is_pre_election: true
I20260130 02:59:09.188269 21947 leader_election.cc:304] T 00000000000000000000000000000000 P b01057b6c5ff4fc4b07253801881fe13 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b01057b6c5ff4fc4b07253801881fe13; no voters: 22dedcf69e00438999d3a5fedde57324, 342be83f20a648fd849c7d23454fba08
I20260130 02:59:09.193881 22594 raft_consensus.cc:2749] T 00000000000000000000000000000000 P b01057b6c5ff4fc4b07253801881fe13 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260130 02:59:09.236560 22042 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59992:
name: "default.Table_de96622ce9794752adfc14fb9ceb4559"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260130 02:59:09.239661 22040 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59992:
name: "default.Table_e3b3683b36e144c2b33dd88b292bcb23"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260130 02:59:09.239964 22039 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59992:
name: "default.Table_d6f2cd9cd3b54168b14f1cacabbc74f0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260130 02:59:09.236665 22041 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59992:
name: "default.Table_62e153a4e7844d97863f32a6e87da70b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260130 02:59:09.249396 22042 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_de96622ce9794752adfc14fb9ceb4559 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260130 02:59:09.258772 22040 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_e3b3683b36e144c2b33dd88b292bcb23 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260130 02:59:09.258950 22039 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_d6f2cd9cd3b54168b14f1cacabbc74f0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260130 02:59:09.260051 22041 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_62e153a4e7844d97863f32a6e87da70b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260130 02:59:09.282310 18833 external_mini_cluster.cc:1658] Killing /tmp/dist-test-tasktGe2wI/build/tsan/bin/kudu with pid 21997
W20260130 02:59:09.316859 22592 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.18.100.124:34987)
I20260130 02:59:09.337965 22617 raft_consensus.cc:493] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 342be83f20a648fd849c7d23454fba08)
W20260130 02:59:09.338555 22438 heartbeater.cc:646] Failed to heartbeat to 127.18.100.124:34987 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.18.100.124:34987: connect: Connection refused (error 111)
I20260130 02:59:09.338543 22617 raft_consensus.cc:515] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22dedcf69e00438999d3a5fedde57324" member_type: VOTER last_known_addr { host: "127.18.100.126" port: 42077 } } peers { permanent_uuid: "b01057b6c5ff4fc4b07253801881fe13" member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39157 } } peers { permanent_uuid: "342be83f20a648fd849c7d23454fba08" member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34987 } }
I20260130 02:59:09.340708 22617 leader_election.cc:290] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b01057b6c5ff4fc4b07253801881fe13 (127.18.100.125:39157), 342be83f20a648fd849c7d23454fba08 (127.18.100.124:34987)
I20260130 02:59:09.341878 21971 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "22dedcf69e00438999d3a5fedde57324" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b01057b6c5ff4fc4b07253801881fe13" is_pre_election: true
I20260130 02:59:09.342535 21971 raft_consensus.cc:2468] T 00000000000000000000000000000000 P b01057b6c5ff4fc4b07253801881fe13 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22dedcf69e00438999d3a5fedde57324 in term 1.
I20260130 02:59:09.344012 21874 leader_election.cc:304] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22dedcf69e00438999d3a5fedde57324, b01057b6c5ff4fc4b07253801881fe13; no voters:
I20260130 02:59:09.344895 22617 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260130 02:59:09.345327 22617 raft_consensus.cc:493] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 342be83f20a648fd849c7d23454fba08)
I20260130 02:59:09.346084 22617 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [term 1 FOLLOWER]: Advancing to term 2
I20260130 02:59:09.353780 22617 raft_consensus.cc:515] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22dedcf69e00438999d3a5fedde57324" member_type: VOTER last_known_addr { host: "127.18.100.126" port: 42077 } } peers { permanent_uuid: "b01057b6c5ff4fc4b07253801881fe13" member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39157 } } peers { permanent_uuid: "342be83f20a648fd849c7d23454fba08" member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34987 } }
I20260130 02:59:09.358026 21971 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "22dedcf69e00438999d3a5fedde57324" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b01057b6c5ff4fc4b07253801881fe13"
I20260130 02:59:09.358645 21971 raft_consensus.cc:3060] T 00000000000000000000000000000000 P b01057b6c5ff4fc4b07253801881fe13 [term 1 FOLLOWER]: Advancing to term 2
I20260130 02:59:09.360729 22617 leader_election.cc:290] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [CANDIDATE]: Term 2 election: Requested vote from peers b01057b6c5ff4fc4b07253801881fe13 (127.18.100.125:39157), 342be83f20a648fd849c7d23454fba08 (127.18.100.124:34987)
I20260130 02:59:09.368225 21971 raft_consensus.cc:2468] T 00000000000000000000000000000000 P b01057b6c5ff4fc4b07253801881fe13 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22dedcf69e00438999d3a5fedde57324 in term 2.
I20260130 02:59:09.370316 21874 leader_election.cc:304] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22dedcf69e00438999d3a5fedde57324, b01057b6c5ff4fc4b07253801881fe13; no voters:
I20260130 02:59:09.371415 22617 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [term 2 FOLLOWER]: Leader election won for term 2
W20260130 02:59:09.385910 21872 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.18.100.124:34987: connect: Connection refused (error 111) [suppressed 13 similar messages]
I20260130 02:59:09.390069 22617 raft_consensus.cc:697] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [term 2 LEADER]: Becoming Leader. State: Replica: 22dedcf69e00438999d3a5fedde57324, State: Running, Role: LEADER
I20260130 02:59:09.392253 22617 consensus_queue.cc:237] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22dedcf69e00438999d3a5fedde57324" member_type: VOTER last_known_addr { host: "127.18.100.126" port: 42077 } } peers { permanent_uuid: "b01057b6c5ff4fc4b07253801881fe13" member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39157 } } peers { permanent_uuid: "342be83f20a648fd849c7d23454fba08" member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34987 } }
W20260130 02:59:09.404882 21872 leader_election.cc:336] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 342be83f20a648fd849c7d23454fba08 (127.18.100.124:34987): Network error: Client connection negotiation failed: client connection to 127.18.100.124:34987: connect: Connection refused (error 111)
W20260130 02:59:09.409615 21872 leader_election.cc:336] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 342be83f20a648fd849c7d23454fba08 (127.18.100.124:34987): Network error: Client connection negotiation failed: client connection to 127.18.100.124:34987: connect: Connection refused (error 111)
I20260130 02:59:09.428224 22619 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 22dedcf69e00438999d3a5fedde57324. Latest consensus state: current_term: 2 leader_uuid: "22dedcf69e00438999d3a5fedde57324" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22dedcf69e00438999d3a5fedde57324" member_type: VOTER last_known_addr { host: "127.18.100.126" port: 42077 } } peers { permanent_uuid: "b01057b6c5ff4fc4b07253801881fe13" member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39157 } } peers { permanent_uuid: "342be83f20a648fd849c7d23454fba08" member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34987 } } }
I20260130 02:59:09.429207 22619 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [sys.catalog]: This master's current role is: LEADER
I20260130 02:59:09.435333 22622 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260130 02:59:09.445506 22622 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260130 02:59:09.450160 22622 catalog_manager.cc:1269] Loaded cluster ID: 40eb42871f464e95bd641ceacb567979
I20260130 02:59:09.450634 22622 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260130 02:59:09.456738 22622 catalog_manager.cc:1514] Loading token signing keys...
I20260130 02:59:09.461287 22622 catalog_manager.cc:6038] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324: Loaded TSK: 0
I20260130 02:59:09.462847 22622 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260130 02:59:09.468006 22622 catalog_manager.cc:1537] Loading latest processed Hive Metastore notification log event ID...
I20260130 02:59:09.472435 22622 catalog_manager.cc:5876] Last processed Hive Metastore notification event ID: -1
I20260130 02:59:09.486896 21887 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34990:
name: "default.Table_d6f2cd9cd3b54168b14f1cacabbc74f0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260130 02:59:09.491854 21886 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34990:
name: "default.Table_e3b3683b36e144c2b33dd88b292bcb23"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260130 02:59:09.496253 21884 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34990:
name: "default.Table_de96622ce9794752adfc14fb9ceb4559"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260130 02:59:09.494976 21885 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34990:
name: "default.Table_62e153a4e7844d97863f32a6e87da70b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260130 02:59:09.529100 21885 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_62e153a4e7844d97863f32a6e87da70b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260130 02:59:09.538134 21884 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_de96622ce9794752adfc14fb9ceb4559 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260130 02:59:09.542629 21886 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_e3b3683b36e144c2b33dd88b292bcb23 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260130 02:59:09.546844 21887 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_d6f2cd9cd3b54168b14f1cacabbc74f0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260130 02:59:09.553776 22294 heartbeater.cc:646] Failed to heartbeat to 127.18.100.124:34987 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.18.100.124:34987: connect: Connection refused (error 111)
I20260130 02:59:09.665223 22581 heartbeater.cc:499] Master 127.18.100.124:34987 was elected leader, sending a full tablet report...
W20260130 02:59:09.685211 22581 heartbeater.cc:646] Failed to heartbeat to 127.18.100.124:34987 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.18.100.124:34987: connect: Connection refused (error 111)
W20260130 02:59:09.857950 21872 consensus_peers.cc:597] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 -> Peer 342be83f20a648fd849c7d23454fba08 (127.18.100.124:34987): Couldn't send request to peer 342be83f20a648fd849c7d23454fba08. Status: Network error: Client connection negotiation failed: client connection to 127.18.100.124:34987: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260130 02:59:09.909713 21971 raft_consensus.cc:1275] T 00000000000000000000000000000000 P b01057b6c5ff4fc4b07253801881fe13 [term 2 FOLLOWER]: Refusing update from remote peer 22dedcf69e00438999d3a5fedde57324: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260130 02:59:09.911732 22619 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b01057b6c5ff4fc4b07253801881fe13" member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260130 02:59:09.933291 22627 sys_catalog.cc:455] T 00000000000000000000000000000000 P b01057b6c5ff4fc4b07253801881fe13 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 22dedcf69e00438999d3a5fedde57324. Latest consensus state: current_term: 2 leader_uuid: "22dedcf69e00438999d3a5fedde57324" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22dedcf69e00438999d3a5fedde57324" member_type: VOTER last_known_addr { host: "127.18.100.126" port: 42077 } } peers { permanent_uuid: "b01057b6c5ff4fc4b07253801881fe13" member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39157 } } peers { permanent_uuid: "342be83f20a648fd849c7d23454fba08" member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34987 } } }
I20260130 02:59:09.934000 22627 sys_catalog.cc:458] T 00000000000000000000000000000000 P b01057b6c5ff4fc4b07253801881fe13 [sys.catalog]: This master's current role is: FOLLOWER
I20260130 02:59:09.938350 21885 catalog_manager.cc:2174] created HMS catalog entry for table [id=dee95603eab3439b81355a006bc9f376]
I20260130 02:59:09.953876 22619 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "22dedcf69e00438999d3a5fedde57324" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22dedcf69e00438999d3a5fedde57324" member_type: VOTER last_known_addr { host: "127.18.100.126" port: 42077 } } peers { permanent_uuid: "b01057b6c5ff4fc4b07253801881fe13" member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39157 } } peers { permanent_uuid: "342be83f20a648fd849c7d23454fba08" member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34987 } } }
I20260130 02:59:09.954663 22619 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [sys.catalog]: This master's current role is: LEADER
I20260130 02:59:09.957160 22627 sys_catalog.cc:455] T 00000000000000000000000000000000 P b01057b6c5ff4fc4b07253801881fe13 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "22dedcf69e00438999d3a5fedde57324" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22dedcf69e00438999d3a5fedde57324" member_type: VOTER last_known_addr { host: "127.18.100.126" port: 42077 } } peers { permanent_uuid: "b01057b6c5ff4fc4b07253801881fe13" member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39157 } } peers { permanent_uuid: "342be83f20a648fd849c7d23454fba08" member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34987 } } }
I20260130 02:59:09.957835 22627 sys_catalog.cc:458] T 00000000000000000000000000000000 P b01057b6c5ff4fc4b07253801881fe13 [sys.catalog]: This master's current role is: FOLLOWER
I20260130 02:59:09.999672 21883 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34990:
table { table_name: "default.Table_62e153a4e7844d97863f32a6e87da70b" } new_table_name: "default.Table_9fdc4fc1d8a442389a5c9201bcd4ab84" modify_external_catalogs: true
I20260130 02:59:10.042853 22298 heartbeater.cc:499] Master 127.18.100.126:42077 was elected leader, sending a full tablet report...
I20260130 02:59:10.066773 22369 tablet_service.cc:1505] Processing CreateTablet for tablet 2e4e3542c9ad480bbc5c2960aa8a549d (DEFAULT_TABLE table=default.table_62e153a4e7844d97863f32a6e87da70b [id=dee95603eab3439b81355a006bc9f376]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:59:10.066773 22227 tablet_service.cc:1505] Processing CreateTablet for tablet 2e4e3542c9ad480bbc5c2960aa8a549d (DEFAULT_TABLE table=default.table_62e153a4e7844d97863f32a6e87da70b [id=dee95603eab3439b81355a006bc9f376]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:59:10.069121 22227 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2e4e3542c9ad480bbc5c2960aa8a549d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:59:10.074991 22515 tablet_service.cc:1505] Processing CreateTablet for tablet 2e4e3542c9ad480bbc5c2960aa8a549d (DEFAULT_TABLE table=default.table_62e153a4e7844d97863f32a6e87da70b [id=dee95603eab3439b81355a006bc9f376]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:59:10.076609 22369 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2e4e3542c9ad480bbc5c2960aa8a549d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:59:10.077672 22515 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2e4e3542c9ad480bbc5c2960aa8a549d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:59:10.116350 22642 tablet_bootstrap.cc:492] T 2e4e3542c9ad480bbc5c2960aa8a549d P 789dbb4a27204ffbb68d7ae40f8f5e4a: Bootstrap starting.
I20260130 02:59:10.124487 22641 tablet_bootstrap.cc:492] T 2e4e3542c9ad480bbc5c2960aa8a549d P 111215cc989f4203a6e1517f2291d66f: Bootstrap starting.
I20260130 02:59:10.132170 22641 tablet_bootstrap.cc:654] T 2e4e3542c9ad480bbc5c2960aa8a549d P 111215cc989f4203a6e1517f2291d66f: Neither blocks nor log segments found. Creating new log.
I20260130 02:59:10.134295 18833 external_mini_cluster.cc:1366] Running /tmp/dist-test-tasktGe2wI/build/tsan/bin/kudu
/tmp/dist-test-tasktGe2wI/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1769741859011789-18833-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1769741859011789-18833-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1769741859011789-18833-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1769741859011789-18833-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.100.124:34987
--webserver_interface=127.18.100.124
--webserver_port=34049
--builtin_ntp_servers=127.18.100.84:33971
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.100.126:42077,127.18.100.125:39157,127.18.100.124:34987
--hive_metastore_uris=thrift://127.0.0.1:42185
--hive_metastore_notification_log_poll_period_seconds=1
--enable_metadata_cleanup_for_deleted_tables_and_tablets=true
--catalog_manager_bg_task_wait_ms=10
--metadata_for_deleted_table_and_tablet_reserved_secs=0
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20260130 02:59:10.135165 22642 tablet_bootstrap.cc:654] T 2e4e3542c9ad480bbc5c2960aa8a549d P 789dbb4a27204ffbb68d7ae40f8f5e4a: Neither blocks nor log segments found. Creating new log.
I20260130 02:59:10.135594 22641 log.cc:826] T 2e4e3542c9ad480bbc5c2960aa8a549d P 111215cc989f4203a6e1517f2291d66f: Log is configured to *not* fsync() on all Append() calls
I20260130 02:59:10.137850 22642 log.cc:826] T 2e4e3542c9ad480bbc5c2960aa8a549d P 789dbb4a27204ffbb68d7ae40f8f5e4a: Log is configured to *not* fsync() on all Append() calls
I20260130 02:59:10.140408 22643 tablet_bootstrap.cc:492] T 2e4e3542c9ad480bbc5c2960aa8a549d P dd32a35972b347b89ceaaad2455aef5a: Bootstrap starting.
I20260130 02:59:10.148433 22641 tablet_bootstrap.cc:492] T 2e4e3542c9ad480bbc5c2960aa8a549d P 111215cc989f4203a6e1517f2291d66f: No bootstrap required, opened a new log
I20260130 02:59:10.149173 22641 ts_tablet_manager.cc:1403] T 2e4e3542c9ad480bbc5c2960aa8a549d P 111215cc989f4203a6e1517f2291d66f: Time spent bootstrapping tablet: real 0.025s user 0.016s sys 0.007s
I20260130 02:59:10.163473 22643 tablet_bootstrap.cc:654] T 2e4e3542c9ad480bbc5c2960aa8a549d P dd32a35972b347b89ceaaad2455aef5a: Neither blocks nor log segments found. Creating new log.
I20260130 02:59:10.167109 22642 tablet_bootstrap.cc:492] T 2e4e3542c9ad480bbc5c2960aa8a549d P 789dbb4a27204ffbb68d7ae40f8f5e4a: No bootstrap required, opened a new log
I20260130 02:59:10.167863 22642 ts_tablet_manager.cc:1403] T 2e4e3542c9ad480bbc5c2960aa8a549d P 789dbb4a27204ffbb68d7ae40f8f5e4a: Time spent bootstrapping tablet: real 0.052s user 0.011s sys 0.006s
I20260130 02:59:10.168562 21885 master_service.cc:954] ReplaceTablet: received request to replace tablet 2e4e3542c9ad480bbc5c2960aa8a549d from {username='slave'} at 127.0.0.1:34990
I20260130 02:59:10.170193 22643 log.cc:826] T 2e4e3542c9ad480bbc5c2960aa8a549d P dd32a35972b347b89ceaaad2455aef5a: Log is configured to *not* fsync() on all Append() calls
I20260130 02:59:10.170696 22582 heartbeater.cc:499] Master 127.18.100.126:42077 was elected leader, sending a full tablet report...
I20260130 02:59:10.173326 21882 master_service.cc:954] ReplaceTablet: received request to replace tablet 2e4e3542c9ad480bbc5c2960aa8a549d from {username='slave'} at 127.0.0.1:34990
I20260130 02:59:10.185007 21880 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34990:
name: "default.Table_91ef08c82eb0446fbb6164d207168099"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260130 02:59:10.205433 22643 tablet_bootstrap.cc:492] T 2e4e3542c9ad480bbc5c2960aa8a549d P dd32a35972b347b89ceaaad2455aef5a: No bootstrap required, opened a new log
I20260130 02:59:10.206141 22643 ts_tablet_manager.cc:1403] T 2e4e3542c9ad480bbc5c2960aa8a549d P dd32a35972b347b89ceaaad2455aef5a: Time spent bootstrapping tablet: real 0.066s user 0.018s sys 0.000s
I20260130 02:59:10.207023 21885 catalog_manager.cc:5941] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324: Sending DeleteTablet for 3 replicas of tablet 2e4e3542c9ad480bbc5c2960aa8a549d
I20260130 02:59:10.215624 22515 tablet_service.cc:1552] Processing DeleteTablet for tablet 2e4e3542c9ad480bbc5c2960aa8a549d with delete_type TABLET_DATA_DELETED (replaced by tablet ef0a82e66c1c4f5a95f2c7e74a1b85f0) from {username='slave'} at 127.0.0.1:53088
I20260130 02:59:10.220579 22369 tablet_service.cc:1552] Processing DeleteTablet for tablet 2e4e3542c9ad480bbc5c2960aa8a549d with delete_type TABLET_DATA_DELETED (replaced by tablet ef0a82e66c1c4f5a95f2c7e74a1b85f0) from {username='slave'} at 127.0.0.1:44082
I20260130 02:59:10.220901 22227 tablet_service.cc:1552] Processing DeleteTablet for tablet 2e4e3542c9ad480bbc5c2960aa8a549d with delete_type TABLET_DATA_DELETED (replaced by tablet ef0a82e66c1c4f5a95f2c7e74a1b85f0) from {username='slave'} at 127.0.0.1:53816
I20260130 02:59:10.227070 21885 catalog_manager.cc:6724] ReplaceTablet: tablet 2e4e3542c9ad480bbc5c2960aa8a549d deleted and replaced by tablet ef0a82e66c1c4f5a95f2c7e74a1b85f0
I20260130 02:59:10.216365 22641 raft_consensus.cc:359] T 2e4e3542c9ad480bbc5c2960aa8a549d P 111215cc989f4203a6e1517f2291d66f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "789dbb4a27204ffbb68d7ae40f8f5e4a" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 41825 } } peers { permanent_uuid: "dd32a35972b347b89ceaaad2455aef5a" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 39781 } } peers { permanent_uuid: "111215cc989f4203a6e1517f2291d66f" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 39751 } }
I20260130 02:59:10.229379 22641 raft_consensus.cc:385] T 2e4e3542c9ad480bbc5c2960aa8a549d P 111215cc989f4203a6e1517f2291d66f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:59:10.230237 22641 raft_consensus.cc:740] T 2e4e3542c9ad480bbc5c2960aa8a549d P 111215cc989f4203a6e1517f2291d66f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 111215cc989f4203a6e1517f2291d66f, State: Initialized, Role: FOLLOWER
W20260130 02:59:10.232266 21871 catalog_manager.cc:4965] TS 789dbb4a27204ffbb68d7ae40f8f5e4a (127.18.100.67:41825): delete failed for tablet 2e4e3542c9ad480bbc5c2960aa8a549d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2e4e3542c9ad480bbc5c2960aa8a549d already in progress: creating tablet
I20260130 02:59:10.232512 22641 consensus_queue.cc:260] T 2e4e3542c9ad480bbc5c2960aa8a549d P 111215cc989f4203a6e1517f2291d66f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "789dbb4a27204ffbb68d7ae40f8f5e4a" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 41825 } } peers { permanent_uuid: "dd32a35972b347b89ceaaad2455aef5a" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 39781 } } peers { permanent_uuid: "111215cc989f4203a6e1517f2291d66f" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 39751 } }
W20260130 02:59:10.243091 21872 catalog_manager.cc:4965] TS dd32a35972b347b89ceaaad2455aef5a (127.18.100.66:39781): delete failed for tablet 2e4e3542c9ad480bbc5c2960aa8a549d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2e4e3542c9ad480bbc5c2960aa8a549d already in progress: creating tablet
W20260130 02:59:10.244745 21871 catalog_manager.cc:4965] TS 111215cc989f4203a6e1517f2291d66f (127.18.100.65:39751): delete failed for tablet 2e4e3542c9ad480bbc5c2960aa8a549d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2e4e3542c9ad480bbc5c2960aa8a549d already in progress: creating tablet
I20260130 02:59:10.250114 22369 tablet_service.cc:1505] Processing CreateTablet for tablet ef0a82e66c1c4f5a95f2c7e74a1b85f0 (DEFAULT_TABLE table=default.table_62e153a4e7844d97863f32a6e87da70b [id=dee95603eab3439b81355a006bc9f376]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:59:10.251315 22369 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ef0a82e66c1c4f5a95f2c7e74a1b85f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:59:10.255327 22227 tablet_service.cc:1505] Processing CreateTablet for tablet ef0a82e66c1c4f5a95f2c7e74a1b85f0 (DEFAULT_TABLE table=default.table_62e153a4e7844d97863f32a6e87da70b [id=dee95603eab3439b81355a006bc9f376]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:59:10.259181 22227 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ef0a82e66c1c4f5a95f2c7e74a1b85f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:59:10.259784 22515 tablet_service.cc:1505] Processing CreateTablet for tablet ef0a82e66c1c4f5a95f2c7e74a1b85f0 (DEFAULT_TABLE table=default.table_62e153a4e7844d97863f32a6e87da70b [id=dee95603eab3439b81355a006bc9f376]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:59:10.263701 22515 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ef0a82e66c1c4f5a95f2c7e74a1b85f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:59:10.268072 22642 raft_consensus.cc:359] T 2e4e3542c9ad480bbc5c2960aa8a549d P 789dbb4a27204ffbb68d7ae40f8f5e4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "789dbb4a27204ffbb68d7ae40f8f5e4a" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 41825 } } peers { permanent_uuid: "dd32a35972b347b89ceaaad2455aef5a" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 39781 } } peers { permanent_uuid: "111215cc989f4203a6e1517f2291d66f" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 39751 } }
I20260130 02:59:10.273733 22641 ts_tablet_manager.cc:1434] T 2e4e3542c9ad480bbc5c2960aa8a549d P 111215cc989f4203a6e1517f2291d66f: Time spent starting tablet: real 0.124s user 0.027s sys 0.010s
W20260130 02:59:10.274022 22301 tablet.cc:2378] T 2e4e3542c9ad480bbc5c2960aa8a549d P 111215cc989f4203a6e1517f2291d66f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260130 02:59:10.281183 22642 raft_consensus.cc:385] T 2e4e3542c9ad480bbc5c2960aa8a549d P 789dbb4a27204ffbb68d7ae40f8f5e4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:59:10.285475 22226 tablet_service.cc:1552] Processing DeleteTablet for tablet 2e4e3542c9ad480bbc5c2960aa8a549d with delete_type TABLET_DATA_DELETED (replaced by tablet ef0a82e66c1c4f5a95f2c7e74a1b85f0) from {username='slave'} at 127.0.0.1:53816
I20260130 02:59:10.286725 22642 raft_consensus.cc:740] T 2e4e3542c9ad480bbc5c2960aa8a549d P 789dbb4a27204ffbb68d7ae40f8f5e4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 789dbb4a27204ffbb68d7ae40f8f5e4a, State: Initialized, Role: FOLLOWER
I20260130 02:59:10.289275 22641 tablet_bootstrap.cc:492] T ef0a82e66c1c4f5a95f2c7e74a1b85f0 P 111215cc989f4203a6e1517f2291d66f: Bootstrap starting.
I20260130 02:59:10.289775 22642 consensus_queue.cc:260] T 2e4e3542c9ad480bbc5c2960aa8a549d P 789dbb4a27204ffbb68d7ae40f8f5e4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "789dbb4a27204ffbb68d7ae40f8f5e4a" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 41825 } } peers { permanent_uuid: "dd32a35972b347b89ceaaad2455aef5a" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 39781 } } peers { permanent_uuid: "111215cc989f4203a6e1517f2291d66f" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 39751 } }
I20260130 02:59:10.292374 22651 tablet_replica.cc:333] T 2e4e3542c9ad480bbc5c2960aa8a549d P 111215cc989f4203a6e1517f2291d66f: stopping tablet replica
I20260130 02:59:10.295753 22651 raft_consensus.cc:2243] T 2e4e3542c9ad480bbc5c2960aa8a549d P 111215cc989f4203a6e1517f2291d66f [term 0 FOLLOWER]: Raft consensus shutting down.
I20260130 02:59:10.295370 22643 raft_consensus.cc:359] T 2e4e3542c9ad480bbc5c2960aa8a549d P dd32a35972b347b89ceaaad2455aef5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "789dbb4a27204ffbb68d7ae40f8f5e4a" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 41825 } } peers { permanent_uuid: "dd32a35972b347b89ceaaad2455aef5a" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 39781 } } peers { permanent_uuid: "111215cc989f4203a6e1517f2291d66f" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 39751 } }
I20260130 02:59:10.296530 22643 raft_consensus.cc:385] T 2e4e3542c9ad480bbc5c2960aa8a549d P dd32a35972b347b89ceaaad2455aef5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:59:10.296998 22643 raft_consensus.cc:740] T 2e4e3542c9ad480bbc5c2960aa8a549d P dd32a35972b347b89ceaaad2455aef5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd32a35972b347b89ceaaad2455aef5a, State: Initialized, Role: FOLLOWER
I20260130 02:59:10.298007 22651 raft_consensus.cc:2272] T 2e4e3542c9ad480bbc5c2960aa8a549d P 111215cc989f4203a6e1517f2291d66f [term 0 FOLLOWER]: Raft consensus is shut down!
I20260130 02:59:10.296109 22641 tablet_bootstrap.cc:654] T ef0a82e66c1c4f5a95f2c7e74a1b85f0 P 111215cc989f4203a6e1517f2291d66f: Neither blocks nor log segments found. Creating new log.
I20260130 02:59:10.298733 22643 consensus_queue.cc:260] T 2e4e3542c9ad480bbc5c2960aa8a549d P dd32a35972b347b89ceaaad2455aef5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "789dbb4a27204ffbb68d7ae40f8f5e4a" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 41825 } } peers { permanent_uuid: "dd32a35972b347b89ceaaad2455aef5a" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 39781 } } peers { permanent_uuid: "111215cc989f4203a6e1517f2291d66f" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 39751 } }
I20260130 02:59:10.301560 22651 ts_tablet_manager.cc:1916] T 2e4e3542c9ad480bbc5c2960aa8a549d P 111215cc989f4203a6e1517f2291d66f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260130 02:59:10.309136 22641 tablet_bootstrap.cc:492] T ef0a82e66c1c4f5a95f2c7e74a1b85f0 P 111215cc989f4203a6e1517f2291d66f: No bootstrap required, opened a new log
I20260130 02:59:10.309698 22641 ts_tablet_manager.cc:1403] T ef0a82e66c1c4f5a95f2c7e74a1b85f0 P 111215cc989f4203a6e1517f2291d66f: Time spent bootstrapping tablet: real 0.021s user 0.010s sys 0.001s
I20260130 02:59:10.312655 22641 raft_consensus.cc:359] T ef0a82e66c1c4f5a95f2c7e74a1b85f0 P 111215cc989f4203a6e1517f2291d66f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111215cc989f4203a6e1517f2291d66f" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 39751 } } peers { permanent_uuid: "dd32a35972b347b89ceaaad2455aef5a" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 39781 } } peers { permanent_uuid: "789dbb4a27204ffbb68d7ae40f8f5e4a" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 41825 } }
I20260130 02:59:10.313441 22641 raft_consensus.cc:385] T ef0a82e66c1c4f5a95f2c7e74a1b85f0 P 111215cc989f4203a6e1517f2291d66f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:59:10.313764 22641 raft_consensus.cc:740] T ef0a82e66c1c4f5a95f2c7e74a1b85f0 P 111215cc989f4203a6e1517f2291d66f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 111215cc989f4203a6e1517f2291d66f, State: Initialized, Role: FOLLOWER
I20260130 02:59:10.314493 22641 consensus_queue.cc:260] T ef0a82e66c1c4f5a95f2c7e74a1b85f0 P 111215cc989f4203a6e1517f2291d66f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111215cc989f4203a6e1517f2291d66f" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 39751 } } peers { permanent_uuid: "dd32a35972b347b89ceaaad2455aef5a" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 39781 } } peers { permanent_uuid: "789dbb4a27204ffbb68d7ae40f8f5e4a" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 41825 } }
I20260130 02:59:10.316679 22641 ts_tablet_manager.cc:1434] T ef0a82e66c1c4f5a95f2c7e74a1b85f0 P 111215cc989f4203a6e1517f2291d66f: Time spent starting tablet: real 0.007s user 0.002s sys 0.003s
I20260130 02:59:10.321754 22643 ts_tablet_manager.cc:1434] T 2e4e3542c9ad480bbc5c2960aa8a549d P dd32a35972b347b89ceaaad2455aef5a: Time spent starting tablet: real 0.115s user 0.029s sys 0.011s
I20260130 02:59:10.327840 22439 heartbeater.cc:499] Master 127.18.100.126:42077 was elected leader, sending a full tablet report...
I20260130 02:59:10.330533 22643 tablet_bootstrap.cc:492] T ef0a82e66c1c4f5a95f2c7e74a1b85f0 P dd32a35972b347b89ceaaad2455aef5a: Bootstrap starting.
I20260130 02:59:10.337069 22369 tablet_service.cc:1552] Processing DeleteTablet for tablet 2e4e3542c9ad480bbc5c2960aa8a549d with delete_type TABLET_DATA_DELETED (replaced by tablet ef0a82e66c1c4f5a95f2c7e74a1b85f0) from {username='slave'} at 127.0.0.1:44082
I20260130 02:59:10.337815 22643 tablet_bootstrap.cc:654] T ef0a82e66c1c4f5a95f2c7e74a1b85f0 P dd32a35972b347b89ceaaad2455aef5a: Neither blocks nor log segments found. Creating new log.
I20260130 02:59:10.338037 22650 tablet_replica.cc:333] T 2e4e3542c9ad480bbc5c2960aa8a549d P dd32a35972b347b89ceaaad2455aef5a: stopping tablet replica
I20260130 02:59:10.339138 22650 raft_consensus.cc:2243] T 2e4e3542c9ad480bbc5c2960aa8a549d P dd32a35972b347b89ceaaad2455aef5a [term 0 FOLLOWER]: Raft consensus shutting down.
I20260130 02:59:10.339774 22650 raft_consensus.cc:2272] T 2e4e3542c9ad480bbc5c2960aa8a549d P dd32a35972b347b89ceaaad2455aef5a [term 0 FOLLOWER]: Raft consensus is shut down!
I20260130 02:59:10.349421 22650 ts_tablet_manager.cc:1916] T 2e4e3542c9ad480bbc5c2960aa8a549d P dd32a35972b347b89ceaaad2455aef5a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260130 02:59:10.371600 22642 ts_tablet_manager.cc:1434] T 2e4e3542c9ad480bbc5c2960aa8a549d P 789dbb4a27204ffbb68d7ae40f8f5e4a: Time spent starting tablet: real 0.201s user 0.056s sys 0.011s
I20260130 02:59:10.376945 22514 tablet_service.cc:1552] Processing DeleteTablet for tablet 2e4e3542c9ad480bbc5c2960aa8a549d with delete_type TABLET_DATA_DELETED (replaced by tablet ef0a82e66c1c4f5a95f2c7e74a1b85f0) from {username='slave'} at 127.0.0.1:53088
W20260130 02:59:10.379549 21871 catalog_manager.cc:4965] TS 789dbb4a27204ffbb68d7ae40f8f5e4a (127.18.100.67:41825): delete failed for tablet 2e4e3542c9ad480bbc5c2960aa8a549d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2e4e3542c9ad480bbc5c2960aa8a549d already in progress: creating tablet
I20260130 02:59:10.392580 22642 tablet_bootstrap.cc:492] T ef0a82e66c1c4f5a95f2c7e74a1b85f0 P 789dbb4a27204ffbb68d7ae40f8f5e4a: Bootstrap starting.
W20260130 02:59:10.392683 22587 tablet.cc:2378] T 2e4e3542c9ad480bbc5c2960aa8a549d P 789dbb4a27204ffbb68d7ae40f8f5e4a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260130 02:59:10.398068 22515 tablet_service.cc:1552] Processing DeleteTablet for tablet 2e4e3542c9ad480bbc5c2960aa8a549d with delete_type TABLET_DATA_DELETED (replaced by tablet ef0a82e66c1c4f5a95f2c7e74a1b85f0) from {username='slave'} at 127.0.0.1:53088
I20260130 02:59:10.398535 22642 tablet_bootstrap.cc:654] T ef0a82e66c1c4f5a95f2c7e74a1b85f0 P 789dbb4a27204ffbb68d7ae40f8f5e4a: Neither blocks nor log segments found. Creating new log.
I20260130 02:59:10.399058 22649 tablet_replica.cc:333] T 2e4e3542c9ad480bbc5c2960aa8a549d P 789dbb4a27204ffbb68d7ae40f8f5e4a: stopping tablet replica
I20260130 02:59:10.407388 22649 raft_consensus.cc:2243] T 2e4e3542c9ad480bbc5c2960aa8a549d P 789dbb4a27204ffbb68d7ae40f8f5e4a [term 0 FOLLOWER]: Raft consensus shutting down.
I20260130 02:59:10.407660 22650 ts_tablet_manager.cc:1929] T 2e4e3542c9ad480bbc5c2960aa8a549d P dd32a35972b347b89ceaaad2455aef5a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260130 02:59:10.407945 22649 raft_consensus.cc:2272] T 2e4e3542c9ad480bbc5c2960aa8a549d P 789dbb4a27204ffbb68d7ae40f8f5e4a [term 0 FOLLOWER]: Raft consensus is shut down!
I20260130 02:59:10.408082 22650 log.cc:1199] T 2e4e3542c9ad480bbc5c2960aa8a549d P dd32a35972b347b89ceaaad2455aef5a: Deleting WAL directory at /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1769741859011789-18833-0/minicluster-data/ts-1/wal/wals/2e4e3542c9ad480bbc5c2960aa8a549d
I20260130 02:59:10.408927 22650 ts_tablet_manager.cc:1950] T 2e4e3542c9ad480bbc5c2960aa8a549d P dd32a35972b347b89ceaaad2455aef5a: Deleting consensus metadata
I20260130 02:59:10.410617 22649 ts_tablet_manager.cc:1916] T 2e4e3542c9ad480bbc5c2960aa8a549d P 789dbb4a27204ffbb68d7ae40f8f5e4a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260130 02:59:10.412381 22643 tablet_bootstrap.cc:492] T ef0a82e66c1c4f5a95f2c7e74a1b85f0 P dd32a35972b347b89ceaaad2455aef5a: No bootstrap required, opened a new log
I20260130 02:59:10.412789 22643 ts_tablet_manager.cc:1403] T ef0a82e66c1c4f5a95f2c7e74a1b85f0 P dd32a35972b347b89ceaaad2455aef5a: Time spent bootstrapping tablet: real 0.082s user 0.006s sys 0.009s
I20260130 02:59:10.415269 22643 raft_consensus.cc:359] T ef0a82e66c1c4f5a95f2c7e74a1b85f0 P dd32a35972b347b89ceaaad2455aef5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111215cc989f4203a6e1517f2291d66f" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 39751 } } peers { permanent_uuid: "dd32a35972b347b89ceaaad2455aef5a" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 39781 } } peers { permanent_uuid: "789dbb4a27204ffbb68d7ae40f8f5e4a" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 41825 } }
I20260130 02:59:10.415987 22643 raft_consensus.cc:385] T ef0a82e66c1c4f5a95f2c7e74a1b85f0 P dd32a35972b347b89ceaaad2455aef5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:59:10.416326 22643 raft_consensus.cc:740] T ef0a82e66c1c4f5a95f2c7e74a1b85f0 P dd32a35972b347b89ceaaad2455aef5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd32a35972b347b89ceaaad2455aef5a, State: Initialized, Role: FOLLOWER
I20260130 02:59:10.419971 22652 raft_consensus.cc:493] T ef0a82e66c1c4f5a95f2c7e74a1b85f0 P 111215cc989f4203a6e1517f2291d66f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:59:10.420485 22652 raft_consensus.cc:515] T ef0a82e66c1c4f5a95f2c7e74a1b85f0 P 111215cc989f4203a6e1517f2291d66f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111215cc989f4203a6e1517f2291d66f" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 39751 } } peers { permanent_uuid: "dd32a35972b347b89ceaaad2455aef5a" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 39781 } } peers { permanent_uuid: "789dbb4a27204ffbb68d7ae40f8f5e4a" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 41825 } }
I20260130 02:59:10.424304 21872 catalog_manager.cc:4985] TS dd32a35972b347b89ceaaad2455aef5a (127.18.100.66:39781): tablet 2e4e3542c9ad480bbc5c2960aa8a549d (table default.table_62e153a4e7844d97863f32a6e87da70b [id=dee95603eab3439b81355a006bc9f376]) successfully deleted
I20260130 02:59:10.424533 22651 ts_tablet_manager.cc:1929] T 2e4e3542c9ad480bbc5c2960aa8a549d P 111215cc989f4203a6e1517f2291d66f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260130 02:59:10.424237 22643 consensus_queue.cc:260] T ef0a82e66c1c4f5a95f2c7e74a1b85f0 P dd32a35972b347b89ceaaad2455aef5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111215cc989f4203a6e1517f2291d66f" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 39751 } } peers { permanent_uuid: "dd32a35972b347b89ceaaad2455aef5a" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 39781 } } peers { permanent_uuid: "789dbb4a27204ffbb68d7ae40f8f5e4a" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 41825 } }
I20260130 02:59:10.424911 22651 log.cc:1199] T 2e4e3542c9ad480bbc5c2960aa8a549d P 111215cc989f4203a6e1517f2291d66f: Deleting WAL directory at /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1769741859011789-18833-0/minicluster-data/ts-0/wal/wals/2e4e3542c9ad480bbc5c2960aa8a549d
I20260130 02:59:10.435324 22643 ts_tablet_manager.cc:1434] T ef0a82e66c1c4f5a95f2c7e74a1b85f0 P dd32a35972b347b89ceaaad2455aef5a: Time spent starting tablet: real 0.022s user 0.005s sys 0.003s
I20260130 02:59:10.442977 21885 master_service.cc:954] ReplaceTablet: received request to replace tablet 2e4e3542c9ad480bbc5c2960aa8a549d from {username='slave'} at 127.0.0.1:34990
I20260130 02:59:10.445262 22535 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ef0a82e66c1c4f5a95f2c7e74a1b85f0" candidate_uuid: "111215cc989f4203a6e1517f2291d66f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "789dbb4a27204ffbb68d7ae40f8f5e4a" is_pre_election: true
I20260130 02:59:10.447474 21885 master_service.cc:954] ReplaceTablet: received request to replace tablet 2e4e3542c9ad480bbc5c2960aa8a549d from {username='slave'} at 127.0.0.1:34990
W20260130 02:59:10.447322 22179 leader_election.cc:343] T ef0a82e66c1c4f5a95f2c7e74a1b85f0 P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 789dbb4a27204ffbb68d7ae40f8f5e4a (127.18.100.67:41825): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:59:10.447947 22652 leader_election.cc:290] T ef0a82e66c1c4f5a95f2c7e74a1b85f0 P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dd32a35972b347b89ceaaad2455aef5a (127.18.100.66:39781), 789dbb4a27204ffbb68d7ae40f8f5e4a (127.18.100.67:41825)
I20260130 02:59:10.457386 22642 tablet_bootstrap.cc:492] T ef0a82e66c1c4f5a95f2c7e74a1b85f0 P 789dbb4a27204ffbb68d7ae40f8f5e4a: No bootstrap required, opened a new log
I20260130 02:59:10.457943 22642 ts_tablet_manager.cc:1403] T ef0a82e66c1c4f5a95f2c7e74a1b85f0 P 789dbb4a27204ffbb68d7ae40f8f5e4a: Time spent bootstrapping tablet: real 0.066s user 0.014s sys 0.000s
I20260130 02:59:10.459290 22651 ts_tablet_manager.cc:1950] T 2e4e3542c9ad480bbc5c2960aa8a549d P 111215cc989f4203a6e1517f2291d66f: Deleting consensus metadata
I20260130 02:59:10.460597 22642 raft_consensus.cc:359] T ef0a82e66c1c4f5a95f2c7e74a1b85f0 P 789dbb4a27204ffbb68d7ae40f8f5e4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111215cc989f4203a6e1517f2291d66f" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 39751 } } peers { permanent_uuid: "dd32a35972b347b89ceaaad2455aef5a" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 39781 } } peers { permanent_uuid: "789dbb4a27204ffbb68d7ae40f8f5e4a" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 41825 } }
I20260130 02:59:10.461562 22642 raft_consensus.cc:385] T ef0a82e66c1c4f5a95f2c7e74a1b85f0 P 789dbb4a27204ffbb68d7ae40f8f5e4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:59:10.462378 22642 raft_consensus.cc:740] T ef0a82e66c1c4f5a95f2c7e74a1b85f0 P 789dbb4a27204ffbb68d7ae40f8f5e4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 789dbb4a27204ffbb68d7ae40f8f5e4a, State: Initialized, Role: FOLLOWER
I20260130 02:59:10.465778 22642 consensus_queue.cc:260] T ef0a82e66c1c4f5a95f2c7e74a1b85f0 P 789dbb4a27204ffbb68d7ae40f8f5e4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111215cc989f4203a6e1517f2291d66f" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 39751 } } peers { permanent_uuid: "dd32a35972b347b89ceaaad2455aef5a" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 39781 } } peers { permanent_uuid: "789dbb4a27204ffbb68d7ae40f8f5e4a" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 41825 } }
I20260130 02:59:10.469856 22642 ts_tablet_manager.cc:1434] T ef0a82e66c1c4f5a95f2c7e74a1b85f0 P 789dbb4a27204ffbb68d7ae40f8f5e4a: Time spent starting tablet: real 0.012s user 0.005s sys 0.000s
I20260130 02:59:10.467097 22649 ts_tablet_manager.cc:1929] T 2e4e3542c9ad480bbc5c2960aa8a549d P 789dbb4a27204ffbb68d7ae40f8f5e4a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260130 02:59:10.482026 22649 log.cc:1199] T 2e4e3542c9ad480bbc5c2960aa8a549d P 789dbb4a27204ffbb68d7ae40f8f5e4a: Deleting WAL directory at /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1769741859011789-18833-0/minicluster-data/ts-2/wal/wals/2e4e3542c9ad480bbc5c2960aa8a549d
I20260130 02:59:10.482352 22391 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ef0a82e66c1c4f5a95f2c7e74a1b85f0" candidate_uuid: "111215cc989f4203a6e1517f2291d66f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd32a35972b347b89ceaaad2455aef5a" is_pre_election: true
I20260130 02:59:10.483191 22649 ts_tablet_manager.cc:1950] T 2e4e3542c9ad480bbc5c2960aa8a549d P 789dbb4a27204ffbb68d7ae40f8f5e4a: Deleting consensus metadata
I20260130 02:59:10.483179 22391 raft_consensus.cc:2468] T ef0a82e66c1c4f5a95f2c7e74a1b85f0 P dd32a35972b347b89ceaaad2455aef5a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 111215cc989f4203a6e1517f2291d66f in term 0.
I20260130 02:59:10.484690 22180 leader_election.cc:304] T ef0a82e66c1c4f5a95f2c7e74a1b85f0 P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 111215cc989f4203a6e1517f2291d66f, dd32a35972b347b89ceaaad2455aef5a; no voters: 789dbb4a27204ffbb68d7ae40f8f5e4a
I20260130 02:59:10.485579 22652 raft_consensus.cc:2804] T ef0a82e66c1c4f5a95f2c7e74a1b85f0 P 111215cc989f4203a6e1517f2291d66f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260130 02:59:10.485894 22652 raft_consensus.cc:493] T ef0a82e66c1c4f5a95f2c7e74a1b85f0 P 111215cc989f4203a6e1517f2291d66f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260130 02:59:10.486183 22652 raft_consensus.cc:3060] T ef0a82e66c1c4f5a95f2c7e74a1b85f0 P 111215cc989f4203a6e1517f2291d66f [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:59:10.492990 22652 raft_consensus.cc:515] T ef0a82e66c1c4f5a95f2c7e74a1b85f0 P 111215cc989f4203a6e1517f2291d66f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111215cc989f4203a6e1517f2291d66f" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 39751 } } peers { permanent_uuid: "dd32a35972b347b89ceaaad2455aef5a" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 39781 } } peers { permanent_uuid: "789dbb4a27204ffbb68d7ae40f8f5e4a" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 41825 } }
I20260130 02:59:10.495971 22391 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ef0a82e66c1c4f5a95f2c7e74a1b85f0" candidate_uuid: "111215cc989f4203a6e1517f2291d66f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd32a35972b347b89ceaaad2455aef5a"
I20260130 02:59:10.496529 22391 raft_consensus.cc:3060] T ef0a82e66c1c4f5a95f2c7e74a1b85f0 P dd32a35972b347b89ceaaad2455aef5a [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:59:10.500689 21871 catalog_manager.cc:4985] TS 789dbb4a27204ffbb68d7ae40f8f5e4a (127.18.100.67:41825): tablet 2e4e3542c9ad480bbc5c2960aa8a549d (table default.table_62e153a4e7844d97863f32a6e87da70b [id=dee95603eab3439b81355a006bc9f376]) successfully deleted
I20260130 02:59:10.503407 22652 leader_election.cc:290] T ef0a82e66c1c4f5a95f2c7e74a1b85f0 P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 election: Requested vote from peers dd32a35972b347b89ceaaad2455aef5a (127.18.100.66:39781), 789dbb4a27204ffbb68d7ae40f8f5e4a (127.18.100.67:41825)
I20260130 02:59:10.503994 22535 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ef0a82e66c1c4f5a95f2c7e74a1b85f0" candidate_uuid: "111215cc989f4203a6e1517f2291d66f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "789dbb4a27204ffbb68d7ae40f8f5e4a"
I20260130 02:59:10.504539 22535 raft_consensus.cc:3060] T ef0a82e66c1c4f5a95f2c7e74a1b85f0 P 789dbb4a27204ffbb68d7ae40f8f5e4a [term 0 FOLLOWER]: Advancing to term 1
W20260130 02:59:10.508832 22444 tablet.cc:2378] T ef0a82e66c1c4f5a95f2c7e74a1b85f0 P dd32a35972b347b89ceaaad2455aef5a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260130 02:59:10.518694 21871 catalog_manager.cc:4985] TS 111215cc989f4203a6e1517f2291d66f (127.18.100.65:39751): tablet 2e4e3542c9ad480bbc5c2960aa8a549d (table default.table_62e153a4e7844d97863f32a6e87da70b [id=dee95603eab3439b81355a006bc9f376]) successfully deleted
I20260130 02:59:10.523082 22535 raft_consensus.cc:2468] T ef0a82e66c1c4f5a95f2c7e74a1b85f0 P 789dbb4a27204ffbb68d7ae40f8f5e4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 111215cc989f4203a6e1517f2291d66f in term 1.
I20260130 02:59:10.524339 22179 leader_election.cc:304] T ef0a82e66c1c4f5a95f2c7e74a1b85f0 P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 111215cc989f4203a6e1517f2291d66f, 789dbb4a27204ffbb68d7ae40f8f5e4a; no voters:
I20260130 02:59:10.525179 22652 raft_consensus.cc:2804] T ef0a82e66c1c4f5a95f2c7e74a1b85f0 P 111215cc989f4203a6e1517f2291d66f [term 1 FOLLOWER]: Leader election won for term 1
I20260130 02:59:10.522393 22391 raft_consensus.cc:2468] T ef0a82e66c1c4f5a95f2c7e74a1b85f0 P dd32a35972b347b89ceaaad2455aef5a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 111215cc989f4203a6e1517f2291d66f in term 1.
I20260130 02:59:10.528178 22652 raft_consensus.cc:697] T ef0a82e66c1c4f5a95f2c7e74a1b85f0 P 111215cc989f4203a6e1517f2291d66f [term 1 LEADER]: Becoming Leader. State: Replica: 111215cc989f4203a6e1517f2291d66f, State: Running, Role: LEADER
I20260130 02:59:10.529966 22652 consensus_queue.cc:237] T ef0a82e66c1c4f5a95f2c7e74a1b85f0 P 111215cc989f4203a6e1517f2291d66f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111215cc989f4203a6e1517f2291d66f" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 39751 } } peers { permanent_uuid: "dd32a35972b347b89ceaaad2455aef5a" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 39781 } } peers { permanent_uuid: "789dbb4a27204ffbb68d7ae40f8f5e4a" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 41825 } }
02:59:10.534 [HMS - ERROR - pool-6-thread-3] (RetryingHMSHandler.java:201) AlreadyExistsException(message:Table hive.default.table_de96622ce9794752adfc14fb9ceb4559 already exists)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_core(HiveMetaStore.java:1839)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_core(HiveMetaStore.java:1786)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_with_environment_context(HiveMetaStore.java:2035)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
at com.sun.proxy.$Proxy25.create_table_with_environment_context(Unknown Source)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$create_table_with_environment_context.getResult(ThriftHiveMetastore.java:14800)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$create_table_with_environment_context.getResult(ThriftHiveMetastore.java:14784)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
at org.apache.hadoop.hive.metastore.TUGIBasedProcessor.process(TUGIBasedProcessor.java:104)
at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
W20260130 02:59:10.538458 21884 catalog_manager.cc:2170] Already present: failed to create HMS catalog entry for table [id=34ba4c5fef4f4f6197f0ce6bb7d85873]: failed to create Hive MetaStore table: TException - service has thrown: AlreadyExistsException(message=Table hive.default.table_de96622ce9794752adfc14fb9ceb4559 already exists)
I20260130 02:59:10.539387 21884 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:34990 (request call id 11) took 1045 ms. Trace:
I20260130 02:59:10.539753 21884 rpcz_store.cc:276] 0130 02:59:09.493915 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:59:09.495655 (+ 1740us) service_pool.cc:225] Handling call
0130 02:59:09.538749 (+ 43094us) catalog_manager.cc:2101] Acquired catalog manager lock
0130 02:59:09.540751 (+ 2002us) catalog_manager.cc:2147] Created new table and tablet info
0130 02:59:10.539319 (+998568us) inbound_call.cc:173] Queueing success response
Metrics: {"Hive Metastore.queue_time_us":396806,"Hive Metastore.run_cpu_time_us":1741,"Hive Metastore.run_wall_time_us":599480}
I20260130 02:59:10.544770 21884 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34990:
name: "default.Table_66d14bc2eefe4c4cabbfe8ea4ca2a287"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260130 02:59:10.557642 21885 catalog_manager.cc:5654] T ef0a82e66c1c4f5a95f2c7e74a1b85f0 P 111215cc989f4203a6e1517f2291d66f reported cstate change: term changed from 0 to 1, leader changed from <none> to 111215cc989f4203a6e1517f2291d66f (127.18.100.65). New cstate: current_term: 1 leader_uuid: "111215cc989f4203a6e1517f2291d66f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111215cc989f4203a6e1517f2291d66f" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 39751 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dd32a35972b347b89ceaaad2455aef5a" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 39781 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "789dbb4a27204ffbb68d7ae40f8f5e4a" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 41825 } health_report { overall_health: UNKNOWN } } }
I20260130 02:59:10.604297 21886 catalog_manager.cc:2174] created HMS catalog entry for table [id=318c872fbfef4f93b4c280fd078cabd9]
I20260130 02:59:10.618866 21886 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:34990 (request call id 9) took 1127 ms. Trace:
I20260130 02:59:10.643903 22227 tablet_service.cc:1505] Processing CreateTablet for tablet 8db03bbf33c641b68deb9de324745ac0 (DEFAULT_TABLE table=default.table_e3b3683b36e144c2b33dd88b292bcb23 [id=318c872fbfef4f93b4c280fd078cabd9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:59:10.645139 22227 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8db03bbf33c641b68deb9de324745ac0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:59:10.619206 21886 rpcz_store.cc:276] 0130 02:59:09.491071 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:59:09.491258 (+ 187us) service_pool.cc:225] Handling call
0130 02:59:09.543099 (+ 51841us) catalog_manager.cc:2101] Acquired catalog manager lock
0130 02:59:09.545258 (+ 2159us) catalog_manager.cc:2147] Created new table and tablet info
0130 02:59:10.604241 (+1058983us) catalog_manager.cc:2173] Created new table in HMS catalog
0130 02:59:10.618142 (+ 13901us) catalog_manager.cc:2207] Wrote table and tablets to system table
0130 02:59:10.618467 (+ 325us) catalog_manager.cc:2241] Inserted table and tablets into CatalogManager maps
0130 02:59:10.618800 (+ 333us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0130 02:59:10.606157 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0130 02:59:10.606725 (+ 568us) write_op.cc:432] Acquiring schema lock in shared mode
0130 02:59:10.606748 (+ 23us) write_op.cc:435] Acquired schema lock
0130 02:59:10.606762 (+ 14us) tablet.cc:660] Decoding operations
0130 02:59:10.607093 (+ 331us) write_op.cc:620] Acquiring the partition lock for write op
0130 02:59:10.607137 (+ 44us) write_op.cc:641] Partition lock acquired for write op
0130 02:59:10.607167 (+ 30us) tablet.cc:683] Acquiring locks for 2 operations
0130 02:59:10.607388 (+ 221us) tablet.cc:699] Row locks acquired
0130 02:59:10.607404 (+ 16us) write_op.cc:260] PREPARE: finished
0130 02:59:10.607530 (+ 126us) write_op.cc:270] Start()
0130 02:59:10.607618 (+ 88us) write_op.cc:276] Timestamp: P: 1769741950607462 usec, L: 0
0130 02:59:10.607637 (+ 19us) op_driver.cc:348] REPLICATION: starting
0130 02:59:10.608187 (+ 550us) log.cc:844] Serialized 432 byte log entry
0130 02:59:10.616332 (+ 8145us) op_driver.cc:464] REPLICATION: finished
0130 02:59:10.616684 (+ 352us) write_op.cc:301] APPLY: starting
0130 02:59:10.616737 (+ 53us) tablet.cc:1365] starting BulkCheckPresence
0130 02:59:10.616833 (+ 96us) tablet.cc:1368] finished BulkCheckPresence
0130 02:59:10.616929 (+ 96us) tablet.cc:1370] starting ApplyRowOperation cycle
0130 02:59:10.617289 (+ 360us) tablet.cc:1381] finished ApplyRowOperation cycle
0130 02:59:10.617417 (+ 128us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0130 02:59:10.617462 (+ 45us) write_op.cc:312] APPLY: finished
0130 02:59:10.617630 (+ 168us) log.cc:844] Serialized 28 byte log entry
0130 02:59:10.617778 (+ 148us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:59:10.617850 (+ 72us) write_op.cc:454] Released schema lock
0130 02:59:10.617883 (+ 33us) write_op.cc:341] FINISH: Updating metrics
0130 02:59:10.618097 (+ 214us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:59:10.618120 (+ 23us) write_op.cc:454] Released schema lock
Metrics: {"Hive Metastore.queue_time_us":991967,"Hive Metastore.run_cpu_time_us":654,"Hive Metastore.run_wall_time_us":66393,"mutex_wait_us":540,"child_traces":[["op",{"apply.queue_time_us":208,"apply.run_cpu_time_us":1387,"apply.run_wall_time_us":1386,"num_ops":2,"prepare.queue_time_us":172,"prepare.run_cpu_time_us":2337,"prepare.run_wall_time_us":2385,"raft.queue_time_us":485,"raft.run_cpu_time_us":633,"raft.run_wall_time_us":633,"replication_time_us":8582}]]}
I20260130 02:59:10.647398 22369 tablet_service.cc:1505] Processing CreateTablet for tablet 8db03bbf33c641b68deb9de324745ac0 (DEFAULT_TABLE table=default.table_e3b3683b36e144c2b33dd88b292bcb23 [id=318c872fbfef4f93b4c280fd078cabd9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:59:10.648466 22369 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8db03bbf33c641b68deb9de324745ac0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:59:10.650785 22515 tablet_service.cc:1505] Processing CreateTablet for tablet 8db03bbf33c641b68deb9de324745ac0 (DEFAULT_TABLE table=default.table_e3b3683b36e144c2b33dd88b292bcb23 [id=318c872fbfef4f93b4c280fd078cabd9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:59:10.651855 22515 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8db03bbf33c641b68deb9de324745ac0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:59:10.655718 21887 catalog_manager.cc:2174] created HMS catalog entry for table [id=db113a9d4bf545faa72c7c94427d6dba]
I20260130 02:59:10.658232 21885 master_service.cc:954] ReplaceTablet: received request to replace tablet ef0a82e66c1c4f5a95f2c7e74a1b85f0 from {username='slave'} at 127.0.0.1:34990
I20260130 02:59:10.661545 21881 master_service.cc:954] ReplaceTablet: received request to replace tablet ef0a82e66c1c4f5a95f2c7e74a1b85f0 from {username='slave'} at 127.0.0.1:34990
I20260130 02:59:10.667984 21886 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34990:
table { table_name: "default.Table_e3b3683b36e144c2b33dd88b292bcb23" } new_table_name: "default.Table_1b5ff18b6b2146a98161fe3210afe549" modify_external_catalogs: true
I20260130 02:59:10.672925 22641 tablet_bootstrap.cc:492] T 8db03bbf33c641b68deb9de324745ac0 P 111215cc989f4203a6e1517f2291d66f: Bootstrap starting.
I20260130 02:59:10.674981 22643 tablet_bootstrap.cc:492] T 8db03bbf33c641b68deb9de324745ac0 P dd32a35972b347b89ceaaad2455aef5a: Bootstrap starting.
I20260130 02:59:10.679602 22641 tablet_bootstrap.cc:654] T 8db03bbf33c641b68deb9de324745ac0 P 111215cc989f4203a6e1517f2291d66f: Neither blocks nor log segments found. Creating new log.
I20260130 02:59:10.679638 21887 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:34990 (request call id 8) took 1193 ms. Trace:
I20260130 02:59:10.679962 21887 rpcz_store.cc:276] 0130 02:59:09.486029 (+ 0us) service_pool.cc:168] Inserting onto call queue
0130 02:59:09.486212 (+ 183us) service_pool.cc:225] Handling call
0130 02:59:09.547427 (+ 61215us) catalog_manager.cc:2101] Acquired catalog manager lock
0130 02:59:09.549255 (+ 1828us) catalog_manager.cc:2147] Created new table and tablet info
0130 02:59:10.655670 (+1106415us) catalog_manager.cc:2173] Created new table in HMS catalog
0130 02:59:10.678880 (+ 23210us) catalog_manager.cc:2207] Wrote table and tablets to system table
0130 02:59:10.679236 (+ 356us) catalog_manager.cc:2241] Inserted table and tablets into CatalogManager maps
0130 02:59:10.679555 (+ 319us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0130 02:59:10.657460 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0130 02:59:10.657911 (+ 451us) write_op.cc:432] Acquiring schema lock in shared mode
0130 02:59:10.657937 (+ 26us) write_op.cc:435] Acquired schema lock
0130 02:59:10.657952 (+ 15us) tablet.cc:660] Decoding operations
0130 02:59:10.658388 (+ 436us) write_op.cc:620] Acquiring the partition lock for write op
0130 02:59:10.658435 (+ 47us) write_op.cc:641] Partition lock acquired for write op
0130 02:59:10.658465 (+ 30us) tablet.cc:683] Acquiring locks for 2 operations
0130 02:59:10.658688 (+ 223us) tablet.cc:699] Row locks acquired
0130 02:59:10.658705 (+ 17us) write_op.cc:260] PREPARE: finished
0130 02:59:10.658833 (+ 128us) write_op.cc:270] Start()
0130 02:59:10.658959 (+ 126us) write_op.cc:276] Timestamp: P: 1769741950658760 usec, L: 0
0130 02:59:10.658979 (+ 20us) op_driver.cc:348] REPLICATION: starting
0130 02:59:10.659493 (+ 514us) log.cc:844] Serialized 432 byte log entry
0130 02:59:10.676812 (+ 17319us) op_driver.cc:464] REPLICATION: finished
0130 02:59:10.677360 (+ 548us) write_op.cc:301] APPLY: starting
0130 02:59:10.677416 (+ 56us) tablet.cc:1365] starting BulkCheckPresence
0130 02:59:10.677522 (+ 106us) tablet.cc:1368] finished BulkCheckPresence
0130 02:59:10.677617 (+ 95us) tablet.cc:1370] starting ApplyRowOperation cycle
0130 02:59:10.678006 (+ 389us) tablet.cc:1381] finished ApplyRowOperation cycle
0130 02:59:10.678144 (+ 138us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0130 02:59:10.678187 (+ 43us) write_op.cc:312] APPLY: finished
0130 02:59:10.678395 (+ 208us) log.cc:844] Serialized 28 byte log entry
0130 02:59:10.678547 (+ 152us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:59:10.678634 (+ 87us) write_op.cc:454] Released schema lock
0130 02:59:10.678668 (+ 34us) write_op.cc:341] FINISH: Updating metrics
0130 02:59:10.678893 (+ 225us) write_op.cc:489] Releasing partition, row and schema locks
0130 02:59:10.678916 (+ 23us) write_op.cc:454] Released schema lock
Metrics: {"Hive Metastore.queue_time_us":1054495,"Hive Metastore.run_cpu_time_us":623,"Hive Metastore.run_wall_time_us":51326,"child_traces":[["op",{"apply.queue_time_us":389,"apply.run_cpu_time_us":1500,"apply.run_wall_time_us":1499,"num_ops":2,"prepare.queue_time_us":136,"prepare.run_cpu_time_us":2352,"prepare.run_wall_time_us":2352,"raft.queue_time_us":2661,"raft.run_cpu_time_us":645,"raft.run_wall_time_us":646,"replication_time_us":17723}]]}
I20260130 02:59:10.687258 22643 tablet_bootstrap.cc:654] T 8db03bbf33c641b68deb9de324745ac0 P dd32a35972b347b89ceaaad2455aef5a: Neither blocks nor log segments found. Creating new log.
I20260130 02:59:10.690711 21887 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34990:
table { table_name: "default.Table_d6f2cd9cd3b54168b14f1cacabbc74f0" } modify_external_catalogs: true
I20260130 02:59:10.691493 21887 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34990:
table { table_name: "default.Table_d6f2cd9cd3b54168b14f1cacabbc74f0" } modify_external_catalogs: true
F20260130 02:59:10.694654 22151 hms_notification_log_listener.cc:287] non-monotonous event IDs from HMS: current 1, previous 1; dumping first 2 out of 4 received events: NotificationEvent(eventId=1, eventTime=1769741949, eventType=CREATE_TABLE, dbName=default, tableName=table_de96622ce9794752adfc14fb9ceb4559, message={"server":"","servicePrincipal":"","db":"default","table":"table_de96622ce9794752adfc14fb9ceb4559","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_de96622ce9794752adfc14fb9ceb4559\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1769741949},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1769741859011789-18833-0/minicluster-data/warehouse/table_de96622ce9794752adfc14fb9ceb4559\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_de96622ce9794752adfc14fb9ceb4559\",\"kudu.table_id\":\"e5276a565ee84745a424387f988a65e9\",\"transient_lastDdlTime\":\"1769741949\",\"comment\":\"\",\"kudu.master_addresses\":\"127.18.100.126:42077,127.18.100.125:39157,127.18.100.124:34987\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"40eb42871f464e95bd641ceacb567979\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1769741949,"files":[]}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=1, eventTime=1769741949, eventType=CREATE_TABLE, dbName=default, tableName=table_62e153a4e7844d97863f32a6e87da70b, message={"server":"","servicePrincipal":"","db":"default","table":"table_62e153a4e7844d97863f32a6e87da70b","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_62e153a4e7844d97863f32a6e87da70b\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1769741949},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1769741859011789-18833-0/minicluster-data/warehouse/table_62e153a4e7844d97863f32a6e87da70b\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_62e153a4e7844d97863f32a6e87da70b\",\"kudu.table_id\":\"dee95603eab3439b81355a006bc9f376\",\"transient_lastDdlTime\":\"1769741949\",\"comment\":\"\",\"kudu.master_addresses\":\"127.18.100.126:42077,127.18.100.125:39157,127.18.100.124:34987\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"40eb42871f464e95bd641ceacb567979\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1769741949,"files":[]}, messageFormat=json-0.2, catName=hive);
*** Check failure stack trace: ***
I20260130 02:59:10.696911 21885 catalog_manager.cc:5941] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324: Sending DeleteTablet for 3 replicas of tablet ef0a82e66c1c4f5a95f2c7e74a1b85f0
I20260130 02:59:10.698752 21885 catalog_manager.cc:6724] ReplaceTablet: tablet ef0a82e66c1c4f5a95f2c7e74a1b85f0 deleted and replaced by tablet f41b5d6226db4f4287f8a4cdea9a0bdc
I20260130 02:59:10.699435 22369 tablet_service.cc:1552] Processing DeleteTablet for tablet ef0a82e66c1c4f5a95f2c7e74a1b85f0 with delete_type TABLET_DATA_DELETED (replaced by tablet f41b5d6226db4f4287f8a4cdea9a0bdc) from {username='slave'} at 127.0.0.1:44082
I20260130 02:59:10.700584 22650 tablet_replica.cc:333] T ef0a82e66c1c4f5a95f2c7e74a1b85f0 P dd32a35972b347b89ceaaad2455aef5a: stopping tablet replica
I20260130 02:59:10.701303 22650 raft_consensus.cc:2243] T ef0a82e66c1c4f5a95f2c7e74a1b85f0 P dd32a35972b347b89ceaaad2455aef5a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260130 02:59:10.701808 22650 raft_consensus.cc:2272] T ef0a82e66c1c4f5a95f2c7e74a1b85f0 P dd32a35972b347b89ceaaad2455aef5a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260130 02:59:10.702878 22227 tablet_service.cc:1552] Processing DeleteTablet for tablet ef0a82e66c1c4f5a95f2c7e74a1b85f0 with delete_type TABLET_DATA_DELETED (replaced by tablet f41b5d6226db4f4287f8a4cdea9a0bdc) from {username='slave'} at 127.0.0.1:53816
I20260130 02:59:10.703184 22514 tablet_service.cc:1552] Processing DeleteTablet for tablet ef0a82e66c1c4f5a95f2c7e74a1b85f0 with delete_type TABLET_DATA_DELETED (replaced by tablet f41b5d6226db4f4287f8a4cdea9a0bdc) from {username='slave'} at 127.0.0.1:53088
I20260130 02:59:10.703612 22651 tablet_replica.cc:333] T ef0a82e66c1c4f5a95f2c7e74a1b85f0 P 111215cc989f4203a6e1517f2291d66f: stopping tablet replica
I20260130 02:59:10.704284 22651 raft_consensus.cc:2243] T ef0a82e66c1c4f5a95f2c7e74a1b85f0 P 111215cc989f4203a6e1517f2291d66f [term 1 LEADER]: Raft consensus shutting down.
I20260130 02:59:10.705055 22651 pending_rounds.cc:70] T ef0a82e66c1c4f5a95f2c7e74a1b85f0 P 111215cc989f4203a6e1517f2291d66f: Trying to abort 1 pending ops.
I20260130 02:59:10.705031 22649 tablet_replica.cc:333] T ef0a82e66c1c4f5a95f2c7e74a1b85f0 P 789dbb4a27204ffbb68d7ae40f8f5e4a: stopping tablet replica
I20260130 02:59:10.705328 22651 pending_rounds.cc:77] T ef0a82e66c1c4f5a95f2c7e74a1b85f0 P 111215cc989f4203a6e1517f2291d66f: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7248863029397585920 op_type: NO_OP noop_request { }
I20260130 02:59:10.705731 22649 raft_consensus.cc:2243] T ef0a82e66c1c4f5a95f2c7e74a1b85f0 P 789dbb4a27204ffbb68d7ae40f8f5e4a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260130 02:59:10.705758 22651 raft_consensus.cc:2889] T ef0a82e66c1c4f5a95f2c7e74a1b85f0 P 111215cc989f4203a6e1517f2291d66f [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20260130 02:59:10.706130 22651 raft_consensus.cc:2272] T ef0a82e66c1c4f5a95f2c7e74a1b85f0 P 111215cc989f4203a6e1517f2291d66f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260130 02:59:10.706228 22649 raft_consensus.cc:2272] T ef0a82e66c1c4f5a95f2c7e74a1b85f0 P 789dbb4a27204ffbb68d7ae40f8f5e4a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260130 02:59:10.708338 22649 ts_tablet_manager.cc:1916] T ef0a82e66c1c4f5a95f2c7e74a1b85f0 P 789dbb4a27204ffbb68d7ae40f8f5e4a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260130 02:59:10.708882 22651 ts_tablet_manager.cc:1916] T ef0a82e66c1c4f5a95f2c7e74a1b85f0 P 111215cc989f4203a6e1517f2291d66f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260130 02:59:10.709657 22650 ts_tablet_manager.cc:1916] T ef0a82e66c1c4f5a95f2c7e74a1b85f0 P dd32a35972b347b89ceaaad2455aef5a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260130 02:59:10.720428 22641 tablet_bootstrap.cc:492] T 8db03bbf33c641b68deb9de324745ac0 P 111215cc989f4203a6e1517f2291d66f: No bootstrap required, opened a new log
I20260130 02:59:10.720956 22641 ts_tablet_manager.cc:1403] T 8db03bbf33c641b68deb9de324745ac0 P 111215cc989f4203a6e1517f2291d66f: Time spent bootstrapping tablet: real 0.048s user 0.008s sys 0.004s
I20260130 02:59:10.723649 22641 raft_consensus.cc:359] T 8db03bbf33c641b68deb9de324745ac0 P 111215cc989f4203a6e1517f2291d66f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111215cc989f4203a6e1517f2291d66f" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 39751 } } peers { permanent_uuid: "dd32a35972b347b89ceaaad2455aef5a" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 39781 } } peers { permanent_uuid: "789dbb4a27204ffbb68d7ae40f8f5e4a" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 41825 } }
I20260130 02:59:10.726769 22641 raft_consensus.cc:385] T 8db03bbf33c641b68deb9de324745ac0 P 111215cc989f4203a6e1517f2291d66f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:59:10.727159 22641 raft_consensus.cc:740] T 8db03bbf33c641b68deb9de324745ac0 P 111215cc989f4203a6e1517f2291d66f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 111215cc989f4203a6e1517f2291d66f, State: Initialized, Role: FOLLOWER
I20260130 02:59:10.724261 22651 ts_tablet_manager.cc:1929] T ef0a82e66c1c4f5a95f2c7e74a1b85f0 P 111215cc989f4203a6e1517f2291d66f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260130 02:59:10.728446 22649 ts_tablet_manager.cc:1929] T ef0a82e66c1c4f5a95f2c7e74a1b85f0 P 789dbb4a27204ffbb68d7ae40f8f5e4a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260130 02:59:10.728590 22651 log.cc:1199] T ef0a82e66c1c4f5a95f2c7e74a1b85f0 P 111215cc989f4203a6e1517f2291d66f: Deleting WAL directory at /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1769741859011789-18833-0/minicluster-data/ts-0/wal/wals/ef0a82e66c1c4f5a95f2c7e74a1b85f0
I20260130 02:59:10.728941 22649 log.cc:1199] T ef0a82e66c1c4f5a95f2c7e74a1b85f0 P 789dbb4a27204ffbb68d7ae40f8f5e4a: Deleting WAL directory at /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1769741859011789-18833-0/minicluster-data/ts-2/wal/wals/ef0a82e66c1c4f5a95f2c7e74a1b85f0
I20260130 02:59:10.729295 22651 ts_tablet_manager.cc:1950] T ef0a82e66c1c4f5a95f2c7e74a1b85f0 P 111215cc989f4203a6e1517f2291d66f: Deleting consensus metadata
I20260130 02:59:10.729669 22649 ts_tablet_manager.cc:1950] T ef0a82e66c1c4f5a95f2c7e74a1b85f0 P 789dbb4a27204ffbb68d7ae40f8f5e4a: Deleting consensus metadata
I20260130 02:59:10.732102 21871 catalog_manager.cc:4985] TS 789dbb4a27204ffbb68d7ae40f8f5e4a (127.18.100.67:41825): tablet ef0a82e66c1c4f5a95f2c7e74a1b85f0 (table default.table_62e153a4e7844d97863f32a6e87da70b [id=dee95603eab3439b81355a006bc9f376]) successfully deleted
I20260130 02:59:10.727807 22641 consensus_queue.cc:260] T 8db03bbf33c641b68deb9de324745ac0 P 111215cc989f4203a6e1517f2291d66f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111215cc989f4203a6e1517f2291d66f" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 39751 } } peers { permanent_uuid: "dd32a35972b347b89ceaaad2455aef5a" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 39781 } } peers { permanent_uuid: "789dbb4a27204ffbb68d7ae40f8f5e4a" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 41825 } }
I20260130 02:59:10.734849 21871 catalog_manager.cc:4985] TS 111215cc989f4203a6e1517f2291d66f (127.18.100.65:39751): tablet ef0a82e66c1c4f5a95f2c7e74a1b85f0 (table default.table_62e153a4e7844d97863f32a6e87da70b [id=dee95603eab3439b81355a006bc9f376]) successfully deleted
I20260130 02:59:10.736037 22641 ts_tablet_manager.cc:1434] T 8db03bbf33c641b68deb9de324745ac0 P 111215cc989f4203a6e1517f2291d66f: Time spent starting tablet: real 0.015s user 0.005s sys 0.000s
@ 0x7fb49a2f4950 google::LogMessage::Flush() at ??:0
I20260130 02:59:10.749624 22643 tablet_bootstrap.cc:492] T 8db03bbf33c641b68deb9de324745ac0 P dd32a35972b347b89ceaaad2455aef5a: No bootstrap required, opened a new log
I20260130 02:59:10.750052 22643 ts_tablet_manager.cc:1403] T 8db03bbf33c641b68deb9de324745ac0 P dd32a35972b347b89ceaaad2455aef5a: Time spent bootstrapping tablet: real 0.075s user 0.042s sys 0.018s
I20260130 02:59:10.752364 22650 ts_tablet_manager.cc:1929] T ef0a82e66c1c4f5a95f2c7e74a1b85f0 P dd32a35972b347b89ceaaad2455aef5a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260130 02:59:10.752750 22650 log.cc:1199] T ef0a82e66c1c4f5a95f2c7e74a1b85f0 P dd32a35972b347b89ceaaad2455aef5a: Deleting WAL directory at /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1769741859011789-18833-0/minicluster-data/ts-1/wal/wals/ef0a82e66c1c4f5a95f2c7e74a1b85f0
I20260130 02:59:10.752627 22643 raft_consensus.cc:359] T 8db03bbf33c641b68deb9de324745ac0 P dd32a35972b347b89ceaaad2455aef5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111215cc989f4203a6e1517f2291d66f" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 39751 } } peers { permanent_uuid: "dd32a35972b347b89ceaaad2455aef5a" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 39781 } } peers { permanent_uuid: "789dbb4a27204ffbb68d7ae40f8f5e4a" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 41825 } }
I20260130 02:59:10.753269 22643 raft_consensus.cc:385] T 8db03bbf33c641b68deb9de324745ac0 P dd32a35972b347b89ceaaad2455aef5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:59:10.753405 22650 ts_tablet_manager.cc:1950] T ef0a82e66c1c4f5a95f2c7e74a1b85f0 P dd32a35972b347b89ceaaad2455aef5a: Deleting consensus metadata
I20260130 02:59:10.753561 22643 raft_consensus.cc:740] T 8db03bbf33c641b68deb9de324745ac0 P dd32a35972b347b89ceaaad2455aef5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd32a35972b347b89ceaaad2455aef5a, State: Initialized, Role: FOLLOWER
I20260130 02:59:10.755000 22642 tablet_bootstrap.cc:492] T 8db03bbf33c641b68deb9de324745ac0 P 789dbb4a27204ffbb68d7ae40f8f5e4a: Bootstrap starting.
I20260130 02:59:10.754649 22643 consensus_queue.cc:260] T 8db03bbf33c641b68deb9de324745ac0 P dd32a35972b347b89ceaaad2455aef5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111215cc989f4203a6e1517f2291d66f" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 39751 } } peers { permanent_uuid: "dd32a35972b347b89ceaaad2455aef5a" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 39781 } } peers { permanent_uuid: "789dbb4a27204ffbb68d7ae40f8f5e4a" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 41825 } }
I20260130 02:59:10.756748 22643 ts_tablet_manager.cc:1434] T 8db03bbf33c641b68deb9de324745ac0 P dd32a35972b347b89ceaaad2455aef5a: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260130 02:59:10.768747 22642 tablet_bootstrap.cc:654] T 8db03bbf33c641b68deb9de324745ac0 P 789dbb4a27204ffbb68d7ae40f8f5e4a: Neither blocks nor log segments found. Creating new log.
@ 0x7fb49a2f46bf google::LogMessage::~LogMessage() at ??:0
I20260130 02:59:10.774698 21872 catalog_manager.cc:4985] TS dd32a35972b347b89ceaaad2455aef5a (127.18.100.66:39781): tablet ef0a82e66c1c4f5a95f2c7e74a1b85f0 (table default.table_62e153a4e7844d97863f32a6e87da70b [id=dee95603eab3439b81355a006bc9f376]) successfully deleted
I20260130 02:59:10.777995 22642 tablet_bootstrap.cc:492] T 8db03bbf33c641b68deb9de324745ac0 P 789dbb4a27204ffbb68d7ae40f8f5e4a: No bootstrap required, opened a new log
I20260130 02:59:10.778416 22642 ts_tablet_manager.cc:1403] T 8db03bbf33c641b68deb9de324745ac0 P 789dbb4a27204ffbb68d7ae40f8f5e4a: Time spent bootstrapping tablet: real 0.024s user 0.001s sys 0.008s
I20260130 02:59:10.780943 22642 raft_consensus.cc:359] T 8db03bbf33c641b68deb9de324745ac0 P 789dbb4a27204ffbb68d7ae40f8f5e4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111215cc989f4203a6e1517f2291d66f" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 39751 } } peers { permanent_uuid: "dd32a35972b347b89ceaaad2455aef5a" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 39781 } } peers { permanent_uuid: "789dbb4a27204ffbb68d7ae40f8f5e4a" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 41825 } }
I20260130 02:59:10.781926 22642 raft_consensus.cc:385] T 8db03bbf33c641b68deb9de324745ac0 P 789dbb4a27204ffbb68d7ae40f8f5e4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:59:10.782204 22642 raft_consensus.cc:740] T 8db03bbf33c641b68deb9de324745ac0 P 789dbb4a27204ffbb68d7ae40f8f5e4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 789dbb4a27204ffbb68d7ae40f8f5e4a, State: Initialized, Role: FOLLOWER
I20260130 02:59:10.782945 22642 consensus_queue.cc:260] T 8db03bbf33c641b68deb9de324745ac0 P 789dbb4a27204ffbb68d7ae40f8f5e4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111215cc989f4203a6e1517f2291d66f" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 39751 } } peers { permanent_uuid: "dd32a35972b347b89ceaaad2455aef5a" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 39781 } } peers { permanent_uuid: "789dbb4a27204ffbb68d7ae40f8f5e4a" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 41825 } }
@ 0x7fb4a5393fdc kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
I20260130 02:59:10.795850 22642 ts_tablet_manager.cc:1434] T 8db03bbf33c641b68deb9de324745ac0 P 789dbb4a27204ffbb68d7ae40f8f5e4a: Time spent starting tablet: real 0.017s user 0.005s sys 0.000s
I20260130 02:59:10.802690 22369 tablet_service.cc:1505] Processing CreateTablet for tablet 9ee21c80f1ee4d16b777130f87124d56 (DEFAULT_TABLE table=default.table_d6f2cd9cd3b54168b14f1cacabbc74f0 [id=db113a9d4bf545faa72c7c94427d6dba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:59:10.803776 22369 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9ee21c80f1ee4d16b777130f87124d56. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:59:10.805424 22227 tablet_service.cc:1505] Processing CreateTablet for tablet 9ee21c80f1ee4d16b777130f87124d56 (DEFAULT_TABLE table=default.table_d6f2cd9cd3b54168b14f1cacabbc74f0 [id=db113a9d4bf545faa72c7c94427d6dba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:59:10.806450 22227 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9ee21c80f1ee4d16b777130f87124d56. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:59:10.811666 22515 tablet_service.cc:1505] Processing CreateTablet for tablet 9ee21c80f1ee4d16b777130f87124d56 (DEFAULT_TABLE table=default.table_d6f2cd9cd3b54168b14f1cacabbc74f0 [id=db113a9d4bf545faa72c7c94427d6dba]), partition=RANGE (key) PARTITION UNBOUNDED
@ 0x7fb4a539379c kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
I20260130 02:59:10.818147 22515 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9ee21c80f1ee4d16b777130f87124d56. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:59:10.821036 22368 tablet_service.cc:1505] Processing CreateTablet for tablet f41b5d6226db4f4287f8a4cdea9a0bdc (DEFAULT_TABLE table=default.table_62e153a4e7844d97863f32a6e87da70b [id=dee95603eab3439b81355a006bc9f376]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:59:10.822170 22368 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f41b5d6226db4f4287f8a4cdea9a0bdc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:59:10.828253 22226 tablet_service.cc:1505] Processing CreateTablet for tablet f41b5d6226db4f4287f8a4cdea9a0bdc (DEFAULT_TABLE table=default.table_62e153a4e7844d97863f32a6e87da70b [id=dee95603eab3439b81355a006bc9f376]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:59:10.829372 22226 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f41b5d6226db4f4287f8a4cdea9a0bdc. 1 dirs total, 0 dirs full, 0 dirs failed
@ 0x7fb4a5397812 kudu::master::HmsNotificationLogListenerTask::Init()::$_0::operator()() at ??:0
I20260130 02:59:10.843606 22643 tablet_bootstrap.cc:492] T 9ee21c80f1ee4d16b777130f87124d56 P dd32a35972b347b89ceaaad2455aef5a: Bootstrap starting.
I20260130 02:59:10.848033 22514 tablet_service.cc:1505] Processing CreateTablet for tablet f41b5d6226db4f4287f8a4cdea9a0bdc (DEFAULT_TABLE table=default.table_62e153a4e7844d97863f32a6e87da70b [id=dee95603eab3439b81355a006bc9f376]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:59:10.849408 22514 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f41b5d6226db4f4287f8a4cdea9a0bdc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:59:10.849751 22643 tablet_bootstrap.cc:654] T 9ee21c80f1ee4d16b777130f87124d56 P dd32a35972b347b89ceaaad2455aef5a: Neither blocks nor log segments found. Creating new log.
I20260130 02:59:10.853979 22642 tablet_bootstrap.cc:492] T 9ee21c80f1ee4d16b777130f87124d56 P 789dbb4a27204ffbb68d7ae40f8f5e4a: Bootstrap starting.
@ 0x7fb4a53977ca _ZNSt3__18__invokeIRZN4kudu6master30HmsNotificationLogListenerTask4InitEvE3$_0JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
I20260130 02:59:10.859817 21881 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34990:
name: "default.Table_2dde477c96844e51bc37ab8d6bd13248"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260130 02:59:10.860358 22642 tablet_bootstrap.cc:654] T 9ee21c80f1ee4d16b777130f87124d56 P 789dbb4a27204ffbb68d7ae40f8f5e4a: Neither blocks nor log segments found. Creating new log.
I20260130 02:59:10.870211 22642 tablet_bootstrap.cc:492] T 9ee21c80f1ee4d16b777130f87124d56 P 789dbb4a27204ffbb68d7ae40f8f5e4a: No bootstrap required, opened a new log
I20260130 02:59:10.870659 22642 ts_tablet_manager.cc:1403] T 9ee21c80f1ee4d16b777130f87124d56 P 789dbb4a27204ffbb68d7ae40f8f5e4a: Time spent bootstrapping tablet: real 0.017s user 0.009s sys 0.004s
I20260130 02:59:10.873160 22642 raft_consensus.cc:359] T 9ee21c80f1ee4d16b777130f87124d56 P 789dbb4a27204ffbb68d7ae40f8f5e4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111215cc989f4203a6e1517f2291d66f" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 39751 } } peers { permanent_uuid: "dd32a35972b347b89ceaaad2455aef5a" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 39781 } } peers { permanent_uuid: "789dbb4a27204ffbb68d7ae40f8f5e4a" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 41825 } }
I20260130 02:59:10.873821 22642 raft_consensus.cc:385] T 9ee21c80f1ee4d16b777130f87124d56 P 789dbb4a27204ffbb68d7ae40f8f5e4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:59:10.874097 22642 raft_consensus.cc:740] T 9ee21c80f1ee4d16b777130f87124d56 P 789dbb4a27204ffbb68d7ae40f8f5e4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 789dbb4a27204ffbb68d7ae40f8f5e4a, State: Initialized, Role: FOLLOWER
@ 0x7fb4a539775a std::__1::__invoke_void_return_wrapper<>::__call<>() at ??:0
I20260130 02:59:10.880399 22642 consensus_queue.cc:260] T 9ee21c80f1ee4d16b777130f87124d56 P 789dbb4a27204ffbb68d7ae40f8f5e4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111215cc989f4203a6e1517f2291d66f" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 39751 } } peers { permanent_uuid: "dd32a35972b347b89ceaaad2455aef5a" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 39781 } } peers { permanent_uuid: "789dbb4a27204ffbb68d7ae40f8f5e4a" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 41825 } }
I20260130 02:59:10.894138 22642 ts_tablet_manager.cc:1434] T 9ee21c80f1ee4d16b777130f87124d56 P 789dbb4a27204ffbb68d7ae40f8f5e4a: Time spent starting tablet: real 0.023s user 0.005s sys 0.001s
I20260130 02:59:10.895972 21885 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34990:
name: "default.Table_53c1c4a7b44f4bb095c3167a66cf3315"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
@ 0x7fb4a5397722 std::__1::__function::__alloc_func<>::operator()() at ??:0
I20260130 02:59:10.916349 21879 master_service.cc:954] ReplaceTablet: received request to replace tablet 8db03bbf33c641b68deb9de324745ac0 from {username='slave'} at 127.0.0.1:34990
I20260130 02:59:10.919031 21878 master_service.cc:954] ReplaceTablet: received request to replace tablet 8db03bbf33c641b68deb9de324745ac0 from {username='slave'} at 127.0.0.1:34990
@ 0x7fb4a5396a1e std::__1::__function::__func<>::operator()() at ??:0
@ 0x5aa2b5 std::__1::__function::__value_func<>::operator()() at ??:?
@ 0x5aa0ea std::__1::function<>::operator()() at ??:?
@ 0x7fb49c51669b kudu::Thread::SuperviseThread() at ??:0
@ 0x474190 __tsan_thread_start_func at ??:?
I20260130 02:59:10.948416 21879 catalog_manager.cc:5941] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324: Sending DeleteTablet for 3 replicas of tablet 8db03bbf33c641b68deb9de324745ac0
I20260130 02:59:10.951390 22225 tablet_service.cc:1552] Processing DeleteTablet for tablet 8db03bbf33c641b68deb9de324745ac0 with delete_type TABLET_DATA_DELETED (replaced by tablet 7e4c634c4cbf4dfbb7caf795e9f9e5c2) from {username='slave'} at 127.0.0.1:53816
I20260130 02:59:10.952136 22651 tablet_replica.cc:333] T 8db03bbf33c641b68deb9de324745ac0 P 111215cc989f4203a6e1517f2291d66f: stopping tablet replica
I20260130 02:59:10.952629 21879 catalog_manager.cc:6724] ReplaceTablet: tablet 8db03bbf33c641b68deb9de324745ac0 deleted and replaced by tablet 7e4c634c4cbf4dfbb7caf795e9f9e5c2
I20260130 02:59:10.952783 22651 raft_consensus.cc:2243] T 8db03bbf33c641b68deb9de324745ac0 P 111215cc989f4203a6e1517f2291d66f [term 0 FOLLOWER]: Raft consensus shutting down.
I20260130 02:59:10.952732 22515 tablet_service.cc:1552] Processing DeleteTablet for tablet 8db03bbf33c641b68deb9de324745ac0 with delete_type TABLET_DATA_DELETED (replaced by tablet 7e4c634c4cbf4dfbb7caf795e9f9e5c2) from {username='slave'} at 127.0.0.1:53088
I20260130 02:59:10.953234 22651 raft_consensus.cc:2272] T 8db03bbf33c641b68deb9de324745ac0 P 111215cc989f4203a6e1517f2291d66f [term 0 FOLLOWER]: Raft consensus is shut down!
I20260130 02:59:10.953955 22649 tablet_replica.cc:333] T 8db03bbf33c641b68deb9de324745ac0 P 789dbb4a27204ffbb68d7ae40f8f5e4a: stopping tablet replica
I20260130 02:59:10.954661 22649 raft_consensus.cc:2243] T 8db03bbf33c641b68deb9de324745ac0 P 789dbb4a27204ffbb68d7ae40f8f5e4a [term 0 FOLLOWER]: Raft consensus shutting down.
I20260130 02:59:10.955109 22649 raft_consensus.cc:2272] T 8db03bbf33c641b68deb9de324745ac0 P 789dbb4a27204ffbb68d7ae40f8f5e4a [term 0 FOLLOWER]: Raft consensus is shut down!
I20260130 02:59:10.955353 22651 ts_tablet_manager.cc:1916] T 8db03bbf33c641b68deb9de324745ac0 P 111215cc989f4203a6e1517f2291d66f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260130 02:59:10.957077 22649 ts_tablet_manager.cc:1916] T 8db03bbf33c641b68deb9de324745ac0 P 789dbb4a27204ffbb68d7ae40f8f5e4a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260130 02:59:10.957198 22369 tablet_service.cc:1552] Processing DeleteTablet for tablet 8db03bbf33c641b68deb9de324745ac0 with delete_type TABLET_DATA_DELETED (replaced by tablet 7e4c634c4cbf4dfbb7caf795e9f9e5c2) from {username='slave'} at 127.0.0.1:44082
I20260130 02:59:10.957952 22650 tablet_replica.cc:333] T 8db03bbf33c641b68deb9de324745ac0 P dd32a35972b347b89ceaaad2455aef5a: stopping tablet replica
I20260130 02:59:10.958588 22650 raft_consensus.cc:2243] T 8db03bbf33c641b68deb9de324745ac0 P dd32a35972b347b89ceaaad2455aef5a [term 0 FOLLOWER]: Raft consensus shutting down.
I20260130 02:59:10.959014 22650 raft_consensus.cc:2272] T 8db03bbf33c641b68deb9de324745ac0 P dd32a35972b347b89ceaaad2455aef5a [term 0 FOLLOWER]: Raft consensus is shut down!
I20260130 02:59:10.961000 22650 ts_tablet_manager.cc:1916] T 8db03bbf33c641b68deb9de324745ac0 P dd32a35972b347b89ceaaad2455aef5a: Deleting tablet data with delete state TABLET_DATA_DELETED
@ 0x7fb49dc216db start_thread at ??:0
I20260130 02:59:10.973744 22649 ts_tablet_manager.cc:1929] T 8db03bbf33c641b68deb9de324745ac0 P 789dbb4a27204ffbb68d7ae40f8f5e4a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260130 02:59:10.976388 22649 log.cc:1199] T 8db03bbf33c641b68deb9de324745ac0 P 789dbb4a27204ffbb68d7ae40f8f5e4a: Deleting WAL directory at /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1769741859011789-18833-0/minicluster-data/ts-2/wal/wals/8db03bbf33c641b68deb9de324745ac0
@ 0x7fb498f9971f clone at ??:0
I20260130 02:59:10.977008 22649 ts_tablet_manager.cc:1950] T 8db03bbf33c641b68deb9de324745ac0 P 789dbb4a27204ffbb68d7ae40f8f5e4a: Deleting consensus metadata
I20260130 02:59:10.989514 22650 ts_tablet_manager.cc:1929] T 8db03bbf33c641b68deb9de324745ac0 P dd32a35972b347b89ceaaad2455aef5a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260130 02:59:10.989908 22650 log.cc:1199] T 8db03bbf33c641b68deb9de324745ac0 P dd32a35972b347b89ceaaad2455aef5a: Deleting WAL directory at /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1769741859011789-18833-0/minicluster-data/ts-1/wal/wals/8db03bbf33c641b68deb9de324745ac0
I20260130 02:59:10.990592 22650 ts_tablet_manager.cc:1950] T 8db03bbf33c641b68deb9de324745ac0 P dd32a35972b347b89ceaaad2455aef5a: Deleting consensus metadata
I20260130 02:59:10.991910 22651 ts_tablet_manager.cc:1929] T 8db03bbf33c641b68deb9de324745ac0 P 111215cc989f4203a6e1517f2291d66f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260130 02:59:10.992286 22651 log.cc:1199] T 8db03bbf33c641b68deb9de324745ac0 P 111215cc989f4203a6e1517f2291d66f: Deleting WAL directory at /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1769741859011789-18833-0/minicluster-data/ts-0/wal/wals/8db03bbf33c641b68deb9de324745ac0
I20260130 02:59:10.992869 22651 ts_tablet_manager.cc:1950] T 8db03bbf33c641b68deb9de324745ac0 P 111215cc989f4203a6e1517f2291d66f: Deleting consensus metadata
I20260130 02:59:10.994901 22643 tablet_bootstrap.cc:492] T 9ee21c80f1ee4d16b777130f87124d56 P dd32a35972b347b89ceaaad2455aef5a: No bootstrap required, opened a new log
I20260130 02:59:10.995342 22643 ts_tablet_manager.cc:1403] T 9ee21c80f1ee4d16b777130f87124d56 P dd32a35972b347b89ceaaad2455aef5a: Time spent bootstrapping tablet: real 0.152s user 0.015s sys 0.064s
I20260130 02:59:10.998020 22641 tablet_bootstrap.cc:492] T 9ee21c80f1ee4d16b777130f87124d56 P 111215cc989f4203a6e1517f2291d66f: Bootstrap starting.
I20260130 02:59:11.003790 22641 tablet_bootstrap.cc:654] T 9ee21c80f1ee4d16b777130f87124d56 P 111215cc989f4203a6e1517f2291d66f: Neither blocks nor log segments found. Creating new log.
I20260130 02:59:11.007752 22643 raft_consensus.cc:359] T 9ee21c80f1ee4d16b777130f87124d56 P dd32a35972b347b89ceaaad2455aef5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111215cc989f4203a6e1517f2291d66f" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 39751 } } peers { permanent_uuid: "dd32a35972b347b89ceaaad2455aef5a" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 39781 } } peers { permanent_uuid: "789dbb4a27204ffbb68d7ae40f8f5e4a" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 41825 } }
I20260130 02:59:11.008445 22643 raft_consensus.cc:385] T 9ee21c80f1ee4d16b777130f87124d56 P dd32a35972b347b89ceaaad2455aef5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:59:11.008711 22643 raft_consensus.cc:740] T 9ee21c80f1ee4d16b777130f87124d56 P dd32a35972b347b89ceaaad2455aef5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd32a35972b347b89ceaaad2455aef5a, State: Initialized, Role: FOLLOWER
I20260130 02:59:11.009835 22642 tablet_bootstrap.cc:492] T f41b5d6226db4f4287f8a4cdea9a0bdc P 789dbb4a27204ffbb68d7ae40f8f5e4a: Bootstrap starting.
I20260130 02:59:11.015792 22641 tablet_bootstrap.cc:492] T 9ee21c80f1ee4d16b777130f87124d56 P 111215cc989f4203a6e1517f2291d66f: No bootstrap required, opened a new log
I20260130 02:59:11.016232 22641 ts_tablet_manager.cc:1403] T 9ee21c80f1ee4d16b777130f87124d56 P 111215cc989f4203a6e1517f2291d66f: Time spent bootstrapping tablet: real 0.018s user 0.002s sys 0.011s
I20260130 02:59:11.016247 22642 tablet_bootstrap.cc:654] T f41b5d6226db4f4287f8a4cdea9a0bdc P 789dbb4a27204ffbb68d7ae40f8f5e4a: Neither blocks nor log segments found. Creating new log.
I20260130 02:59:11.020210 22643 consensus_queue.cc:260] T 9ee21c80f1ee4d16b777130f87124d56 P dd32a35972b347b89ceaaad2455aef5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111215cc989f4203a6e1517f2291d66f" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 39751 } } peers { permanent_uuid: "dd32a35972b347b89ceaaad2455aef5a" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 39781 } } peers { permanent_uuid: "789dbb4a27204ffbb68d7ae40f8f5e4a" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 41825 } }
I20260130 02:59:11.018713 22641 raft_consensus.cc:359] T 9ee21c80f1ee4d16b777130f87124d56 P 111215cc989f4203a6e1517f2291d66f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111215cc989f4203a6e1517f2291d66f" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 39751 } } peers { permanent_uuid: "dd32a35972b347b89ceaaad2455aef5a" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 39781 } } peers { permanent_uuid: "789dbb4a27204ffbb68d7ae40f8f5e4a" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 41825 } }
I20260130 02:59:11.021364 22641 raft_consensus.cc:385] T 9ee21c80f1ee4d16b777130f87124d56 P 111215cc989f4203a6e1517f2291d66f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:59:11.021631 22641 raft_consensus.cc:740] T 9ee21c80f1ee4d16b777130f87124d56 P 111215cc989f4203a6e1517f2291d66f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 111215cc989f4203a6e1517f2291d66f, State: Initialized, Role: FOLLOWER
I20260130 02:59:11.027443 22643 ts_tablet_manager.cc:1434] T 9ee21c80f1ee4d16b777130f87124d56 P dd32a35972b347b89ceaaad2455aef5a: Time spent starting tablet: real 0.032s user 0.007s sys 0.000s
I20260130 02:59:11.022245 22641 consensus_queue.cc:260] T 9ee21c80f1ee4d16b777130f87124d56 P 111215cc989f4203a6e1517f2291d66f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111215cc989f4203a6e1517f2291d66f" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 39751 } } peers { permanent_uuid: "dd32a35972b347b89ceaaad2455aef5a" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 39781 } } peers { permanent_uuid: "789dbb4a27204ffbb68d7ae40f8f5e4a" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 41825 } }
I20260130 02:59:11.031577 22641 ts_tablet_manager.cc:1434] T 9ee21c80f1ee4d16b777130f87124d56 P 111215cc989f4203a6e1517f2291d66f: Time spent starting tablet: real 0.015s user 0.005s sys 0.002s
I20260130 02:59:11.032428 22641 tablet_bootstrap.cc:492] T f41b5d6226db4f4287f8a4cdea9a0bdc P 111215cc989f4203a6e1517f2291d66f: Bootstrap starting.
I20260130 02:59:11.036257 22643 tablet_bootstrap.cc:492] T f41b5d6226db4f4287f8a4cdea9a0bdc P dd32a35972b347b89ceaaad2455aef5a: Bootstrap starting.
I20260130 02:59:11.041966 22643 tablet_bootstrap.cc:654] T f41b5d6226db4f4287f8a4cdea9a0bdc P dd32a35972b347b89ceaaad2455aef5a: Neither blocks nor log segments found. Creating new log.
I20260130 02:59:11.044355 22642 tablet_bootstrap.cc:492] T f41b5d6226db4f4287f8a4cdea9a0bdc P 789dbb4a27204ffbb68d7ae40f8f5e4a: No bootstrap required, opened a new log
I20260130 02:59:11.044803 22642 ts_tablet_manager.cc:1403] T f41b5d6226db4f4287f8a4cdea9a0bdc P 789dbb4a27204ffbb68d7ae40f8f5e4a: Time spent bootstrapping tablet: real 0.035s user 0.010s sys 0.000s
I20260130 02:59:11.047880 22643 tablet_bootstrap.cc:492] T f41b5d6226db4f4287f8a4cdea9a0bdc P dd32a35972b347b89ceaaad2455aef5a: No bootstrap required, opened a new log
I20260130 02:59:11.047647 22642 raft_consensus.cc:359] T f41b5d6226db4f4287f8a4cdea9a0bdc P 789dbb4a27204ffbb68d7ae40f8f5e4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111215cc989f4203a6e1517f2291d66f" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 39751 } } peers { permanent_uuid: "dd32a35972b347b89ceaaad2455aef5a" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 39781 } } peers { permanent_uuid: "789dbb4a27204ffbb68d7ae40f8f5e4a" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 41825 } }
I20260130 02:59:11.048291 22642 raft_consensus.cc:385] T f41b5d6226db4f4287f8a4cdea9a0bdc P 789dbb4a27204ffbb68d7ae40f8f5e4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:59:11.048385 22643 ts_tablet_manager.cc:1403] T f41b5d6226db4f4287f8a4cdea9a0bdc P dd32a35972b347b89ceaaad2455aef5a: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.004s
I20260130 02:59:11.048547 22642 raft_consensus.cc:740] T f41b5d6226db4f4287f8a4cdea9a0bdc P 789dbb4a27204ffbb68d7ae40f8f5e4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 789dbb4a27204ffbb68d7ae40f8f5e4a, State: Initialized, Role: FOLLOWER
I20260130 02:59:11.049221 22642 consensus_queue.cc:260] T f41b5d6226db4f4287f8a4cdea9a0bdc P 789dbb4a27204ffbb68d7ae40f8f5e4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111215cc989f4203a6e1517f2291d66f" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 39751 } } peers { permanent_uuid: "dd32a35972b347b89ceaaad2455aef5a" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 39781 } } peers { permanent_uuid: "789dbb4a27204ffbb68d7ae40f8f5e4a" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 41825 } }
I20260130 02:59:11.051057 22643 raft_consensus.cc:359] T f41b5d6226db4f4287f8a4cdea9a0bdc P dd32a35972b347b89ceaaad2455aef5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111215cc989f4203a6e1517f2291d66f" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 39751 } } peers { permanent_uuid: "dd32a35972b347b89ceaaad2455aef5a" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 39781 } } peers { permanent_uuid: "789dbb4a27204ffbb68d7ae40f8f5e4a" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 41825 } }
I20260130 02:59:11.051683 22643 raft_consensus.cc:385] T f41b5d6226db4f4287f8a4cdea9a0bdc P dd32a35972b347b89ceaaad2455aef5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:59:11.051945 22643 raft_consensus.cc:740] T f41b5d6226db4f4287f8a4cdea9a0bdc P dd32a35972b347b89ceaaad2455aef5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd32a35972b347b89ceaaad2455aef5a, State: Initialized, Role: FOLLOWER
I20260130 02:59:11.052662 22643 consensus_queue.cc:260] T f41b5d6226db4f4287f8a4cdea9a0bdc P dd32a35972b347b89ceaaad2455aef5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111215cc989f4203a6e1517f2291d66f" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 39751 } } peers { permanent_uuid: "dd32a35972b347b89ceaaad2455aef5a" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 39781 } } peers { permanent_uuid: "789dbb4a27204ffbb68d7ae40f8f5e4a" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 41825 } }
I20260130 02:59:11.056025 22641 tablet_bootstrap.cc:654] T f41b5d6226db4f4287f8a4cdea9a0bdc P 111215cc989f4203a6e1517f2291d66f: Neither blocks nor log segments found. Creating new log.
I20260130 02:59:11.057564 22643 ts_tablet_manager.cc:1434] T f41b5d6226db4f4287f8a4cdea9a0bdc P dd32a35972b347b89ceaaad2455aef5a: Time spent starting tablet: real 0.009s user 0.007s sys 0.000s
I20260130 02:59:11.060575 22642 ts_tablet_manager.cc:1434] T f41b5d6226db4f4287f8a4cdea9a0bdc P 789dbb4a27204ffbb68d7ae40f8f5e4a: Time spent starting tablet: real 0.015s user 0.007s sys 0.000s
I20260130 02:59:11.073280 22641 tablet_bootstrap.cc:492] T f41b5d6226db4f4287f8a4cdea9a0bdc P 111215cc989f4203a6e1517f2291d66f: No bootstrap required, opened a new log
I20260130 02:59:11.073755 22641 ts_tablet_manager.cc:1403] T f41b5d6226db4f4287f8a4cdea9a0bdc P 111215cc989f4203a6e1517f2291d66f: Time spent bootstrapping tablet: real 0.042s user 0.009s sys 0.006s
I20260130 02:59:11.076525 22641 raft_consensus.cc:359] T f41b5d6226db4f4287f8a4cdea9a0bdc P 111215cc989f4203a6e1517f2291d66f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111215cc989f4203a6e1517f2291d66f" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 39751 } } peers { permanent_uuid: "dd32a35972b347b89ceaaad2455aef5a" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 39781 } } peers { permanent_uuid: "789dbb4a27204ffbb68d7ae40f8f5e4a" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 41825 } }
I20260130 02:59:11.077193 22641 raft_consensus.cc:385] T f41b5d6226db4f4287f8a4cdea9a0bdc P 111215cc989f4203a6e1517f2291d66f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:59:11.077474 22641 raft_consensus.cc:740] T f41b5d6226db4f4287f8a4cdea9a0bdc P 111215cc989f4203a6e1517f2291d66f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 111215cc989f4203a6e1517f2291d66f, State: Initialized, Role: FOLLOWER
I20260130 02:59:11.078270 22641 consensus_queue.cc:260] T f41b5d6226db4f4287f8a4cdea9a0bdc P 111215cc989f4203a6e1517f2291d66f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111215cc989f4203a6e1517f2291d66f" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 39751 } } peers { permanent_uuid: "dd32a35972b347b89ceaaad2455aef5a" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 39781 } } peers { permanent_uuid: "789dbb4a27204ffbb68d7ae40f8f5e4a" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 41825 } }
I20260130 02:59:11.080789 22641 ts_tablet_manager.cc:1434] T f41b5d6226db4f4287f8a4cdea9a0bdc P 111215cc989f4203a6e1517f2291d66f: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
W20260130 02:59:11.110826 22646 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260130 02:59:11.111601 22646 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260130 02:59:11.112314 22646 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260130 02:59:11.169258 22646 flags.cc:425] Enabled experimental flag: --enable_metadata_cleanup_for_deleted_tables_and_tablets=true
W20260130 02:59:11.169557 22646 flags.cc:425] Enabled experimental flag: --metadata_for_deleted_table_and_tablet_reserved_secs=0
W20260130 02:59:11.169817 22646 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260130 02:59:11.170087 22646 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260130 02:59:11.170358 22646 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260130 02:59:11.170694 22646 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260130 02:59:11.209551 22652 raft_consensus.cc:493] T f41b5d6226db4f4287f8a4cdea9a0bdc P 111215cc989f4203a6e1517f2291d66f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:59:11.209975 22652 raft_consensus.cc:515] T f41b5d6226db4f4287f8a4cdea9a0bdc P 111215cc989f4203a6e1517f2291d66f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111215cc989f4203a6e1517f2291d66f" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 39751 } } peers { permanent_uuid: "dd32a35972b347b89ceaaad2455aef5a" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 39781 } } peers { permanent_uuid: "789dbb4a27204ffbb68d7ae40f8f5e4a" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 41825 } }
I20260130 02:59:11.211743 22652 leader_election.cc:290] T f41b5d6226db4f4287f8a4cdea9a0bdc P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dd32a35972b347b89ceaaad2455aef5a (127.18.100.66:39781), 789dbb4a27204ffbb68d7ae40f8f5e4a (127.18.100.67:41825)
I20260130 02:59:11.213696 22535 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f41b5d6226db4f4287f8a4cdea9a0bdc" candidate_uuid: "111215cc989f4203a6e1517f2291d66f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "789dbb4a27204ffbb68d7ae40f8f5e4a" is_pre_election: true
I20260130 02:59:11.214306 22535 raft_consensus.cc:2468] T f41b5d6226db4f4287f8a4cdea9a0bdc P 789dbb4a27204ffbb68d7ae40f8f5e4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 111215cc989f4203a6e1517f2291d66f in term 0.
I20260130 02:59:11.215627 22391 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f41b5d6226db4f4287f8a4cdea9a0bdc" candidate_uuid: "111215cc989f4203a6e1517f2291d66f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd32a35972b347b89ceaaad2455aef5a" is_pre_election: true
I20260130 02:59:11.216188 22391 raft_consensus.cc:2468] T f41b5d6226db4f4287f8a4cdea9a0bdc P dd32a35972b347b89ceaaad2455aef5a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 111215cc989f4203a6e1517f2291d66f in term 0.
I20260130 02:59:11.217058 22179 leader_election.cc:304] T f41b5d6226db4f4287f8a4cdea9a0bdc P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 111215cc989f4203a6e1517f2291d66f, 789dbb4a27204ffbb68d7ae40f8f5e4a; no voters:
I20260130 02:59:11.218708 22652 raft_consensus.cc:2804] T f41b5d6226db4f4287f8a4cdea9a0bdc P 111215cc989f4203a6e1517f2291d66f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260130 02:59:11.219110 22652 raft_consensus.cc:493] T f41b5d6226db4f4287f8a4cdea9a0bdc P 111215cc989f4203a6e1517f2291d66f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260130 02:59:11.219403 22652 raft_consensus.cc:3060] T f41b5d6226db4f4287f8a4cdea9a0bdc P 111215cc989f4203a6e1517f2291d66f [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:59:11.225572 22652 raft_consensus.cc:515] T f41b5d6226db4f4287f8a4cdea9a0bdc P 111215cc989f4203a6e1517f2291d66f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111215cc989f4203a6e1517f2291d66f" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 39751 } } peers { permanent_uuid: "dd32a35972b347b89ceaaad2455aef5a" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 39781 } } peers { permanent_uuid: "789dbb4a27204ffbb68d7ae40f8f5e4a" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 41825 } }
I20260130 02:59:11.228350 22391 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f41b5d6226db4f4287f8a4cdea9a0bdc" candidate_uuid: "111215cc989f4203a6e1517f2291d66f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd32a35972b347b89ceaaad2455aef5a"
I20260130 02:59:11.228510 22535 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f41b5d6226db4f4287f8a4cdea9a0bdc" candidate_uuid: "111215cc989f4203a6e1517f2291d66f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "789dbb4a27204ffbb68d7ae40f8f5e4a"
I20260130 02:59:11.228888 22391 raft_consensus.cc:3060] T f41b5d6226db4f4287f8a4cdea9a0bdc P dd32a35972b347b89ceaaad2455aef5a [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:59:11.229005 22535 raft_consensus.cc:3060] T f41b5d6226db4f4287f8a4cdea9a0bdc P 789dbb4a27204ffbb68d7ae40f8f5e4a [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:59:11.229213 22646 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.100.84:33971
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1769741859011789-18833-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1769741859011789-18833-0/minicluster-data/master-2/wal
--hive_metastore_uris=thrift://127.0.0.1:42185
--catalog_manager_bg_task_wait_ms=10
--enable_metadata_cleanup_for_deleted_tables_and_tablets=true
--metadata_for_deleted_table_and_tablet_reserved_secs=0
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.18.100.126:42077,127.18.100.125:39157,127.18.100.124:34987
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.100.124:34987
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1769741859011789-18833-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.18.100.124
--webserver_port=34049
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1769741859011789-18833-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type FASTDEBUG
built by None at 30 Jan 2026 02:43:21 UTC on 5fd53c4cbb9d
build id 10257
TSAN enabled
I20260130 02:59:11.230930 22646 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260130 02:59:11.233038 22646 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20260130 02:59:11.236148 22535 raft_consensus.cc:2468] T f41b5d6226db4f4287f8a4cdea9a0bdc P 789dbb4a27204ffbb68d7ae40f8f5e4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 111215cc989f4203a6e1517f2291d66f in term 1.
I20260130 02:59:11.237228 22179 leader_election.cc:304] T f41b5d6226db4f4287f8a4cdea9a0bdc P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 111215cc989f4203a6e1517f2291d66f, 789dbb4a27204ffbb68d7ae40f8f5e4a; no voters:
I20260130 02:59:11.242949 22652 leader_election.cc:290] T f41b5d6226db4f4287f8a4cdea9a0bdc P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 election: Requested vote from peers dd32a35972b347b89ceaaad2455aef5a (127.18.100.66:39781), 789dbb4a27204ffbb68d7ae40f8f5e4a (127.18.100.67:41825)
I20260130 02:59:11.243873 22652 raft_consensus.cc:2804] T f41b5d6226db4f4287f8a4cdea9a0bdc P 111215cc989f4203a6e1517f2291d66f [term 1 FOLLOWER]: Leader election won for term 1
I20260130 02:59:11.244870 22652 raft_consensus.cc:697] T f41b5d6226db4f4287f8a4cdea9a0bdc P 111215cc989f4203a6e1517f2291d66f [term 1 LEADER]: Becoming Leader. State: Replica: 111215cc989f4203a6e1517f2291d66f, State: Running, Role: LEADER
I20260130 02:59:11.244633 22391 raft_consensus.cc:2468] T f41b5d6226db4f4287f8a4cdea9a0bdc P dd32a35972b347b89ceaaad2455aef5a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 111215cc989f4203a6e1517f2291d66f in term 1.
I20260130 02:59:11.246171 22652 consensus_queue.cc:237] T f41b5d6226db4f4287f8a4cdea9a0bdc P 111215cc989f4203a6e1517f2291d66f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111215cc989f4203a6e1517f2291d66f" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 39751 } } peers { permanent_uuid: "dd32a35972b347b89ceaaad2455aef5a" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 39781 } } peers { permanent_uuid: "789dbb4a27204ffbb68d7ae40f8f5e4a" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 41825 } }
W20260130 02:59:11.269152 22693 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260130 02:59:11.269778 22653 raft_consensus.cc:493] T 9ee21c80f1ee4d16b777130f87124d56 P 789dbb4a27204ffbb68d7ae40f8f5e4a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:59:11.270280 22653 raft_consensus.cc:515] T 9ee21c80f1ee4d16b777130f87124d56 P 789dbb4a27204ffbb68d7ae40f8f5e4a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111215cc989f4203a6e1517f2291d66f" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 39751 } } peers { permanent_uuid: "dd32a35972b347b89ceaaad2455aef5a" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 39781 } } peers { permanent_uuid: "789dbb4a27204ffbb68d7ae40f8f5e4a" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 41825 } }
I20260130 02:59:11.285017 22653 leader_election.cc:290] T 9ee21c80f1ee4d16b777130f87124d56 P 789dbb4a27204ffbb68d7ae40f8f5e4a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 111215cc989f4203a6e1517f2291d66f (127.18.100.65:39751), dd32a35972b347b89ceaaad2455aef5a (127.18.100.66:39781)
W20260130 02:59:11.289263 22179 connection.cc:537] server connection from 127.0.0.1:53816 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260130 02:59:11.293897 22323 connection.cc:537] client connection to 127.18.100.126:42077 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260130 02:59:11.294497 22593 connection.cc:537] client connection to 127.18.100.126:42077 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260130 02:59:11.295660 22470 connection.cc:537] client connection to 127.18.100.126:42077 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260130 02:59:11.296911 22593 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.18.100.126:42077)
W20260130 02:59:11.298141 22593 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.18.100.126:42077)
W20260130 02:59:11.298849 22593 master_proxy_rpc.cc:203] Re-attempting DeleteTable request to leader Master (127.18.100.126:42077)
W20260130 02:59:11.298791 22323 connection.cc:537] server connection from 127.0.0.1:44082 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260130 02:59:11.299576 22471 connection.cc:537] server connection from 127.0.0.1:53088 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260130 02:59:11.300364 22181 connection.cc:537] client connection to 127.18.100.126:42077 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260130 02:59:11.301198 22593 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.18.100.126:42077)
W20260130 02:59:11.303879 22626 negotiation.cc:337] Failed RPC negotiation. Trace:
0130 02:59:11.295660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.100.126:42077 (local address 127.18.100.66:56671)
0130 02:59:11.296555 (+ 895us) negotiation.cc:107] Waiting for socket to connect
0130 02:59:11.296598 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0130 02:59:11.296853 (+ 255us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0130 02:59:11.302752 (+ 5899us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.100.126:42077: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":294}
W20260130 02:59:11.306133 22648 negotiation.cc:337] Failed RPC negotiation. Trace:
0130 02:59:11.297799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.100.126:42077 (local address 127.18.100.67:44177)
0130 02:59:11.300028 (+ 2229us) negotiation.cc:107] Waiting for socket to connect
0130 02:59:11.300085 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0130 02:59:11.300332 (+ 247us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0130 02:59:11.305100 (+ 4768us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.100.126:42077: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":304}
W20260130 02:59:11.309788 22659 negotiation.cc:337] Failed RPC negotiation. Trace:
0130 02:59:11.301893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.100.126:42077 (local address 127.18.100.65:46493)
0130 02:59:11.302688 (+ 795us) negotiation.cc:107] Waiting for socket to connect
0130 02:59:11.302736 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0130 02:59:11.302987 (+ 251us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0130 02:59:11.308651 (+ 5664us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.100.126:42077: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":82}
I20260130 02:59:11.310927 22248 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9ee21c80f1ee4d16b777130f87124d56" candidate_uuid: "789dbb4a27204ffbb68d7ae40f8f5e4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "111215cc989f4203a6e1517f2291d66f" is_pre_election: true
I20260130 02:59:11.311715 22248 raft_consensus.cc:2468] T 9ee21c80f1ee4d16b777130f87124d56 P 111215cc989f4203a6e1517f2291d66f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 789dbb4a27204ffbb68d7ae40f8f5e4a in term 0.
I20260130 02:59:11.313838 22468 leader_election.cc:304] T 9ee21c80f1ee4d16b777130f87124d56 P 789dbb4a27204ffbb68d7ae40f8f5e4a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 111215cc989f4203a6e1517f2291d66f, 789dbb4a27204ffbb68d7ae40f8f5e4a; no voters:
I20260130 02:59:11.314977 22653 raft_consensus.cc:2804] T 9ee21c80f1ee4d16b777130f87124d56 P 789dbb4a27204ffbb68d7ae40f8f5e4a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260130 02:59:11.315373 22653 raft_consensus.cc:493] T 9ee21c80f1ee4d16b777130f87124d56 P 789dbb4a27204ffbb68d7ae40f8f5e4a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260130 02:59:11.315702 22653 raft_consensus.cc:3060] T 9ee21c80f1ee4d16b777130f87124d56 P 789dbb4a27204ffbb68d7ae40f8f5e4a [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:59:11.324780 22653 raft_consensus.cc:515] T 9ee21c80f1ee4d16b777130f87124d56 P 789dbb4a27204ffbb68d7ae40f8f5e4a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111215cc989f4203a6e1517f2291d66f" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 39751 } } peers { permanent_uuid: "dd32a35972b347b89ceaaad2455aef5a" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 39781 } } peers { permanent_uuid: "789dbb4a27204ffbb68d7ae40f8f5e4a" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 41825 } }
I20260130 02:59:11.325587 22391 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9ee21c80f1ee4d16b777130f87124d56" candidate_uuid: "789dbb4a27204ffbb68d7ae40f8f5e4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd32a35972b347b89ceaaad2455aef5a" is_pre_election: true
I20260130 02:59:11.326195 22391 raft_consensus.cc:2468] T 9ee21c80f1ee4d16b777130f87124d56 P dd32a35972b347b89ceaaad2455aef5a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 789dbb4a27204ffbb68d7ae40f8f5e4a in term 0.
I20260130 02:59:11.328274 22248 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9ee21c80f1ee4d16b777130f87124d56" candidate_uuid: "789dbb4a27204ffbb68d7ae40f8f5e4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "111215cc989f4203a6e1517f2291d66f"
I20260130 02:59:11.328984 22248 raft_consensus.cc:3060] T 9ee21c80f1ee4d16b777130f87124d56 P 111215cc989f4203a6e1517f2291d66f [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:59:11.341605 22391 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9ee21c80f1ee4d16b777130f87124d56" candidate_uuid: "789dbb4a27204ffbb68d7ae40f8f5e4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd32a35972b347b89ceaaad2455aef5a"
I20260130 02:59:11.342332 22391 raft_consensus.cc:3060] T 9ee21c80f1ee4d16b777130f87124d56 P dd32a35972b347b89ceaaad2455aef5a [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:59:11.351881 22391 raft_consensus.cc:2468] T 9ee21c80f1ee4d16b777130f87124d56 P dd32a35972b347b89ceaaad2455aef5a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 789dbb4a27204ffbb68d7ae40f8f5e4a in term 1.
I20260130 02:59:11.353775 22469 leader_election.cc:304] T 9ee21c80f1ee4d16b777130f87124d56 P 789dbb4a27204ffbb68d7ae40f8f5e4a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 789dbb4a27204ffbb68d7ae40f8f5e4a, dd32a35972b347b89ceaaad2455aef5a; no voters:
I20260130 02:59:11.363914 22653 leader_election.cc:290] T 9ee21c80f1ee4d16b777130f87124d56 P 789dbb4a27204ffbb68d7ae40f8f5e4a [CANDIDATE]: Term 1 election: Requested vote from peers 111215cc989f4203a6e1517f2291d66f (127.18.100.65:39751), dd32a35972b347b89ceaaad2455aef5a (127.18.100.66:39781)
I20260130 02:59:11.368023 22248 raft_consensus.cc:2468] T 9ee21c80f1ee4d16b777130f87124d56 P 111215cc989f4203a6e1517f2291d66f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 789dbb4a27204ffbb68d7ae40f8f5e4a in term 1.
I20260130 02:59:11.369026 22702 raft_consensus.cc:2804] T 9ee21c80f1ee4d16b777130f87124d56 P 789dbb4a27204ffbb68d7ae40f8f5e4a [term 1 FOLLOWER]: Leader election won for term 1
I20260130 02:59:11.372325 22702 raft_consensus.cc:697] T 9ee21c80f1ee4d16b777130f87124d56 P 789dbb4a27204ffbb68d7ae40f8f5e4a [term 1 LEADER]: Becoming Leader. State: Replica: 789dbb4a27204ffbb68d7ae40f8f5e4a, State: Running, Role: LEADER
I20260130 02:59:11.374372 22702 consensus_queue.cc:237] T 9ee21c80f1ee4d16b777130f87124d56 P 789dbb4a27204ffbb68d7ae40f8f5e4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111215cc989f4203a6e1517f2291d66f" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 39751 } } peers { permanent_uuid: "dd32a35972b347b89ceaaad2455aef5a" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 39781 } } peers { permanent_uuid: "789dbb4a27204ffbb68d7ae40f8f5e4a" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 41825 } }
I20260130 02:59:11.456651 22704 raft_consensus.cc:493] T 00000000000000000000000000000000 P b01057b6c5ff4fc4b07253801881fe13 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 22dedcf69e00438999d3a5fedde57324)
I20260130 02:59:11.457305 22704 raft_consensus.cc:515] T 00000000000000000000000000000000 P b01057b6c5ff4fc4b07253801881fe13 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22dedcf69e00438999d3a5fedde57324" member_type: VOTER last_known_addr { host: "127.18.100.126" port: 42077 } } peers { permanent_uuid: "b01057b6c5ff4fc4b07253801881fe13" member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39157 } } peers { permanent_uuid: "342be83f20a648fd849c7d23454fba08" member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34987 } }
I20260130 02:59:11.467085 22704 leader_election.cc:290] T 00000000000000000000000000000000 P b01057b6c5ff4fc4b07253801881fe13 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 22dedcf69e00438999d3a5fedde57324 (127.18.100.126:42077), 342be83f20a648fd849c7d23454fba08 (127.18.100.124:34987)
W20260130 02:59:11.477723 21946 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.18.100.124:34987: connect: Connection refused (error 111) [suppressed 6 similar messages]
W20260130 02:59:11.482813 21946 leader_election.cc:336] T 00000000000000000000000000000000 P b01057b6c5ff4fc4b07253801881fe13 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 342be83f20a648fd849c7d23454fba08 (127.18.100.124:34987): Network error: Client connection negotiation failed: client connection to 127.18.100.124:34987: connect: Connection refused (error 111)
W20260130 02:59:11.488529 21947 leader_election.cc:336] T 00000000000000000000000000000000 P b01057b6c5ff4fc4b07253801881fe13 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 22dedcf69e00438999d3a5fedde57324 (127.18.100.126:42077): Network error: Client connection negotiation failed: client connection to 127.18.100.126:42077: connect: Connection refused (error 111)
I20260130 02:59:11.489064 21947 leader_election.cc:304] T 00000000000000000000000000000000 P b01057b6c5ff4fc4b07253801881fe13 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b01057b6c5ff4fc4b07253801881fe13; no voters: 22dedcf69e00438999d3a5fedde57324, 342be83f20a648fd849c7d23454fba08
I20260130 02:59:11.489799 22704 raft_consensus.cc:2749] T 00000000000000000000000000000000 P b01057b6c5ff4fc4b07253801881fe13 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260130 02:59:12.142361 22710 raft_consensus.cc:493] T 00000000000000000000000000000000 P b01057b6c5ff4fc4b07253801881fe13 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 22dedcf69e00438999d3a5fedde57324)
I20260130 02:59:12.142795 22710 raft_consensus.cc:515] T 00000000000000000000000000000000 P b01057b6c5ff4fc4b07253801881fe13 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22dedcf69e00438999d3a5fedde57324" member_type: VOTER last_known_addr { host: "127.18.100.126" port: 42077 } } peers { permanent_uuid: "b01057b6c5ff4fc4b07253801881fe13" member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39157 } } peers { permanent_uuid: "342be83f20a648fd849c7d23454fba08" member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34987 } }
I20260130 02:59:12.151491 22710 leader_election.cc:290] T 00000000000000000000000000000000 P b01057b6c5ff4fc4b07253801881fe13 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 22dedcf69e00438999d3a5fedde57324 (127.18.100.126:42077), 342be83f20a648fd849c7d23454fba08 (127.18.100.124:34987)
W20260130 02:59:12.168217 21947 leader_election.cc:336] T 00000000000000000000000000000000 P b01057b6c5ff4fc4b07253801881fe13 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 22dedcf69e00438999d3a5fedde57324 (127.18.100.126:42077): Network error: Client connection negotiation failed: client connection to 127.18.100.126:42077: connect: Connection refused (error 111)
W20260130 02:59:12.175462 21946 leader_election.cc:336] T 00000000000000000000000000000000 P b01057b6c5ff4fc4b07253801881fe13 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 342be83f20a648fd849c7d23454fba08 (127.18.100.124:34987): Network error: Client connection negotiation failed: client connection to 127.18.100.124:34987: connect: Connection refused (error 111)
I20260130 02:59:12.175999 21946 leader_election.cc:304] T 00000000000000000000000000000000 P b01057b6c5ff4fc4b07253801881fe13 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b01057b6c5ff4fc4b07253801881fe13; no voters: 22dedcf69e00438999d3a5fedde57324, 342be83f20a648fd849c7d23454fba08
I20260130 02:59:12.176913 22710 raft_consensus.cc:2749] T 00000000000000000000000000000000 P b01057b6c5ff4fc4b07253801881fe13 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260130 02:59:12.180250 22714 consensus_queue.cc:1048] T f41b5d6226db4f4287f8a4cdea9a0bdc P 111215cc989f4203a6e1517f2291d66f [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd32a35972b347b89ceaaad2455aef5a" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 39781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260130 02:59:12.231336 22714 consensus_queue.cc:1048] T f41b5d6226db4f4287f8a4cdea9a0bdc P 111215cc989f4203a6e1517f2291d66f [LEADER]: Connected to new peer: Peer: permanent_uuid: "789dbb4a27204ffbb68d7ae40f8f5e4a" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 41825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260130 02:59:12.261858 22721 consensus_queue.cc:1048] T 9ee21c80f1ee4d16b777130f87124d56 P 789dbb4a27204ffbb68d7ae40f8f5e4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "111215cc989f4203a6e1517f2291d66f" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 39751 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260130 02:59:12.305020 22721 consensus_queue.cc:1048] T 9ee21c80f1ee4d16b777130f87124d56 P 789dbb4a27204ffbb68d7ae40f8f5e4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd32a35972b347b89ceaaad2455aef5a" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 39781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260130 02:59:11.273663 22694 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260130 02:59:12.670280 22691 debug-util.cc:398] Leaking SignalData structure 0x7b0800028120 after lost signal to thread 22646
W20260130 02:59:12.671942 22646 thread.cc:641] GCE (cloud detector) Time spent creating pthread: real 1.400s user 0.500s sys 0.788s
W20260130 02:59:12.672410 22646 thread.cc:608] GCE (cloud detector) Time spent starting thread: real 1.401s user 0.500s sys 0.788s
I20260130 02:59:12.680330 22646 server_base.cc:1060] running on GCE node
W20260130 02:59:12.680610 22731 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260130 02:59:12.681856 22646 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260130 02:59:12.684203 22646 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260130 02:59:12.685571 22646 hybrid_clock.cc:648] HybridClock initialized: now 1769741952685540 us; error 31 us; skew 500 ppm
I20260130 02:59:12.686434 22646 server_base.cc:860] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260130 02:59:12.691149 22734 raft_consensus.cc:493] T 00000000000000000000000000000000 P b01057b6c5ff4fc4b07253801881fe13 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 22dedcf69e00438999d3a5fedde57324)
I20260130 02:59:12.691632 22734 raft_consensus.cc:515] T 00000000000000000000000000000000 P b01057b6c5ff4fc4b07253801881fe13 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22dedcf69e00438999d3a5fedde57324" member_type: VOTER last_known_addr { host: "127.18.100.126" port: 42077 } } peers { permanent_uuid: "b01057b6c5ff4fc4b07253801881fe13" member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39157 } } peers { permanent_uuid: "342be83f20a648fd849c7d23454fba08" member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34987 } }
I20260130 02:59:12.693596 22734 leader_election.cc:290] T 00000000000000000000000000000000 P b01057b6c5ff4fc4b07253801881fe13 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 22dedcf69e00438999d3a5fedde57324 (127.18.100.126:42077), 342be83f20a648fd849c7d23454fba08 (127.18.100.124:34987)
I20260130 02:59:12.693787 22646 webserver.cc:492] Webserver started at http://127.18.100.124:34049/ using document root <none> and password file <none>
I20260130 02:59:12.696034 22646 fs_manager.cc:362] Metadata directory not provided
I20260130 02:59:12.696389 22646 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
W20260130 02:59:12.699311 21946 leader_election.cc:336] T 00000000000000000000000000000000 P b01057b6c5ff4fc4b07253801881fe13 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 342be83f20a648fd849c7d23454fba08 (127.18.100.124:34987): Network error: Client connection negotiation failed: client connection to 127.18.100.124:34987: connect: Connection refused (error 111)
W20260130 02:59:12.699805 21947 leader_election.cc:336] T 00000000000000000000000000000000 P b01057b6c5ff4fc4b07253801881fe13 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 22dedcf69e00438999d3a5fedde57324 (127.18.100.126:42077): Network error: Client connection negotiation failed: client connection to 127.18.100.126:42077: connect: Connection refused (error 111)
I20260130 02:59:12.700098 21947 leader_election.cc:304] T 00000000000000000000000000000000 P b01057b6c5ff4fc4b07253801881fe13 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b01057b6c5ff4fc4b07253801881fe13; no voters: 22dedcf69e00438999d3a5fedde57324, 342be83f20a648fd849c7d23454fba08
I20260130 02:59:12.700637 22734 raft_consensus.cc:2749] T 00000000000000000000000000000000 P b01057b6c5ff4fc4b07253801881fe13 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260130 02:59:12.705245 22646 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.005s sys 0.001s
I20260130 02:59:12.710790 22740 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260130 02:59:12.711809 22646 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260130 02:59:12.712113 22646 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1769741859011789-18833-0/minicluster-data/master-2/data,/tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1769741859011789-18833-0/minicluster-data/master-2/wal
uuid: "342be83f20a648fd849c7d23454fba08"
format_stamp: "Formatted at 2026-01-30 02:59:00 on dist-test-slave-mn07"
I20260130 02:59:12.713989 22646 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1769741859011789-18833-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1769741859011789-18833-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1769741859011789-18833-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260130 02:59:12.760947 22646 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260130 02:59:12.762400 22646 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260130 02:59:12.762835 22646 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260130 02:59:12.843868 22646 rpc_server.cc:307] RPC server started. Bound to: 127.18.100.124:34987
I20260130 02:59:12.843899 22797 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.100.124:34987 every 8 connection(s)
I20260130 02:59:12.846966 22646 server_base.cc:1192] Dumped server information to /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1769741859011789-18833-0/minicluster-data/master-2/data/info.pb
I20260130 02:59:12.854192 18833 external_mini_cluster.cc:1428] Started /tmp/dist-test-tasktGe2wI/build/tsan/bin/kudu as pid 22646
I20260130 02:59:12.859562 22798 sys_catalog.cc:263] Verifying existing consensus state
I20260130 02:59:12.866479 22798 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 342be83f20a648fd849c7d23454fba08: Bootstrap starting.
I20260130 02:59:12.883554 22798 log.cc:826] T 00000000000000000000000000000000 P 342be83f20a648fd849c7d23454fba08: Log is configured to *not* fsync() on all Append() calls
I20260130 02:59:12.901978 22798 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 342be83f20a648fd849c7d23454fba08: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260130 02:59:12.902833 22798 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 342be83f20a648fd849c7d23454fba08: Bootstrap complete.
I20260130 02:59:12.922200 22798 raft_consensus.cc:359] T 00000000000000000000000000000000 P 342be83f20a648fd849c7d23454fba08 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22dedcf69e00438999d3a5fedde57324" member_type: VOTER last_known_addr { host: "127.18.100.126" port: 42077 } } peers { permanent_uuid: "b01057b6c5ff4fc4b07253801881fe13" member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39157 } } peers { permanent_uuid: "342be83f20a648fd849c7d23454fba08" member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34987 } }
I20260130 02:59:12.924782 22798 raft_consensus.cc:740] T 00000000000000000000000000000000 P 342be83f20a648fd849c7d23454fba08 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 342be83f20a648fd849c7d23454fba08, State: Initialized, Role: FOLLOWER
I20260130 02:59:12.925621 22798 consensus_queue.cc:260] T 00000000000000000000000000000000 P 342be83f20a648fd849c7d23454fba08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22dedcf69e00438999d3a5fedde57324" member_type: VOTER last_known_addr { host: "127.18.100.126" port: 42077 } } peers { permanent_uuid: "b01057b6c5ff4fc4b07253801881fe13" member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39157 } } peers { permanent_uuid: "342be83f20a648fd849c7d23454fba08" member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34987 } }
I20260130 02:59:12.927908 22803 sys_catalog.cc:455] T 00000000000000000000000000000000 P 342be83f20a648fd849c7d23454fba08 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22dedcf69e00438999d3a5fedde57324" member_type: VOTER last_known_addr { host: "127.18.100.126" port: 42077 } } peers { permanent_uuid: "b01057b6c5ff4fc4b07253801881fe13" member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39157 } } peers { permanent_uuid: "342be83f20a648fd849c7d23454fba08" member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34987 } } }
I20260130 02:59:12.928524 22803 sys_catalog.cc:458] T 00000000000000000000000000000000 P 342be83f20a648fd849c7d23454fba08 [sys.catalog]: This master's current role is: FOLLOWER
I20260130 02:59:12.929330 22798 sys_catalog.cc:565] T 00000000000000000000000000000000 P 342be83f20a648fd849c7d23454fba08 [sys.catalog]: configured and running, proceeding with master startup.
I20260130 02:59:12.963014 22818 catalog_manager.cc:1269] Loaded cluster ID: 40eb42871f464e95bd641ceacb567979
I20260130 02:59:12.963393 22818 catalog_manager.cc:1562] T 00000000000000000000000000000000 P 342be83f20a648fd849c7d23454fba08: loading cluster ID for follower catalog manager: success
I20260130 02:59:12.978205 22818 catalog_manager.cc:1584] T 00000000000000000000000000000000 P 342be83f20a648fd849c7d23454fba08: acquiring CA information for follower catalog manager: success
I20260130 02:59:12.984000 22818 catalog_manager.cc:1612] T 00000000000000000000000000000000 P 342be83f20a648fd849c7d23454fba08: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260130 02:59:12.964447 22646 master_runner.cc:186] Error getting master registration for 127.18.100.126:42077: Network error: Client connection negotiation failed: client connection to 127.18.100.126:42077: connect: Connection refused (error 111),
I20260130 02:59:12.987381 22646 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260130 02:59:13.154042 22581 heartbeater.cc:344] Connected to a master server at 127.18.100.124:34987
I20260130 02:59:13.154764 22581 heartbeater.cc:499] Master 127.18.100.124:34987 was elected leader, sending a full tablet report...
I20260130 02:59:13.168165 22757 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "789dbb4a27204ffbb68d7ae40f8f5e4a" instance_seqno: 1769741949079638) as {username='slave'} at 127.18.100.67:41921; Asking this server to re-register.
I20260130 02:59:13.170405 22581 heartbeater.cc:461] Registering TS with master...
I20260130 02:59:13.174734 22757 ts_manager.cc:194] Registered new tserver with Master: 789dbb4a27204ffbb68d7ae40f8f5e4a (127.18.100.67:41825)
I20260130 02:59:13.192587 22294 heartbeater.cc:344] Connected to a master server at 127.18.100.124:34987
I20260130 02:59:13.196619 22757 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "111215cc989f4203a6e1517f2291d66f" instance_seqno: 1769741945288540) as {username='slave'} at 127.18.100.65:56935; Asking this server to re-register.
I20260130 02:59:13.197710 22294 heartbeater.cc:461] Registering TS with master...
I20260130 02:59:13.200582 22757 ts_manager.cc:194] Registered new tserver with Master: 111215cc989f4203a6e1517f2291d66f (127.18.100.65:39751)
I20260130 02:59:13.314098 22803 raft_consensus.cc:493] T 00000000000000000000000000000000 P 342be83f20a648fd849c7d23454fba08 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:59:13.314649 22803 raft_consensus.cc:515] T 00000000000000000000000000000000 P 342be83f20a648fd849c7d23454fba08 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22dedcf69e00438999d3a5fedde57324" member_type: VOTER last_known_addr { host: "127.18.100.126" port: 42077 } } peers { permanent_uuid: "b01057b6c5ff4fc4b07253801881fe13" member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39157 } } peers { permanent_uuid: "342be83f20a648fd849c7d23454fba08" member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34987 } }
I20260130 02:59:13.318547 22803 leader_election.cc:290] T 00000000000000000000000000000000 P 342be83f20a648fd849c7d23454fba08 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 22dedcf69e00438999d3a5fedde57324 (127.18.100.126:42077), b01057b6c5ff4fc4b07253801881fe13 (127.18.100.125:39157)
W20260130 02:59:13.319350 22743 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.18.100.126:42077: connect: Connection refused (error 111)
W20260130 02:59:13.334858 22743 leader_election.cc:336] T 00000000000000000000000000000000 P 342be83f20a648fd849c7d23454fba08 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 22dedcf69e00438999d3a5fedde57324 (127.18.100.126:42077): Network error: Client connection negotiation failed: client connection to 127.18.100.126:42077: connect: Connection refused (error 111)
I20260130 02:59:13.339977 21971 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "342be83f20a648fd849c7d23454fba08" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b01057b6c5ff4fc4b07253801881fe13" is_pre_election: true
I20260130 02:59:13.340793 21971 raft_consensus.cc:2393] T 00000000000000000000000000000000 P b01057b6c5ff4fc4b07253801881fe13 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 342be83f20a648fd849c7d23454fba08 in current term 2: Already voted for candidate 22dedcf69e00438999d3a5fedde57324 in this term.
I20260130 02:59:13.343114 22744 leader_election.cc:304] T 00000000000000000000000000000000 P 342be83f20a648fd849c7d23454fba08 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 342be83f20a648fd849c7d23454fba08; no voters: 22dedcf69e00438999d3a5fedde57324, b01057b6c5ff4fc4b07253801881fe13
I20260130 02:59:13.344815 22803 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 342be83f20a648fd849c7d23454fba08 [term 1 FOLLOWER]: Advancing to term 2
I20260130 02:59:13.351310 22438 heartbeater.cc:344] Connected to a master server at 127.18.100.124:34987
I20260130 02:59:13.354439 22803 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 342be83f20a648fd849c7d23454fba08 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260130 02:59:13.356289 22756 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "dd32a35972b347b89ceaaad2455aef5a" instance_seqno: 1769741947198445) as {username='slave'} at 127.18.100.66:35371; Asking this server to re-register.
I20260130 02:59:13.358768 22438 heartbeater.cc:461] Registering TS with master...
I20260130 02:59:13.361418 22757 ts_manager.cc:194] Registered new tserver with Master: dd32a35972b347b89ceaaad2455aef5a (127.18.100.66:39781)
I20260130 02:59:13.447784 22832 raft_consensus.cc:493] T 00000000000000000000000000000000 P b01057b6c5ff4fc4b07253801881fe13 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 22dedcf69e00438999d3a5fedde57324)
I20260130 02:59:13.448276 22832 raft_consensus.cc:515] T 00000000000000000000000000000000 P b01057b6c5ff4fc4b07253801881fe13 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22dedcf69e00438999d3a5fedde57324" member_type: VOTER last_known_addr { host: "127.18.100.126" port: 42077 } } peers { permanent_uuid: "b01057b6c5ff4fc4b07253801881fe13" member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39157 } } peers { permanent_uuid: "342be83f20a648fd849c7d23454fba08" member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34987 } }
I20260130 02:59:13.449800 22832 leader_election.cc:290] T 00000000000000000000000000000000 P b01057b6c5ff4fc4b07253801881fe13 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 22dedcf69e00438999d3a5fedde57324 (127.18.100.126:42077), 342be83f20a648fd849c7d23454fba08 (127.18.100.124:34987)
W20260130 02:59:13.455068 21947 leader_election.cc:336] T 00000000000000000000000000000000 P b01057b6c5ff4fc4b07253801881fe13 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 22dedcf69e00438999d3a5fedde57324 (127.18.100.126:42077): Network error: Client connection negotiation failed: client connection to 127.18.100.126:42077: connect: Connection refused (error 111)
I20260130 02:59:13.462344 22769 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b01057b6c5ff4fc4b07253801881fe13" candidate_term: 3 candidate_status { last_received { term: 2 index: 16 } } ignore_live_leader: false dest_uuid: "342be83f20a648fd849c7d23454fba08" is_pre_election: true
I20260130 02:59:13.463025 22769 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 342be83f20a648fd849c7d23454fba08 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b01057b6c5ff4fc4b07253801881fe13 in term 2.
I20260130 02:59:13.463929 21946 leader_election.cc:304] T 00000000000000000000000000000000 P b01057b6c5ff4fc4b07253801881fe13 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 342be83f20a648fd849c7d23454fba08, b01057b6c5ff4fc4b07253801881fe13; no voters: 22dedcf69e00438999d3a5fedde57324
I20260130 02:59:13.464576 22832 raft_consensus.cc:2804] T 00000000000000000000000000000000 P b01057b6c5ff4fc4b07253801881fe13 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260130 02:59:13.464911 22832 raft_consensus.cc:493] T 00000000000000000000000000000000 P b01057b6c5ff4fc4b07253801881fe13 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 22dedcf69e00438999d3a5fedde57324)
I20260130 02:59:13.465243 22832 raft_consensus.cc:3060] T 00000000000000000000000000000000 P b01057b6c5ff4fc4b07253801881fe13 [term 2 FOLLOWER]: Advancing to term 3
I20260130 02:59:13.472731 22832 raft_consensus.cc:515] T 00000000000000000000000000000000 P b01057b6c5ff4fc4b07253801881fe13 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22dedcf69e00438999d3a5fedde57324" member_type: VOTER last_known_addr { host: "127.18.100.126" port: 42077 } } peers { permanent_uuid: "b01057b6c5ff4fc4b07253801881fe13" member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39157 } } peers { permanent_uuid: "342be83f20a648fd849c7d23454fba08" member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34987 } }
I20260130 02:59:13.474717 22832 leader_election.cc:290] T 00000000000000000000000000000000 P b01057b6c5ff4fc4b07253801881fe13 [CANDIDATE]: Term 3 election: Requested vote from peers 22dedcf69e00438999d3a5fedde57324 (127.18.100.126:42077), 342be83f20a648fd849c7d23454fba08 (127.18.100.124:34987)
I20260130 02:59:13.476897 22769 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b01057b6c5ff4fc4b07253801881fe13" candidate_term: 3 candidate_status { last_received { term: 2 index: 16 } } ignore_live_leader: false dest_uuid: "342be83f20a648fd849c7d23454fba08"
I20260130 02:59:13.477443 22769 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 342be83f20a648fd849c7d23454fba08 [term 2 FOLLOWER]: Advancing to term 3
I20260130 02:59:13.483992 22769 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 342be83f20a648fd849c7d23454fba08 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b01057b6c5ff4fc4b07253801881fe13 in term 3.
I20260130 02:59:13.485206 21946 leader_election.cc:304] T 00000000000000000000000000000000 P b01057b6c5ff4fc4b07253801881fe13 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 342be83f20a648fd849c7d23454fba08, b01057b6c5ff4fc4b07253801881fe13; no voters:
W20260130 02:59:13.485718 21947 leader_election.cc:336] T 00000000000000000000000000000000 P b01057b6c5ff4fc4b07253801881fe13 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 22dedcf69e00438999d3a5fedde57324 (127.18.100.126:42077): Network error: Client connection negotiation failed: client connection to 127.18.100.126:42077: connect: Connection refused (error 111)
I20260130 02:59:13.485967 22832 raft_consensus.cc:2804] T 00000000000000000000000000000000 P b01057b6c5ff4fc4b07253801881fe13 [term 3 FOLLOWER]: Leader election won for term 3
I20260130 02:59:13.487754 22832 raft_consensus.cc:697] T 00000000000000000000000000000000 P b01057b6c5ff4fc4b07253801881fe13 [term 3 LEADER]: Becoming Leader. State: Replica: b01057b6c5ff4fc4b07253801881fe13, State: Running, Role: LEADER
I20260130 02:59:13.488608 22832 consensus_queue.cc:237] T 00000000000000000000000000000000 P b01057b6c5ff4fc4b07253801881fe13 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 2.16, Last appended by leader: 16, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22dedcf69e00438999d3a5fedde57324" member_type: VOTER last_known_addr { host: "127.18.100.126" port: 42077 } } peers { permanent_uuid: "b01057b6c5ff4fc4b07253801881fe13" member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39157 } } peers { permanent_uuid: "342be83f20a648fd849c7d23454fba08" member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34987 } }
I20260130 02:59:13.492965 22836 sys_catalog.cc:455] T 00000000000000000000000000000000 P b01057b6c5ff4fc4b07253801881fe13 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b01057b6c5ff4fc4b07253801881fe13. Latest consensus state: current_term: 3 leader_uuid: "b01057b6c5ff4fc4b07253801881fe13" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22dedcf69e00438999d3a5fedde57324" member_type: VOTER last_known_addr { host: "127.18.100.126" port: 42077 } } peers { permanent_uuid: "b01057b6c5ff4fc4b07253801881fe13" member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39157 } } peers { permanent_uuid: "342be83f20a648fd849c7d23454fba08" member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34987 } } }
I20260130 02:59:13.493734 22836 sys_catalog.cc:458] T 00000000000000000000000000000000 P b01057b6c5ff4fc4b07253801881fe13 [sys.catalog]: This master's current role is: LEADER
I20260130 02:59:13.496105 22838 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260130 02:59:13.502508 22838 catalog_manager.cc:679] Loaded metadata for table default.table_e3b3683b36e144c2b33dd88b292bcb23 [id=318c872fbfef4f93b4c280fd078cabd9]
I20260130 02:59:13.503777 22838 catalog_manager.cc:679] Loaded metadata for table default.table_d6f2cd9cd3b54168b14f1cacabbc74f0 [id=db113a9d4bf545faa72c7c94427d6dba]
I20260130 02:59:13.504901 22838 catalog_manager.cc:679] Loaded metadata for table default.table_62e153a4e7844d97863f32a6e87da70b [id=dee95603eab3439b81355a006bc9f376]
I20260130 02:59:13.510355 22838 tablet_loader.cc:96] loaded metadata for tablet 7e4c634c4cbf4dfbb7caf795e9f9e5c2 (table default.table_e3b3683b36e144c2b33dd88b292bcb23 [id=318c872fbfef4f93b4c280fd078cabd9])
I20260130 02:59:13.511791 22838 tablet_loader.cc:96] loaded metadata for tablet 9ee21c80f1ee4d16b777130f87124d56 (table default.table_d6f2cd9cd3b54168b14f1cacabbc74f0 [id=db113a9d4bf545faa72c7c94427d6dba])
I20260130 02:59:13.513140 22838 tablet_loader.cc:96] loaded metadata for tablet f41b5d6226db4f4287f8a4cdea9a0bdc (table default.table_62e153a4e7844d97863f32a6e87da70b [id=dee95603eab3439b81355a006bc9f376])
I20260130 02:59:13.514070 22838 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260130 02:59:13.517190 22838 catalog_manager.cc:1269] Loaded cluster ID: 40eb42871f464e95bd641ceacb567979
I20260130 02:59:13.517385 22838 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260130 02:59:13.521075 22838 catalog_manager.cc:1514] Loading token signing keys...
I20260130 02:59:13.524093 22838 catalog_manager.cc:6038] T 00000000000000000000000000000000 P b01057b6c5ff4fc4b07253801881fe13: Loaded TSK: 0
I20260130 02:59:13.525300 22838 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260130 02:59:13.528358 22838 catalog_manager.cc:1537] Loading latest processed Hive Metastore notification log event ID...
I20260130 02:59:13.531335 22838 catalog_manager.cc:5876] Last processed Hive Metastore notification event ID: -1
I20260130 02:59:13.551483 22769 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 342be83f20a648fd849c7d23454fba08 [term 3 FOLLOWER]: Refusing update from remote peer b01057b6c5ff4fc4b07253801881fe13: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 18. (index mismatch)
I20260130 02:59:13.553289 22832 consensus_queue.cc:1048] T 00000000000000000000000000000000 P b01057b6c5ff4fc4b07253801881fe13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "342be83f20a648fd849c7d23454fba08" member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34987 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 4, Time since last communication: 0.000s
W20260130 02:59:13.554344 21947 consensus_peers.cc:597] T 00000000000000000000000000000000 P b01057b6c5ff4fc4b07253801881fe13 -> Peer 22dedcf69e00438999d3a5fedde57324 (127.18.100.126:42077): Couldn't send request to peer 22dedcf69e00438999d3a5fedde57324. Status: Network error: Client connection negotiation failed: client connection to 127.18.100.126:42077: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260130 02:59:13.557500 21960 master_service.cc:954] ReplaceTablet: received request to replace tablet 9ee21c80f1ee4d16b777130f87124d56 from {username='slave'} at 127.0.0.1:57858
I20260130 02:59:13.558079 21961 master_service.cc:954] ReplaceTablet: received request to replace tablet f41b5d6226db4f4287f8a4cdea9a0bdc from {username='slave'} at 127.0.0.1:57858
I20260130 02:59:13.558370 21959 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57858:
name: "default.Table_53c1c4a7b44f4bb095c3167a66cf3315"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260130 02:59:13.559333 21958 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57858:
table { table_name: "default.Table_d6f2cd9cd3b54168b14f1cacabbc74f0" } modify_external_catalogs: true
I20260130 02:59:13.560142 21958 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57858:
table { table_name: "default.Table_d6f2cd9cd3b54168b14f1cacabbc74f0" } modify_external_catalogs: true
I20260130 02:59:13.563084 21953 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57858:
table { table_name: "default.Table_e3b3683b36e144c2b33dd88b292bcb23" } new_table_name: "default.Table_1b5ff18b6b2146a98161fe3210afe549" modify_external_catalogs: true
I20260130 02:59:13.564585 21954 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57858:
table { table_name: "default.Table_62e153a4e7844d97863f32a6e87da70b" } new_table_name: "default.Table_9fdc4fc1d8a442389a5c9201bcd4ab84" modify_external_catalogs: true
F20260130 02:59:13.567909 22121 hms_notification_log_listener.cc:287] non-monotonous event IDs from HMS: current 1, previous 1; dumping first 2 out of 4 received events: NotificationEvent(eventId=1, eventTime=1769741949, eventType=CREATE_TABLE, dbName=default, tableName=table_de96622ce9794752adfc14fb9ceb4559, message={"server":"","servicePrincipal":"","db":"default","table":"table_de96622ce9794752adfc14fb9ceb4559","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_de96622ce9794752adfc14fb9ceb4559\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1769741949},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1769741859011789-18833-0/minicluster-data/warehouse/table_de96622ce9794752adfc14fb9ceb4559\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_de96622ce9794752adfc14fb9ceb4559\",\"kudu.table_id\":\"e5276a565ee84745a424387f988a65e9\",\"transient_lastDdlTime\":\"1769741949\",\"comment\":\"\",\"kudu.master_addresses\":\"127.18.100.126:42077,127.18.100.125:39157,127.18.100.124:34987\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"40eb42871f464e95bd641ceacb567979\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1769741949,"files":[]}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=1, eventTime=1769741949, eventType=CREATE_TABLE, dbName=default, tableName=table_62e153a4e7844d97863f32a6e87da70b, message={"server":"","servicePrincipal":"","db":"default","table":"table_62e153a4e7844d97863f32a6e87da70b","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_62e153a4e7844d97863f32a6e87da70b\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1769741949},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1769741859011789-18833-0/minicluster-data/warehouse/table_62e153a4e7844d97863f32a6e87da70b\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_62e153a4e7844d97863f32a6e87da70b\",\"kudu.table_id\":\"dee95603eab3439b81355a006bc9f376\",\"transient_lastDdlTime\":\"1769741949\",\"comment\":\"\",\"kudu.master_addresses\":\"127.18.100.126:42077,127.18.100.125:39157,127.18.100.124:34987\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"40eb42871f464e95bd641ceacb567979\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1769741949,"files":[]}, messageFormat=json-0.2, catName=hive);
*** Check failure stack trace: ***
I20260130 02:59:13.575147 21955 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57858:
name: "default.Table_2dde477c96844e51bc37ab8d6bd13248"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260130 02:59:13.577842 21956 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57858:
name: "default.Table_91ef08c82eb0446fbb6164d207168099"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260130 02:59:13.579785 21957 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57858:
name: "default.Table_66d14bc2eefe4c4cabbfe8ea4ca2a287"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
@ 0x7ffaf62fe950 google::LogMessage::Flush() at ??:0
@ 0x7ffaf62fe6bf google::LogMessage::~LogMessage() at ??:0
I20260130 02:59:13.610312 22843 sys_catalog.cc:455] T 00000000000000000000000000000000 P 342be83f20a648fd849c7d23454fba08 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "b01057b6c5ff4fc4b07253801881fe13" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22dedcf69e00438999d3a5fedde57324" member_type: VOTER last_known_addr { host: "127.18.100.126" port: 42077 } } peers { permanent_uuid: "b01057b6c5ff4fc4b07253801881fe13" member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39157 } } peers { permanent_uuid: "342be83f20a648fd849c7d23454fba08" member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34987 } } }
I20260130 02:59:13.611126 22843 sys_catalog.cc:458] T 00000000000000000000000000000000 P 342be83f20a648fd849c7d23454fba08 [sys.catalog]: This master's current role is: FOLLOWER
I20260130 02:59:13.612142 22803 sys_catalog.cc:455] T 00000000000000000000000000000000 P 342be83f20a648fd849c7d23454fba08 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b01057b6c5ff4fc4b07253801881fe13. Latest consensus state: current_term: 3 leader_uuid: "b01057b6c5ff4fc4b07253801881fe13" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22dedcf69e00438999d3a5fedde57324" member_type: VOTER last_known_addr { host: "127.18.100.126" port: 42077 } } peers { permanent_uuid: "b01057b6c5ff4fc4b07253801881fe13" member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39157 } } peers { permanent_uuid: "342be83f20a648fd849c7d23454fba08" member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34987 } } }
I20260130 02:59:13.612805 22803 sys_catalog.cc:458] T 00000000000000000000000000000000 P 342be83f20a648fd849c7d23454fba08 [sys.catalog]: This master's current role is: FOLLOWER
I20260130 02:59:13.619910 22803 sys_catalog.cc:455] T 00000000000000000000000000000000 P 342be83f20a648fd849c7d23454fba08 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "b01057b6c5ff4fc4b07253801881fe13" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22dedcf69e00438999d3a5fedde57324" member_type: VOTER last_known_addr { host: "127.18.100.126" port: 42077 } } peers { permanent_uuid: "b01057b6c5ff4fc4b07253801881fe13" member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39157 } } peers { permanent_uuid: "342be83f20a648fd849c7d23454fba08" member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34987 } } }
I20260130 02:59:13.620595 22803 sys_catalog.cc:458] T 00000000000000000000000000000000 P 342be83f20a648fd849c7d23454fba08 [sys.catalog]: This master's current role is: FOLLOWER
I20260130 02:59:13.623808 22836 sys_catalog.cc:455] T 00000000000000000000000000000000 P b01057b6c5ff4fc4b07253801881fe13 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "b01057b6c5ff4fc4b07253801881fe13" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22dedcf69e00438999d3a5fedde57324" member_type: VOTER last_known_addr { host: "127.18.100.126" port: 42077 } } peers { permanent_uuid: "b01057b6c5ff4fc4b07253801881fe13" member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39157 } } peers { permanent_uuid: "342be83f20a648fd849c7d23454fba08" member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34987 } } }
I20260130 02:59:13.624511 22836 sys_catalog.cc:458] T 00000000000000000000000000000000 P b01057b6c5ff4fc4b07253801881fe13 [sys.catalog]: This master's current role is: LEADER
@ 0x7ffb0139dfdc kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
I20260130 02:59:13.667230 22840 mvcc.cc:204] Tried to move back new op lower bound from 7248863041730416640 to 7248863024726941696. Current Snapshot: MvccSnapshot[applied={T|T < 7248863041730416640 or (T in {7248863041730416640})}]
I20260130 02:59:13.677417 22226 tablet_service.cc:1505] Processing CreateTablet for tablet 7e4c634c4cbf4dfbb7caf795e9f9e5c2 (DEFAULT_TABLE table=default.table_e3b3683b36e144c2b33dd88b292bcb23 [id=318c872fbfef4f93b4c280fd078cabd9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:59:13.680435 22226 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7e4c634c4cbf4dfbb7caf795e9f9e5c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:59:13.684835 21961 catalog_manager.cc:5941] T 00000000000000000000000000000000 P b01057b6c5ff4fc4b07253801881fe13: Sending DeleteTablet for 3 replicas of tablet f41b5d6226db4f4287f8a4cdea9a0bdc
I20260130 02:59:13.686517 21961 catalog_manager.cc:6724] ReplaceTablet: tablet f41b5d6226db4f4287f8a4cdea9a0bdc deleted and replaced by tablet 5cc07f1d50624e03b1f5498b12ea680b
I20260130 02:59:13.687386 22227 tablet_service.cc:1552] Processing DeleteTablet for tablet f41b5d6226db4f4287f8a4cdea9a0bdc with delete_type TABLET_DATA_DELETED (replaced by tablet 5cc07f1d50624e03b1f5498b12ea680b) from {username='slave'} at 127.0.0.1:53820
I20260130 02:59:13.697670 22368 tablet_service.cc:1505] Processing CreateTablet for tablet 7e4c634c4cbf4dfbb7caf795e9f9e5c2 (DEFAULT_TABLE table=default.table_e3b3683b36e144c2b33dd88b292bcb23 [id=318c872fbfef4f93b4c280fd078cabd9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:59:13.698797 22368 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7e4c634c4cbf4dfbb7caf795e9f9e5c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:59:13.703439 22369 tablet_service.cc:1552] Processing DeleteTablet for tablet f41b5d6226db4f4287f8a4cdea9a0bdc with delete_type TABLET_DATA_DELETED (replaced by tablet 5cc07f1d50624e03b1f5498b12ea680b) from {username='slave'} at 127.0.0.1:44088
I20260130 02:59:13.704882 22854 tablet_replica.cc:333] T f41b5d6226db4f4287f8a4cdea9a0bdc P dd32a35972b347b89ceaaad2455aef5a: stopping tablet replica
I20260130 02:59:13.706106 22854 raft_consensus.cc:2243] T f41b5d6226db4f4287f8a4cdea9a0bdc P dd32a35972b347b89ceaaad2455aef5a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260130 02:59:13.708755 22854 raft_consensus.cc:2272] T f41b5d6226db4f4287f8a4cdea9a0bdc P dd32a35972b347b89ceaaad2455aef5a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260130 02:59:13.709972 22853 tablet_replica.cc:333] T f41b5d6226db4f4287f8a4cdea9a0bdc P 111215cc989f4203a6e1517f2291d66f: stopping tablet replica
I20260130 02:59:13.710268 21960 catalog_manager.cc:5941] T 00000000000000000000000000000000 P b01057b6c5ff4fc4b07253801881fe13: Sending DeleteTablet for 3 replicas of tablet 9ee21c80f1ee4d16b777130f87124d56
I20260130 02:59:13.711835 21960 catalog_manager.cc:6724] ReplaceTablet: tablet 9ee21c80f1ee4d16b777130f87124d56 deleted and replaced by tablet 476f8fd240bc48f1a9c47a0351ad5d27
I20260130 02:59:13.714304 22854 ts_tablet_manager.cc:1916] T f41b5d6226db4f4287f8a4cdea9a0bdc P dd32a35972b347b89ceaaad2455aef5a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260130 02:59:13.712791 22227 tablet_service.cc:1552] Processing DeleteTablet for tablet 9ee21c80f1ee4d16b777130f87124d56 with delete_type TABLET_DATA_DELETED (replaced by tablet 476f8fd240bc48f1a9c47a0351ad5d27) from {username='slave'} at 127.0.0.1:53820
I20260130 02:59:13.715431 22853 raft_consensus.cc:2243] T f41b5d6226db4f4287f8a4cdea9a0bdc P 111215cc989f4203a6e1517f2291d66f [term 1 LEADER]: Raft consensus shutting down.
I20260130 02:59:13.716543 22853 raft_consensus.cc:2272] T f41b5d6226db4f4287f8a4cdea9a0bdc P 111215cc989f4203a6e1517f2291d66f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260130 02:59:13.720208 22369 tablet_service.cc:1552] Processing DeleteTablet for tablet 9ee21c80f1ee4d16b777130f87124d56 with delete_type TABLET_DATA_DELETED (replaced by tablet 476f8fd240bc48f1a9c47a0351ad5d27) from {username='slave'} at 127.0.0.1:44088
I20260130 02:59:13.722452 22853 ts_tablet_manager.cc:1916] T f41b5d6226db4f4287f8a4cdea9a0bdc P 111215cc989f4203a6e1517f2291d66f: Deleting tablet data with delete state TABLET_DATA_DELETED
@ 0x7ffb0139d79c kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
@ 0x7ffb013a1812 kudu::master::HmsNotificationLogListenerTask::Init()::$_0::operator()() at ??:0
I20260130 02:59:13.746757 22369 tablet_service.cc:1505] Processing CreateTablet for tablet 5cc07f1d50624e03b1f5498b12ea680b (DEFAULT_TABLE table=default.table_62e153a4e7844d97863f32a6e87da70b [id=dee95603eab3439b81355a006bc9f376]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:59:13.747303 22367 tablet_service.cc:1505] Processing CreateTablet for tablet 476f8fd240bc48f1a9c47a0351ad5d27 (DEFAULT_TABLE table=default.table_d6f2cd9cd3b54168b14f1cacabbc74f0 [id=db113a9d4bf545faa72c7c94427d6dba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:59:13.747822 22369 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5cc07f1d50624e03b1f5498b12ea680b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:59:13.747782 22854 ts_tablet_manager.cc:1929] T f41b5d6226db4f4287f8a4cdea9a0bdc P dd32a35972b347b89ceaaad2455aef5a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260130 02:59:13.748206 22854 log.cc:1199] T f41b5d6226db4f4287f8a4cdea9a0bdc P dd32a35972b347b89ceaaad2455aef5a: Deleting WAL directory at /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1769741859011789-18833-0/minicluster-data/ts-1/wal/wals/f41b5d6226db4f4287f8a4cdea9a0bdc
I20260130 02:59:13.748472 22367 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 476f8fd240bc48f1a9c47a0351ad5d27. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:59:13.749006 22854 ts_tablet_manager.cc:1950] T f41b5d6226db4f4287f8a4cdea9a0bdc P dd32a35972b347b89ceaaad2455aef5a: Deleting consensus metadata
I20260130 02:59:13.752178 22227 tablet_service.cc:1505] Processing CreateTablet for tablet 5cc07f1d50624e03b1f5498b12ea680b (DEFAULT_TABLE table=default.table_62e153a4e7844d97863f32a6e87da70b [id=dee95603eab3439b81355a006bc9f376]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:59:13.753371 22227 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5cc07f1d50624e03b1f5498b12ea680b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:59:13.755517 22225 tablet_service.cc:1505] Processing CreateTablet for tablet 476f8fd240bc48f1a9c47a0351ad5d27 (DEFAULT_TABLE table=default.table_d6f2cd9cd3b54168b14f1cacabbc74f0 [id=db113a9d4bf545faa72c7c94427d6dba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:59:13.756534 22225 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 476f8fd240bc48f1a9c47a0351ad5d27. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:59:13.760401 22857 tablet_bootstrap.cc:492] T 7e4c634c4cbf4dfbb7caf795e9f9e5c2 P dd32a35972b347b89ceaaad2455aef5a: Bootstrap starting.
I20260130 02:59:13.764729 22514 tablet_service.cc:1505] Processing CreateTablet for tablet 7e4c634c4cbf4dfbb7caf795e9f9e5c2 (DEFAULT_TABLE table=default.table_e3b3683b36e144c2b33dd88b292bcb23 [id=318c872fbfef4f93b4c280fd078cabd9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:59:13.765813 22514 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7e4c634c4cbf4dfbb7caf795e9f9e5c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:59:13.766829 22515 tablet_service.cc:1552] Processing DeleteTablet for tablet f41b5d6226db4f4287f8a4cdea9a0bdc with delete_type TABLET_DATA_DELETED (replaced by tablet 5cc07f1d50624e03b1f5498b12ea680b) from {username='slave'} at 127.0.0.1:53092
I20260130 02:59:13.767724 22513 tablet_service.cc:1552] Processing DeleteTablet for tablet 9ee21c80f1ee4d16b777130f87124d56 with delete_type TABLET_DATA_DELETED (replaced by tablet 476f8fd240bc48f1a9c47a0351ad5d27) from {username='slave'} at 127.0.0.1:53092
I20260130 02:59:13.768939 22512 tablet_service.cc:1505] Processing CreateTablet for tablet 5cc07f1d50624e03b1f5498b12ea680b (DEFAULT_TABLE table=default.table_62e153a4e7844d97863f32a6e87da70b [id=dee95603eab3439b81355a006bc9f376]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:59:13.770550 22511 tablet_service.cc:1505] Processing CreateTablet for tablet 476f8fd240bc48f1a9c47a0351ad5d27 (DEFAULT_TABLE table=default.table_d6f2cd9cd3b54168b14f1cacabbc74f0 [id=db113a9d4bf545faa72c7c94427d6dba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:59:13.774024 22511 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 476f8fd240bc48f1a9c47a0351ad5d27. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:59:13.774868 22512 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5cc07f1d50624e03b1f5498b12ea680b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:59:13.778230 22853 ts_tablet_manager.cc:1929] T f41b5d6226db4f4287f8a4cdea9a0bdc P 111215cc989f4203a6e1517f2291d66f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260130 02:59:13.778712 22853 log.cc:1199] T f41b5d6226db4f4287f8a4cdea9a0bdc P 111215cc989f4203a6e1517f2291d66f: Deleting WAL directory at /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1769741859011789-18833-0/minicluster-data/ts-0/wal/wals/f41b5d6226db4f4287f8a4cdea9a0bdc
I20260130 02:59:13.779593 22853 ts_tablet_manager.cc:1950] T f41b5d6226db4f4287f8a4cdea9a0bdc P 111215cc989f4203a6e1517f2291d66f: Deleting consensus metadata
I20260130 02:59:13.782954 21945 catalog_manager.cc:4985] TS 111215cc989f4203a6e1517f2291d66f (127.18.100.65:39751): tablet f41b5d6226db4f4287f8a4cdea9a0bdc (table default.table_62e153a4e7844d97863f32a6e87da70b [id=dee95603eab3439b81355a006bc9f376]) successfully deleted
I20260130 02:59:13.783982 22853 tablet_replica.cc:333] T 9ee21c80f1ee4d16b777130f87124d56 P 111215cc989f4203a6e1517f2291d66f: stopping tablet replica
I20260130 02:59:13.784621 22853 raft_consensus.cc:2243] T 9ee21c80f1ee4d16b777130f87124d56 P 111215cc989f4203a6e1517f2291d66f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260130 02:59:13.785094 22853 raft_consensus.cc:2272] T 9ee21c80f1ee4d16b777130f87124d56 P 111215cc989f4203a6e1517f2291d66f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260130 02:59:13.787173 22853 ts_tablet_manager.cc:1916] T 9ee21c80f1ee4d16b777130f87124d56 P 111215cc989f4203a6e1517f2291d66f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260130 02:59:13.801082 21946 catalog_manager.cc:4985] TS dd32a35972b347b89ceaaad2455aef5a (127.18.100.66:39781): tablet f41b5d6226db4f4287f8a4cdea9a0bdc (table default.table_62e153a4e7844d97863f32a6e87da70b [id=dee95603eab3439b81355a006bc9f376]) successfully deleted
I20260130 02:59:13.801635 22854 tablet_replica.cc:333] T 9ee21c80f1ee4d16b777130f87124d56 P dd32a35972b347b89ceaaad2455aef5a: stopping tablet replica
I20260130 02:59:13.802284 22854 raft_consensus.cc:2243] T 9ee21c80f1ee4d16b777130f87124d56 P dd32a35972b347b89ceaaad2455aef5a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260130 02:59:13.804142 22854 raft_consensus.cc:2272] T 9ee21c80f1ee4d16b777130f87124d56 P dd32a35972b347b89ceaaad2455aef5a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260130 02:59:13.806066 22854 ts_tablet_manager.cc:1916] T 9ee21c80f1ee4d16b777130f87124d56 P dd32a35972b347b89ceaaad2455aef5a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260130 02:59:13.804924 22853 ts_tablet_manager.cc:1929] T 9ee21c80f1ee4d16b777130f87124d56 P 111215cc989f4203a6e1517f2291d66f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260130 02:59:13.808315 22853 log.cc:1199] T 9ee21c80f1ee4d16b777130f87124d56 P 111215cc989f4203a6e1517f2291d66f: Deleting WAL directory at /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1769741859011789-18833-0/minicluster-data/ts-0/wal/wals/9ee21c80f1ee4d16b777130f87124d56
I20260130 02:59:13.809964 22853 ts_tablet_manager.cc:1950] T 9ee21c80f1ee4d16b777130f87124d56 P 111215cc989f4203a6e1517f2291d66f: Deleting consensus metadata
I20260130 02:59:13.812850 21945 catalog_manager.cc:4985] TS 111215cc989f4203a6e1517f2291d66f (127.18.100.65:39751): tablet 9ee21c80f1ee4d16b777130f87124d56 (table default.table_d6f2cd9cd3b54168b14f1cacabbc74f0 [id=db113a9d4bf545faa72c7c94427d6dba]) successfully deleted
I20260130 02:59:13.823194 22859 tablet_replica.cc:333] T f41b5d6226db4f4287f8a4cdea9a0bdc P 789dbb4a27204ffbb68d7ae40f8f5e4a: stopping tablet replica
I20260130 02:59:13.823940 22859 raft_consensus.cc:2243] T f41b5d6226db4f4287f8a4cdea9a0bdc P 789dbb4a27204ffbb68d7ae40f8f5e4a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260130 02:59:13.824432 22859 raft_consensus.cc:2272] T f41b5d6226db4f4287f8a4cdea9a0bdc P 789dbb4a27204ffbb68d7ae40f8f5e4a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260130 02:59:13.825762 22854 ts_tablet_manager.cc:1929] T 9ee21c80f1ee4d16b777130f87124d56 P dd32a35972b347b89ceaaad2455aef5a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260130 02:59:13.826174 22854 log.cc:1199] T 9ee21c80f1ee4d16b777130f87124d56 P dd32a35972b347b89ceaaad2455aef5a: Deleting WAL directory at /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1769741859011789-18833-0/minicluster-data/ts-1/wal/wals/9ee21c80f1ee4d16b777130f87124d56
I20260130 02:59:13.826345 22859 ts_tablet_manager.cc:1916] T f41b5d6226db4f4287f8a4cdea9a0bdc P 789dbb4a27204ffbb68d7ae40f8f5e4a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260130 02:59:13.826952 22854 ts_tablet_manager.cc:1950] T 9ee21c80f1ee4d16b777130f87124d56 P dd32a35972b347b89ceaaad2455aef5a: Deleting consensus metadata
I20260130 02:59:13.829648 21946 catalog_manager.cc:4985] TS dd32a35972b347b89ceaaad2455aef5a (127.18.100.66:39781): tablet 9ee21c80f1ee4d16b777130f87124d56 (table default.table_d6f2cd9cd3b54168b14f1cacabbc74f0 [id=db113a9d4bf545faa72c7c94427d6dba]) successfully deleted
I20260130 02:59:13.831708 22860 tablet_bootstrap.cc:492] T 7e4c634c4cbf4dfbb7caf795e9f9e5c2 P 111215cc989f4203a6e1517f2291d66f: Bootstrap starting.
I20260130 02:59:13.836369 22857 tablet_bootstrap.cc:654] T 7e4c634c4cbf4dfbb7caf795e9f9e5c2 P dd32a35972b347b89ceaaad2455aef5a: Neither blocks nor log segments found. Creating new log.
I20260130 02:59:13.841109 22861 tablet_bootstrap.cc:492] T 476f8fd240bc48f1a9c47a0351ad5d27 P 789dbb4a27204ffbb68d7ae40f8f5e4a: Bootstrap starting.
I20260130 02:59:13.850797 22861 tablet_bootstrap.cc:654] T 476f8fd240bc48f1a9c47a0351ad5d27 P 789dbb4a27204ffbb68d7ae40f8f5e4a: Neither blocks nor log segments found. Creating new log.
@ 0x7ffb013a17ca _ZNSt3__18__invokeIRZN4kudu6master30HmsNotificationLogListenerTask4InitEvE3$_0JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
I20260130 02:59:13.857308 22860 tablet_bootstrap.cc:654] T 7e4c634c4cbf4dfbb7caf795e9f9e5c2 P 111215cc989f4203a6e1517f2291d66f: Neither blocks nor log segments found. Creating new log.
I20260130 02:59:13.869097 22860 tablet_bootstrap.cc:492] T 7e4c634c4cbf4dfbb7caf795e9f9e5c2 P 111215cc989f4203a6e1517f2291d66f: No bootstrap required, opened a new log
I20260130 02:59:13.869077 22857 tablet_bootstrap.cc:492] T 7e4c634c4cbf4dfbb7caf795e9f9e5c2 P dd32a35972b347b89ceaaad2455aef5a: No bootstrap required, opened a new log
I20260130 02:59:13.869496 22857 ts_tablet_manager.cc:1403] T 7e4c634c4cbf4dfbb7caf795e9f9e5c2 P dd32a35972b347b89ceaaad2455aef5a: Time spent bootstrapping tablet: real 0.109s user 0.015s sys 0.004s
I20260130 02:59:13.869496 22860 ts_tablet_manager.cc:1403] T 7e4c634c4cbf4dfbb7caf795e9f9e5c2 P 111215cc989f4203a6e1517f2291d66f: Time spent bootstrapping tablet: real 0.038s user 0.012s sys 0.000s
I20260130 02:59:13.872059 22857 raft_consensus.cc:359] T 7e4c634c4cbf4dfbb7caf795e9f9e5c2 P dd32a35972b347b89ceaaad2455aef5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111215cc989f4203a6e1517f2291d66f" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 39751 } } peers { permanent_uuid: "789dbb4a27204ffbb68d7ae40f8f5e4a" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 41825 } } peers { permanent_uuid: "dd32a35972b347b89ceaaad2455aef5a" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 39781 } }
I20260130 02:59:13.872707 22857 raft_consensus.cc:385] T 7e4c634c4cbf4dfbb7caf795e9f9e5c2 P dd32a35972b347b89ceaaad2455aef5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:59:13.877527 22857 raft_consensus.cc:740] T 7e4c634c4cbf4dfbb7caf795e9f9e5c2 P dd32a35972b347b89ceaaad2455aef5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd32a35972b347b89ceaaad2455aef5a, State: Initialized, Role: FOLLOWER
I20260130 02:59:13.878397 22857 consensus_queue.cc:260] T 7e4c634c4cbf4dfbb7caf795e9f9e5c2 P dd32a35972b347b89ceaaad2455aef5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111215cc989f4203a6e1517f2291d66f" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 39751 } } peers { permanent_uuid: "789dbb4a27204ffbb68d7ae40f8f5e4a" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 41825 } } peers { permanent_uuid: "dd32a35972b347b89ceaaad2455aef5a" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 39781 } }
I20260130 02:59:13.878644 22860 raft_consensus.cc:359] T 7e4c634c4cbf4dfbb7caf795e9f9e5c2 P 111215cc989f4203a6e1517f2291d66f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111215cc989f4203a6e1517f2291d66f" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 39751 } } peers { permanent_uuid: "789dbb4a27204ffbb68d7ae40f8f5e4a" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 41825 } } peers { permanent_uuid: "dd32a35972b347b89ceaaad2455aef5a" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 39781 } }
I20260130 02:59:13.879355 22860 raft_consensus.cc:385] T 7e4c634c4cbf4dfbb7caf795e9f9e5c2 P 111215cc989f4203a6e1517f2291d66f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:59:13.879614 22860 raft_consensus.cc:740] T 7e4c634c4cbf4dfbb7caf795e9f9e5c2 P 111215cc989f4203a6e1517f2291d66f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 111215cc989f4203a6e1517f2291d66f, State: Initialized, Role: FOLLOWER
I20260130 02:59:13.880290 22860 consensus_queue.cc:260] T 7e4c634c4cbf4dfbb7caf795e9f9e5c2 P 111215cc989f4203a6e1517f2291d66f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111215cc989f4203a6e1517f2291d66f" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 39751 } } peers { permanent_uuid: "789dbb4a27204ffbb68d7ae40f8f5e4a" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 41825 } } peers { permanent_uuid: "dd32a35972b347b89ceaaad2455aef5a" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 39781 } }
I20260130 02:59:13.884624 22861 tablet_bootstrap.cc:492] T 476f8fd240bc48f1a9c47a0351ad5d27 P 789dbb4a27204ffbb68d7ae40f8f5e4a: No bootstrap required, opened a new log
I20260130 02:59:13.885039 22861 ts_tablet_manager.cc:1403] T 476f8fd240bc48f1a9c47a0351ad5d27 P 789dbb4a27204ffbb68d7ae40f8f5e4a: Time spent bootstrapping tablet: real 0.045s user 0.005s sys 0.006s
I20260130 02:59:13.887637 22437 heartbeater.cc:499] Master 127.18.100.125:39157 was elected leader, sending a full tablet report...
I20260130 02:59:13.887583 22861 raft_consensus.cc:359] T 476f8fd240bc48f1a9c47a0351ad5d27 P 789dbb4a27204ffbb68d7ae40f8f5e4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111215cc989f4203a6e1517f2291d66f" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 39751 } } peers { permanent_uuid: "dd32a35972b347b89ceaaad2455aef5a" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 39781 } } peers { permanent_uuid: "789dbb4a27204ffbb68d7ae40f8f5e4a" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 41825 } }
I20260130 02:59:13.895958 22293 heartbeater.cc:499] Master 127.18.100.125:39157 was elected leader, sending a full tablet report...
I20260130 02:59:13.901324 22867 raft_consensus.cc:493] T 7e4c634c4cbf4dfbb7caf795e9f9e5c2 P 111215cc989f4203a6e1517f2291d66f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:59:13.901769 22867 raft_consensus.cc:515] T 7e4c634c4cbf4dfbb7caf795e9f9e5c2 P 111215cc989f4203a6e1517f2291d66f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111215cc989f4203a6e1517f2291d66f" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 39751 } } peers { permanent_uuid: "789dbb4a27204ffbb68d7ae40f8f5e4a" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 41825 } } peers { permanent_uuid: "dd32a35972b347b89ceaaad2455aef5a" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 39781 } }
I20260130 02:59:13.902667 22857 ts_tablet_manager.cc:1434] T 7e4c634c4cbf4dfbb7caf795e9f9e5c2 P dd32a35972b347b89ceaaad2455aef5a: Time spent starting tablet: real 0.033s user 0.009s sys 0.000s
I20260130 02:59:13.903673 22857 tablet_bootstrap.cc:492] T 5cc07f1d50624e03b1f5498b12ea680b P dd32a35972b347b89ceaaad2455aef5a: Bootstrap starting.
I20260130 02:59:13.906493 22860 ts_tablet_manager.cc:1434] T 7e4c634c4cbf4dfbb7caf795e9f9e5c2 P 111215cc989f4203a6e1517f2291d66f: Time spent starting tablet: real 0.036s user 0.010s sys 0.004s
I20260130 02:59:13.907790 22860 tablet_bootstrap.cc:492] T 5cc07f1d50624e03b1f5498b12ea680b P 111215cc989f4203a6e1517f2291d66f: Bootstrap starting.
I20260130 02:59:13.914974 22859 ts_tablet_manager.cc:1929] T f41b5d6226db4f4287f8a4cdea9a0bdc P 789dbb4a27204ffbb68d7ae40f8f5e4a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260130 02:59:13.915402 22859 log.cc:1199] T f41b5d6226db4f4287f8a4cdea9a0bdc P 789dbb4a27204ffbb68d7ae40f8f5e4a: Deleting WAL directory at /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1769741859011789-18833-0/minicluster-data/ts-2/wal/wals/f41b5d6226db4f4287f8a4cdea9a0bdc
I20260130 02:59:13.915542 22535 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7e4c634c4cbf4dfbb7caf795e9f9e5c2" candidate_uuid: "111215cc989f4203a6e1517f2291d66f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "789dbb4a27204ffbb68d7ae40f8f5e4a" is_pre_election: true
I20260130 02:59:13.916249 22859 ts_tablet_manager.cc:1950] T f41b5d6226db4f4287f8a4cdea9a0bdc P 789dbb4a27204ffbb68d7ae40f8f5e4a: Deleting consensus metadata
W20260130 02:59:13.917055 22179 leader_election.cc:343] T 7e4c634c4cbf4dfbb7caf795e9f9e5c2 P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 789dbb4a27204ffbb68d7ae40f8f5e4a (127.18.100.67:41825): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:59:13.918000 22867 leader_election.cc:290] T 7e4c634c4cbf4dfbb7caf795e9f9e5c2 P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 789dbb4a27204ffbb68d7ae40f8f5e4a (127.18.100.67:41825), dd32a35972b347b89ceaaad2455aef5a (127.18.100.66:39781)
I20260130 02:59:13.918723 22859 tablet_replica.cc:333] T 9ee21c80f1ee4d16b777130f87124d56 P 789dbb4a27204ffbb68d7ae40f8f5e4a: stopping tablet replica
I20260130 02:59:13.919381 22859 raft_consensus.cc:2243] T 9ee21c80f1ee4d16b777130f87124d56 P 789dbb4a27204ffbb68d7ae40f8f5e4a [term 1 LEADER]: Raft consensus shutting down.
I20260130 02:59:13.920389 22859 raft_consensus.cc:2272] T 9ee21c80f1ee4d16b777130f87124d56 P 789dbb4a27204ffbb68d7ae40f8f5e4a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260130 02:59:13.920748 21945 catalog_manager.cc:4985] TS 789dbb4a27204ffbb68d7ae40f8f5e4a (127.18.100.67:41825): tablet f41b5d6226db4f4287f8a4cdea9a0bdc (table default.table_62e153a4e7844d97863f32a6e87da70b [id=dee95603eab3439b81355a006bc9f376]) successfully deleted
I20260130 02:59:13.922246 22859 ts_tablet_manager.cc:1916] T 9ee21c80f1ee4d16b777130f87124d56 P 789dbb4a27204ffbb68d7ae40f8f5e4a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260130 02:59:13.924423 22391 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7e4c634c4cbf4dfbb7caf795e9f9e5c2" candidate_uuid: "111215cc989f4203a6e1517f2291d66f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd32a35972b347b89ceaaad2455aef5a" is_pre_election: true
I20260130 02:59:13.924998 22391 raft_consensus.cc:2468] T 7e4c634c4cbf4dfbb7caf795e9f9e5c2 P dd32a35972b347b89ceaaad2455aef5a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 111215cc989f4203a6e1517f2291d66f in term 0.
I20260130 02:59:13.888211 22861 raft_consensus.cc:385] T 476f8fd240bc48f1a9c47a0351ad5d27 P 789dbb4a27204ffbb68d7ae40f8f5e4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:59:13.927582 22861 raft_consensus.cc:740] T 476f8fd240bc48f1a9c47a0351ad5d27 P 789dbb4a27204ffbb68d7ae40f8f5e4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 789dbb4a27204ffbb68d7ae40f8f5e4a, State: Initialized, Role: FOLLOWER
I20260130 02:59:13.928380 22861 consensus_queue.cc:260] T 476f8fd240bc48f1a9c47a0351ad5d27 P 789dbb4a27204ffbb68d7ae40f8f5e4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111215cc989f4203a6e1517f2291d66f" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 39751 } } peers { permanent_uuid: "dd32a35972b347b89ceaaad2455aef5a" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 39781 } } peers { permanent_uuid: "789dbb4a27204ffbb68d7ae40f8f5e4a" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 41825 } }
I20260130 02:59:13.932015 22580 heartbeater.cc:499] Master 127.18.100.125:39157 was elected leader, sending a full tablet report...
@ 0x7ffb013a175a std::__1::__invoke_void_return_wrapper<>::__call<>() at ??:0
I20260130 02:59:13.930455 22861 ts_tablet_manager.cc:1434] T 476f8fd240bc48f1a9c47a0351ad5d27 P 789dbb4a27204ffbb68d7ae40f8f5e4a: Time spent starting tablet: real 0.045s user 0.005s sys 0.000s
I20260130 02:59:13.938300 22857 tablet_bootstrap.cc:654] T 5cc07f1d50624e03b1f5498b12ea680b P dd32a35972b347b89ceaaad2455aef5a: Neither blocks nor log segments found. Creating new log.
I20260130 02:59:13.943747 22861 tablet_bootstrap.cc:492] T 5cc07f1d50624e03b1f5498b12ea680b P 789dbb4a27204ffbb68d7ae40f8f5e4a: Bootstrap starting.
I20260130 02:59:13.949795 22861 tablet_bootstrap.cc:654] T 5cc07f1d50624e03b1f5498b12ea680b P 789dbb4a27204ffbb68d7ae40f8f5e4a: Neither blocks nor log segments found. Creating new log.
I20260130 02:59:13.957494 21952 master_service.cc:954] ReplaceTablet: received request to replace tablet 5cc07f1d50624e03b1f5498b12ea680b from {username='slave'} at 127.0.0.1:57858
I20260130 02:59:13.958453 21960 master_service.cc:954] ReplaceTablet: received request to replace tablet 476f8fd240bc48f1a9c47a0351ad5d27 from {username='slave'} at 127.0.0.1:57858
I20260130 02:59:13.963438 22857 tablet_bootstrap.cc:492] T 5cc07f1d50624e03b1f5498b12ea680b P dd32a35972b347b89ceaaad2455aef5a: No bootstrap required, opened a new log
I20260130 02:59:13.963838 22857 ts_tablet_manager.cc:1403] T 5cc07f1d50624e03b1f5498b12ea680b P dd32a35972b347b89ceaaad2455aef5a: Time spent bootstrapping tablet: real 0.060s user 0.010s sys 0.000s
I20260130 02:59:13.962070 22180 leader_election.cc:304] T 7e4c634c4cbf4dfbb7caf795e9f9e5c2 P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 111215cc989f4203a6e1517f2291d66f, dd32a35972b347b89ceaaad2455aef5a; no voters: 789dbb4a27204ffbb68d7ae40f8f5e4a
I20260130 02:59:13.967888 22867 raft_consensus.cc:2804] T 7e4c634c4cbf4dfbb7caf795e9f9e5c2 P 111215cc989f4203a6e1517f2291d66f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260130 02:59:13.968214 22867 raft_consensus.cc:493] T 7e4c634c4cbf4dfbb7caf795e9f9e5c2 P 111215cc989f4203a6e1517f2291d66f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260130 02:59:13.968513 22867 raft_consensus.cc:3060] T 7e4c634c4cbf4dfbb7caf795e9f9e5c2 P 111215cc989f4203a6e1517f2291d66f [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:59:13.969978 22860 tablet_bootstrap.cc:654] T 5cc07f1d50624e03b1f5498b12ea680b P 111215cc989f4203a6e1517f2291d66f: Neither blocks nor log segments found. Creating new log.
I20260130 02:59:13.966241 22857 raft_consensus.cc:359] T 5cc07f1d50624e03b1f5498b12ea680b P dd32a35972b347b89ceaaad2455aef5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111215cc989f4203a6e1517f2291d66f" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 39751 } } peers { permanent_uuid: "dd32a35972b347b89ceaaad2455aef5a" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 39781 } } peers { permanent_uuid: "789dbb4a27204ffbb68d7ae40f8f5e4a" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 41825 } }
I20260130 02:59:13.975648 22857 raft_consensus.cc:385] T 5cc07f1d50624e03b1f5498b12ea680b P dd32a35972b347b89ceaaad2455aef5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:59:13.975929 22857 raft_consensus.cc:740] T 5cc07f1d50624e03b1f5498b12ea680b P dd32a35972b347b89ceaaad2455aef5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd32a35972b347b89ceaaad2455aef5a, State: Initialized, Role: FOLLOWER
I20260130 02:59:13.976673 22857 consensus_queue.cc:260] T 5cc07f1d50624e03b1f5498b12ea680b P dd32a35972b347b89ceaaad2455aef5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111215cc989f4203a6e1517f2291d66f" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 39751 } } peers { permanent_uuid: "dd32a35972b347b89ceaaad2455aef5a" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 39781 } } peers { permanent_uuid: "789dbb4a27204ffbb68d7ae40f8f5e4a" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 41825 } }
I20260130 02:59:13.981475 21960 catalog_manager.cc:5941] T 00000000000000000000000000000000 P b01057b6c5ff4fc4b07253801881fe13: Sending DeleteTablet for 3 replicas of tablet 476f8fd240bc48f1a9c47a0351ad5d27
I20260130 02:59:13.983034 21960 catalog_manager.cc:6724] ReplaceTablet: tablet 476f8fd240bc48f1a9c47a0351ad5d27 deleted and replaced by tablet 670a199ebd03458cb3dae713581506f7
I20260130 02:59:13.983397 22226 tablet_service.cc:1552] Processing DeleteTablet for tablet 476f8fd240bc48f1a9c47a0351ad5d27 with delete_type TABLET_DATA_DELETED (replaced by tablet 670a199ebd03458cb3dae713581506f7) from {username='slave'} at 127.0.0.1:53820
I20260130 02:59:13.983989 22511 tablet_service.cc:1552] Processing DeleteTablet for tablet 476f8fd240bc48f1a9c47a0351ad5d27 with delete_type TABLET_DATA_DELETED (replaced by tablet 670a199ebd03458cb3dae713581506f7) from {username='slave'} at 127.0.0.1:53092
W20260130 02:59:13.985103 21945 catalog_manager.cc:4965] TS 111215cc989f4203a6e1517f2291d66f (127.18.100.65:39751): delete failed for tablet 476f8fd240bc48f1a9c47a0351ad5d27 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 476f8fd240bc48f1a9c47a0351ad5d27 already in progress: creating tablet
I20260130 02:59:13.985860 22368 tablet_service.cc:1552] Processing DeleteTablet for tablet 476f8fd240bc48f1a9c47a0351ad5d27 with delete_type TABLET_DATA_DELETED (replaced by tablet 670a199ebd03458cb3dae713581506f7) from {username='slave'} at 127.0.0.1:44088
W20260130 02:59:13.987588 21946 catalog_manager.cc:4965] TS dd32a35972b347b89ceaaad2455aef5a (127.18.100.66:39781): delete failed for tablet 476f8fd240bc48f1a9c47a0351ad5d27 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 476f8fd240bc48f1a9c47a0351ad5d27 already in progress: creating tablet
I20260130 02:59:13.988039 22857 ts_tablet_manager.cc:1434] T 5cc07f1d50624e03b1f5498b12ea680b P dd32a35972b347b89ceaaad2455aef5a: Time spent starting tablet: real 0.024s user 0.006s sys 0.000s
I20260130 02:59:13.994827 22867 raft_consensus.cc:515] T 7e4c634c4cbf4dfbb7caf795e9f9e5c2 P 111215cc989f4203a6e1517f2291d66f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111215cc989f4203a6e1517f2291d66f" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 39751 } } peers { permanent_uuid: "789dbb4a27204ffbb68d7ae40f8f5e4a" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 41825 } } peers { permanent_uuid: "dd32a35972b347b89ceaaad2455aef5a" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 39781 } }
I20260130 02:59:13.997617 22535 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7e4c634c4cbf4dfbb7caf795e9f9e5c2" candidate_uuid: "111215cc989f4203a6e1517f2291d66f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "789dbb4a27204ffbb68d7ae40f8f5e4a"
W20260130 02:59:13.998926 22179 leader_election.cc:343] T 7e4c634c4cbf4dfbb7caf795e9f9e5c2 P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 789dbb4a27204ffbb68d7ae40f8f5e4a (127.18.100.67:41825): Illegal state: must be running to vote when last-logged opid is not known
I20260130 02:59:14.004033 22391 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7e4c634c4cbf4dfbb7caf795e9f9e5c2" candidate_uuid: "111215cc989f4203a6e1517f2291d66f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd32a35972b347b89ceaaad2455aef5a"
I20260130 02:59:14.004570 22391 raft_consensus.cc:3060] T 7e4c634c4cbf4dfbb7caf795e9f9e5c2 P dd32a35972b347b89ceaaad2455aef5a [term 0 FOLLOWER]: Advancing to term 1
@ 0x7ffb013a1722 std::__1::__function::__alloc_func<>::operator()() at ??:0
I20260130 02:59:14.011483 22391 raft_consensus.cc:2468] T 7e4c634c4cbf4dfbb7caf795e9f9e5c2 P dd32a35972b347b89ceaaad2455aef5a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 111215cc989f4203a6e1517f2291d66f in term 1.
I20260130 02:59:14.012626 22180 leader_election.cc:304] T 7e4c634c4cbf4dfbb7caf795e9f9e5c2 P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 111215cc989f4203a6e1517f2291d66f, dd32a35972b347b89ceaaad2455aef5a; no voters: 789dbb4a27204ffbb68d7ae40f8f5e4a
I20260130 02:59:14.012912 21952 catalog_manager.cc:5941] T 00000000000000000000000000000000 P b01057b6c5ff4fc4b07253801881fe13: Sending DeleteTablet for 3 replicas of tablet 5cc07f1d50624e03b1f5498b12ea680b
I20260130 02:59:14.013552 22868 raft_consensus.cc:2804] T 7e4c634c4cbf4dfbb7caf795e9f9e5c2 P 111215cc989f4203a6e1517f2291d66f [term 1 FOLLOWER]: Leader election won for term 1
I20260130 02:59:14.015708 22226 tablet_service.cc:1552] Processing DeleteTablet for tablet 5cc07f1d50624e03b1f5498b12ea680b with delete_type TABLET_DATA_DELETED (replaced by tablet 010faaeaf5e049a7893e4bf97b3f73e8) from {username='slave'} at 127.0.0.1:53820
W20260130 02:59:14.017465 21945 catalog_manager.cc:4965] TS 111215cc989f4203a6e1517f2291d66f (127.18.100.65:39751): delete failed for tablet 5cc07f1d50624e03b1f5498b12ea680b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5cc07f1d50624e03b1f5498b12ea680b already in progress: creating tablet
I20260130 02:59:14.020587 22857 tablet_bootstrap.cc:492] T 476f8fd240bc48f1a9c47a0351ad5d27 P dd32a35972b347b89ceaaad2455aef5a: Bootstrap starting.
I20260130 02:59:14.022519 22368 tablet_service.cc:1552] Processing DeleteTablet for tablet 5cc07f1d50624e03b1f5498b12ea680b with delete_type TABLET_DATA_DELETED (replaced by tablet 010faaeaf5e049a7893e4bf97b3f73e8) from {username='slave'} at 127.0.0.1:44088
I20260130 02:59:14.022621 22511 tablet_service.cc:1552] Processing DeleteTablet for tablet 5cc07f1d50624e03b1f5498b12ea680b with delete_type TABLET_DATA_DELETED (replaced by tablet 010faaeaf5e049a7893e4bf97b3f73e8) from {username='slave'} at 127.0.0.1:53092
I20260130 02:59:14.023113 22867 leader_election.cc:290] T 7e4c634c4cbf4dfbb7caf795e9f9e5c2 P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 election: Requested vote from peers 789dbb4a27204ffbb68d7ae40f8f5e4a (127.18.100.67:41825), dd32a35972b347b89ceaaad2455aef5a (127.18.100.66:39781)
I20260130 02:59:14.023291 22854 tablet_replica.cc:333] T 5cc07f1d50624e03b1f5498b12ea680b P dd32a35972b347b89ceaaad2455aef5a: stopping tablet replica
I20260130 02:59:14.024005 22854 raft_consensus.cc:2243] T 5cc07f1d50624e03b1f5498b12ea680b P dd32a35972b347b89ceaaad2455aef5a [term 0 FOLLOWER]: Raft consensus shutting down.
I20260130 02:59:14.024467 22854 raft_consensus.cc:2272] T 5cc07f1d50624e03b1f5498b12ea680b P dd32a35972b347b89ceaaad2455aef5a [term 0 FOLLOWER]: Raft consensus is shut down!
I20260130 02:59:14.025714 21952 catalog_manager.cc:6724] ReplaceTablet: tablet 5cc07f1d50624e03b1f5498b12ea680b deleted and replaced by tablet 010faaeaf5e049a7893e4bf97b3f73e8
I20260130 02:59:14.031632 22868 raft_consensus.cc:697] T 7e4c634c4cbf4dfbb7caf795e9f9e5c2 P 111215cc989f4203a6e1517f2291d66f [term 1 LEADER]: Becoming Leader. State: Replica: 111215cc989f4203a6e1517f2291d66f, State: Running, Role: LEADER
I20260130 02:59:14.032588 22868 consensus_queue.cc:237] T 7e4c634c4cbf4dfbb7caf795e9f9e5c2 P 111215cc989f4203a6e1517f2291d66f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111215cc989f4203a6e1517f2291d66f" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 39751 } } peers { permanent_uuid: "789dbb4a27204ffbb68d7ae40f8f5e4a" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 41825 } } peers { permanent_uuid: "dd32a35972b347b89ceaaad2455aef5a" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 39781 } }
I20260130 02:59:14.038627 22861 tablet_bootstrap.cc:492] T 5cc07f1d50624e03b1f5498b12ea680b P 789dbb4a27204ffbb68d7ae40f8f5e4a: No bootstrap required, opened a new log
I20260130 02:59:14.039070 22861 ts_tablet_manager.cc:1403] T 5cc07f1d50624e03b1f5498b12ea680b P 789dbb4a27204ffbb68d7ae40f8f5e4a: Time spent bootstrapping tablet: real 0.096s user 0.012s sys 0.000s
I20260130 02:59:14.041792 22861 raft_consensus.cc:359] T 5cc07f1d50624e03b1f5498b12ea680b P 789dbb4a27204ffbb68d7ae40f8f5e4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111215cc989f4203a6e1517f2291d66f" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 39751 } } peers { permanent_uuid: "dd32a35972b347b89ceaaad2455aef5a" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 39781 } } peers { permanent_uuid: "789dbb4a27204ffbb68d7ae40f8f5e4a" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 41825 } }
I20260130 02:59:14.044387 22859 ts_tablet_manager.cc:1929] T 9ee21c80f1ee4d16b777130f87124d56 P 789dbb4a27204ffbb68d7ae40f8f5e4a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260130 02:59:14.049098 22857 tablet_bootstrap.cc:654] T 476f8fd240bc48f1a9c47a0351ad5d27 P dd32a35972b347b89ceaaad2455aef5a: Neither blocks nor log segments found. Creating new log.
I20260130 02:59:14.051477 22859 log.cc:1199] T 9ee21c80f1ee4d16b777130f87124d56 P 789dbb4a27204ffbb68d7ae40f8f5e4a: Deleting WAL directory at /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1769741859011789-18833-0/minicluster-data/ts-2/wal/wals/9ee21c80f1ee4d16b777130f87124d56
I20260130 02:59:14.052248 22859 ts_tablet_manager.cc:1950] T 9ee21c80f1ee4d16b777130f87124d56 P 789dbb4a27204ffbb68d7ae40f8f5e4a: Deleting consensus metadata
I20260130 02:59:14.057093 22859 tablet_replica.cc:333] T 476f8fd240bc48f1a9c47a0351ad5d27 P 789dbb4a27204ffbb68d7ae40f8f5e4a: stopping tablet replica
I20260130 02:59:14.057336 21945 catalog_manager.cc:4985] TS 789dbb4a27204ffbb68d7ae40f8f5e4a (127.18.100.67:41825): tablet 9ee21c80f1ee4d16b777130f87124d56 (table default.table_d6f2cd9cd3b54168b14f1cacabbc74f0 [id=db113a9d4bf545faa72c7c94427d6dba]) successfully deleted
I20260130 02:59:14.057729 22859 raft_consensus.cc:2243] T 476f8fd240bc48f1a9c47a0351ad5d27 P 789dbb4a27204ffbb68d7ae40f8f5e4a [term 0 FOLLOWER]: Raft consensus shutting down.
I20260130 02:59:14.058176 22859 raft_consensus.cc:2272] T 476f8fd240bc48f1a9c47a0351ad5d27 P 789dbb4a27204ffbb68d7ae40f8f5e4a [term 0 FOLLOWER]: Raft consensus is shut down!
I20260130 02:59:14.042483 22861 raft_consensus.cc:385] T 5cc07f1d50624e03b1f5498b12ea680b P 789dbb4a27204ffbb68d7ae40f8f5e4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:59:14.059147 22861 raft_consensus.cc:740] T 5cc07f1d50624e03b1f5498b12ea680b P 789dbb4a27204ffbb68d7ae40f8f5e4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 789dbb4a27204ffbb68d7ae40f8f5e4a, State: Initialized, Role: FOLLOWER
I20260130 02:59:14.061439 22861 consensus_queue.cc:260] T 5cc07f1d50624e03b1f5498b12ea680b P 789dbb4a27204ffbb68d7ae40f8f5e4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111215cc989f4203a6e1517f2291d66f" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 39751 } } peers { permanent_uuid: "dd32a35972b347b89ceaaad2455aef5a" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 39781 } } peers { permanent_uuid: "789dbb4a27204ffbb68d7ae40f8f5e4a" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 41825 } }
W20260130 02:59:14.062059 21961 catalog_manager.cc:5487] Ignoring report from unknown tablet 5cc07f1d50624e03b1f5498b12ea680b
W20260130 02:59:14.070894 21961 catalog_manager.cc:5487] Ignoring report from unknown tablet 5cc07f1d50624e03b1f5498b12ea680b
I20260130 02:59:14.076179 22860 tablet_bootstrap.cc:492] T 5cc07f1d50624e03b1f5498b12ea680b P 111215cc989f4203a6e1517f2291d66f: No bootstrap required, opened a new log
I20260130 02:59:14.076610 22860 ts_tablet_manager.cc:1403] T 5cc07f1d50624e03b1f5498b12ea680b P 111215cc989f4203a6e1517f2291d66f: Time spent bootstrapping tablet: real 0.169s user 0.020s sys 0.027s
@ 0x7ffb013a0a1e std::__1::__function::__func<>::operator()() at ??:0
@ 0x5aa2b5 std::__1::__function::__value_func<>::operator()() at ??:?
I20260130 02:59:14.079352 22860 raft_consensus.cc:359] T 5cc07f1d50624e03b1f5498b12ea680b P 111215cc989f4203a6e1517f2291d66f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111215cc989f4203a6e1517f2291d66f" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 39751 } } peers { permanent_uuid: "dd32a35972b347b89ceaaad2455aef5a" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 39781 } } peers { permanent_uuid: "789dbb4a27204ffbb68d7ae40f8f5e4a" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 41825 } }
I20260130 02:59:14.080035 22860 raft_consensus.cc:385] T 5cc07f1d50624e03b1f5498b12ea680b P 111215cc989f4203a6e1517f2291d66f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
@ 0x5aa0ea std::__1::function<>::operator()() at ??:?
I20260130 02:59:14.080303 22860 raft_consensus.cc:740] T 5cc07f1d50624e03b1f5498b12ea680b P 111215cc989f4203a6e1517f2291d66f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 111215cc989f4203a6e1517f2291d66f, State: Initialized, Role: FOLLOWER
I20260130 02:59:14.080993 22860 consensus_queue.cc:260] T 5cc07f1d50624e03b1f5498b12ea680b P 111215cc989f4203a6e1517f2291d66f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111215cc989f4203a6e1517f2291d66f" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 39751 } } peers { permanent_uuid: "dd32a35972b347b89ceaaad2455aef5a" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 39781 } } peers { permanent_uuid: "789dbb4a27204ffbb68d7ae40f8f5e4a" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 41825 } }
I20260130 02:59:14.083273 22860 ts_tablet_manager.cc:1434] T 5cc07f1d50624e03b1f5498b12ea680b P 111215cc989f4203a6e1517f2291d66f: Time spent starting tablet: real 0.006s user 0.007s sys 0.001s
I20260130 02:59:14.084051 22860 tablet_bootstrap.cc:492] T 476f8fd240bc48f1a9c47a0351ad5d27 P 111215cc989f4203a6e1517f2291d66f: Bootstrap starting.
I20260130 02:59:14.088572 22861 ts_tablet_manager.cc:1434] T 5cc07f1d50624e03b1f5498b12ea680b P 789dbb4a27204ffbb68d7ae40f8f5e4a: Time spent starting tablet: real 0.036s user 0.004s sys 0.004s
I20260130 02:59:14.089555 22861 tablet_bootstrap.cc:492] T 7e4c634c4cbf4dfbb7caf795e9f9e5c2 P 789dbb4a27204ffbb68d7ae40f8f5e4a: Bootstrap starting.
I20260130 02:59:14.095917 22859 ts_tablet_manager.cc:1916] T 476f8fd240bc48f1a9c47a0351ad5d27 P 789dbb4a27204ffbb68d7ae40f8f5e4a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260130 02:59:14.098896 22854 ts_tablet_manager.cc:1916] T 5cc07f1d50624e03b1f5498b12ea680b P dd32a35972b347b89ceaaad2455aef5a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260130 02:59:14.100162 22857 tablet_bootstrap.cc:492] T 476f8fd240bc48f1a9c47a0351ad5d27 P dd32a35972b347b89ceaaad2455aef5a: No bootstrap required, opened a new log
I20260130 02:59:14.100569 22857 ts_tablet_manager.cc:1403] T 476f8fd240bc48f1a9c47a0351ad5d27 P dd32a35972b347b89ceaaad2455aef5a: Time spent bootstrapping tablet: real 0.098s user 0.004s sys 0.007s
I20260130 02:59:14.100993 22860 tablet_bootstrap.cc:654] T 476f8fd240bc48f1a9c47a0351ad5d27 P 111215cc989f4203a6e1517f2291d66f: Neither blocks nor log segments found. Creating new log.
I20260130 02:59:14.103055 22226 tablet_service.cc:1505] Processing CreateTablet for tablet 010faaeaf5e049a7893e4bf97b3f73e8 (DEFAULT_TABLE table=default.table_62e153a4e7844d97863f32a6e87da70b [id=dee95603eab3439b81355a006bc9f376]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:59:14.103086 21960 catalog_manager.cc:5654] T 7e4c634c4cbf4dfbb7caf795e9f9e5c2 P 111215cc989f4203a6e1517f2291d66f reported cstate change: term changed from 0 to 1, leader changed from <none> to 111215cc989f4203a6e1517f2291d66f (127.18.100.65). New cstate: current_term: 1 leader_uuid: "111215cc989f4203a6e1517f2291d66f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111215cc989f4203a6e1517f2291d66f" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 39751 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "789dbb4a27204ffbb68d7ae40f8f5e4a" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 41825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dd32a35972b347b89ceaaad2455aef5a" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 39781 } health_report { overall_health: UNKNOWN } } }
I20260130 02:59:14.103758 22368 tablet_service.cc:1505] Processing CreateTablet for tablet 010faaeaf5e049a7893e4bf97b3f73e8 (DEFAULT_TABLE table=default.table_62e153a4e7844d97863f32a6e87da70b [id=dee95603eab3439b81355a006bc9f376]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:59:14.104115 22226 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 010faaeaf5e049a7893e4bf97b3f73e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:59:14.104774 22368 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 010faaeaf5e049a7893e4bf97b3f73e8. 1 dirs total, 0 dirs full, 0 dirs failed
W20260130 02:59:14.107383 21961 catalog_manager.cc:5487] Ignoring report from unknown tablet 5cc07f1d50624e03b1f5498b12ea680b
I20260130 02:59:14.112860 22857 raft_consensus.cc:359] T 476f8fd240bc48f1a9c47a0351ad5d27 P dd32a35972b347b89ceaaad2455aef5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111215cc989f4203a6e1517f2291d66f" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 39751 } } peers { permanent_uuid: "dd32a35972b347b89ceaaad2455aef5a" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 39781 } } peers { permanent_uuid: "789dbb4a27204ffbb68d7ae40f8f5e4a" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 41825 } }
I20260130 02:59:14.113525 22857 raft_consensus.cc:385] T 476f8fd240bc48f1a9c47a0351ad5d27 P dd32a35972b347b89ceaaad2455aef5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:59:14.113790 22857 raft_consensus.cc:740] T 476f8fd240bc48f1a9c47a0351ad5d27 P dd32a35972b347b89ceaaad2455aef5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd32a35972b347b89ceaaad2455aef5a, State: Initialized, Role: FOLLOWER
I20260130 02:59:14.114826 22857 consensus_queue.cc:260] T 476f8fd240bc48f1a9c47a0351ad5d27 P dd32a35972b347b89ceaaad2455aef5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111215cc989f4203a6e1517f2291d66f" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 39751 } } peers { permanent_uuid: "dd32a35972b347b89ceaaad2455aef5a" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 39781 } } peers { permanent_uuid: "789dbb4a27204ffbb68d7ae40f8f5e4a" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 41825 } }
I20260130 02:59:14.116158 22861 tablet_bootstrap.cc:654] T 7e4c634c4cbf4dfbb7caf795e9f9e5c2 P 789dbb4a27204ffbb68d7ae40f8f5e4a: Neither blocks nor log segments found. Creating new log.
I20260130 02:59:14.117743 22511 tablet_service.cc:1505] Processing CreateTablet for tablet 010faaeaf5e049a7893e4bf97b3f73e8 (DEFAULT_TABLE table=default.table_62e153a4e7844d97863f32a6e87da70b [id=dee95603eab3439b81355a006bc9f376]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:59:14.121382 22511 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 010faaeaf5e049a7893e4bf97b3f73e8. 1 dirs total, 0 dirs full, 0 dirs failed
W20260130 02:59:14.122496 21961 catalog_manager.cc:5487] Ignoring report from unknown tablet 476f8fd240bc48f1a9c47a0351ad5d27
I20260130 02:59:14.125002 22857 ts_tablet_manager.cc:1434] T 476f8fd240bc48f1a9c47a0351ad5d27 P dd32a35972b347b89ceaaad2455aef5a: Time spent starting tablet: real 0.024s user 0.005s sys 0.001s
W20260130 02:59:14.128289 21961 catalog_manager.cc:5487] Ignoring report from unknown tablet 476f8fd240bc48f1a9c47a0351ad5d27
W20260130 02:59:14.142282 21960 catalog_manager.cc:5487] Ignoring report from unknown tablet 5cc07f1d50624e03b1f5498b12ea680b
I20260130 02:59:14.147140 22227 tablet_service.cc:1505] Processing CreateTablet for tablet 670a199ebd03458cb3dae713581506f7 (DEFAULT_TABLE table=default.table_d6f2cd9cd3b54168b14f1cacabbc74f0 [id=db113a9d4bf545faa72c7c94427d6dba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:59:14.148360 22227 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 670a199ebd03458cb3dae713581506f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:59:14.155332 22859 ts_tablet_manager.cc:1929] T 476f8fd240bc48f1a9c47a0351ad5d27 P 789dbb4a27204ffbb68d7ae40f8f5e4a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260130 02:59:14.155745 22859 log.cc:1199] T 476f8fd240bc48f1a9c47a0351ad5d27 P 789dbb4a27204ffbb68d7ae40f8f5e4a: Deleting WAL directory at /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1769741859011789-18833-0/minicluster-data/ts-2/wal/wals/476f8fd240bc48f1a9c47a0351ad5d27
I20260130 02:59:14.156386 22859 ts_tablet_manager.cc:1950] T 476f8fd240bc48f1a9c47a0351ad5d27 P 789dbb4a27204ffbb68d7ae40f8f5e4a: Deleting consensus metadata
I20260130 02:59:14.161278 22859 tablet_replica.cc:333] T 5cc07f1d50624e03b1f5498b12ea680b P 789dbb4a27204ffbb68d7ae40f8f5e4a: stopping tablet replica
I20260130 02:59:14.161917 22859 raft_consensus.cc:2243] T 5cc07f1d50624e03b1f5498b12ea680b P 789dbb4a27204ffbb68d7ae40f8f5e4a [term 0 FOLLOWER]: Raft consensus shutting down.
I20260130 02:59:14.161926 21945 catalog_manager.cc:4985] TS 789dbb4a27204ffbb68d7ae40f8f5e4a (127.18.100.67:41825): tablet 476f8fd240bc48f1a9c47a0351ad5d27 (table default.table_d6f2cd9cd3b54168b14f1cacabbc74f0 [id=db113a9d4bf545faa72c7c94427d6dba]) successfully deleted
I20260130 02:59:14.162364 22859 raft_consensus.cc:2272] T 5cc07f1d50624e03b1f5498b12ea680b P 789dbb4a27204ffbb68d7ae40f8f5e4a [term 0 FOLLOWER]: Raft consensus is shut down!
I20260130 02:59:14.165935 22861 tablet_bootstrap.cc:492] T 7e4c634c4cbf4dfbb7caf795e9f9e5c2 P 789dbb4a27204ffbb68d7ae40f8f5e4a: No bootstrap required, opened a new log
I20260130 02:59:14.166343 22861 ts_tablet_manager.cc:1403] T 7e4c634c4cbf4dfbb7caf795e9f9e5c2 P 789dbb4a27204ffbb68d7ae40f8f5e4a: Time spent bootstrapping tablet: real 0.077s user 0.016s sys 0.000s
I20260130 02:59:14.169749 22861 raft_consensus.cc:359] T 7e4c634c4cbf4dfbb7caf795e9f9e5c2 P 789dbb4a27204ffbb68d7ae40f8f5e4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111215cc989f4203a6e1517f2291d66f" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 39751 } } peers { permanent_uuid: "789dbb4a27204ffbb68d7ae40f8f5e4a" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 41825 } } peers { permanent_uuid: "dd32a35972b347b89ceaaad2455aef5a" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 39781 } }
I20260130 02:59:14.170392 22861 raft_consensus.cc:385] T 7e4c634c4cbf4dfbb7caf795e9f9e5c2 P 789dbb4a27204ffbb68d7ae40f8f5e4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:59:14.170686 22861 raft_consensus.cc:740] T 7e4c634c4cbf4dfbb7caf795e9f9e5c2 P 789dbb4a27204ffbb68d7ae40f8f5e4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 789dbb4a27204ffbb68d7ae40f8f5e4a, State: Initialized, Role: FOLLOWER
I20260130 02:59:14.172137 22367 tablet_service.cc:1505] Processing CreateTablet for tablet 670a199ebd03458cb3dae713581506f7 (DEFAULT_TABLE table=default.table_d6f2cd9cd3b54168b14f1cacabbc74f0 [id=db113a9d4bf545faa72c7c94427d6dba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:59:14.172585 22859 ts_tablet_manager.cc:1916] T 5cc07f1d50624e03b1f5498b12ea680b P 789dbb4a27204ffbb68d7ae40f8f5e4a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260130 02:59:14.173228 22367 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 670a199ebd03458cb3dae713581506f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:59:14.176637 22860 tablet_bootstrap.cc:492] T 476f8fd240bc48f1a9c47a0351ad5d27 P 111215cc989f4203a6e1517f2291d66f: No bootstrap required, opened a new log
I20260130 02:59:14.177064 22860 ts_tablet_manager.cc:1403] T 476f8fd240bc48f1a9c47a0351ad5d27 P 111215cc989f4203a6e1517f2291d66f: Time spent bootstrapping tablet: real 0.093s user 0.006s sys 0.004s
I20260130 02:59:14.179651 22860 raft_consensus.cc:359] T 476f8fd240bc48f1a9c47a0351ad5d27 P 111215cc989f4203a6e1517f2291d66f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111215cc989f4203a6e1517f2291d66f" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 39751 } } peers { permanent_uuid: "dd32a35972b347b89ceaaad2455aef5a" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 39781 } } peers { permanent_uuid: "789dbb4a27204ffbb68d7ae40f8f5e4a" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 41825 } }
I20260130 02:59:14.180289 22860 raft_consensus.cc:385] T 476f8fd240bc48f1a9c47a0351ad5d27 P 111215cc989f4203a6e1517f2291d66f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:59:14.180536 22860 raft_consensus.cc:740] T 476f8fd240bc48f1a9c47a0351ad5d27 P 111215cc989f4203a6e1517f2291d66f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 111215cc989f4203a6e1517f2291d66f, State: Initialized, Role: FOLLOWER
I20260130 02:59:14.181445 22860 consensus_queue.cc:260] T 476f8fd240bc48f1a9c47a0351ad5d27 P 111215cc989f4203a6e1517f2291d66f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111215cc989f4203a6e1517f2291d66f" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 39751 } } peers { permanent_uuid: "dd32a35972b347b89ceaaad2455aef5a" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 39781 } } peers { permanent_uuid: "789dbb4a27204ffbb68d7ae40f8f5e4a" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 41825 } }
I20260130 02:59:14.183718 22860 ts_tablet_manager.cc:1434] T 476f8fd240bc48f1a9c47a0351ad5d27 P 111215cc989f4203a6e1517f2291d66f: Time spent starting tablet: real 0.006s user 0.006s sys 0.001s
I20260130 02:59:14.184662 22860 tablet_bootstrap.cc:492] T 010faaeaf5e049a7893e4bf97b3f73e8 P 111215cc989f4203a6e1517f2291d66f: Bootstrap starting.
W20260130 02:59:14.186189 21960 catalog_manager.cc:5487] Ignoring report from unknown tablet 476f8fd240bc48f1a9c47a0351ad5d27
I20260130 02:59:14.187412 22868 raft_consensus.cc:493] T 476f8fd240bc48f1a9c47a0351ad5d27 P 111215cc989f4203a6e1517f2291d66f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:59:14.187791 22868 raft_consensus.cc:515] T 476f8fd240bc48f1a9c47a0351ad5d27 P 111215cc989f4203a6e1517f2291d66f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111215cc989f4203a6e1517f2291d66f" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 39751 } } peers { permanent_uuid: "dd32a35972b347b89ceaaad2455aef5a" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 39781 } } peers { permanent_uuid: "789dbb4a27204ffbb68d7ae40f8f5e4a" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 41825 } }
I20260130 02:59:14.189459 22854 ts_tablet_manager.cc:1929] T 5cc07f1d50624e03b1f5498b12ea680b P dd32a35972b347b89ceaaad2455aef5a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260130 02:59:14.189821 22854 log.cc:1199] T 5cc07f1d50624e03b1f5498b12ea680b P dd32a35972b347b89ceaaad2455aef5a: Deleting WAL directory at /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1769741859011789-18833-0/minicluster-data/ts-1/wal/wals/5cc07f1d50624e03b1f5498b12ea680b
I20260130 02:59:14.189827 22868 leader_election.cc:290] T 476f8fd240bc48f1a9c47a0351ad5d27 P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dd32a35972b347b89ceaaad2455aef5a (127.18.100.66:39781), 789dbb4a27204ffbb68d7ae40f8f5e4a (127.18.100.67:41825)
I20260130 02:59:14.190464 22854 ts_tablet_manager.cc:1950] T 5cc07f1d50624e03b1f5498b12ea680b P dd32a35972b347b89ceaaad2455aef5a: Deleting consensus metadata
W20260130 02:59:14.192047 22179 leader_election.cc:343] T 476f8fd240bc48f1a9c47a0351ad5d27 P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 789dbb4a27204ffbb68d7ae40f8f5e4a (127.18.100.67:41825): Not found: Tablet not found: 476f8fd240bc48f1a9c47a0351ad5d27
I20260130 02:59:14.193677 22391 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "476f8fd240bc48f1a9c47a0351ad5d27" candidate_uuid: "111215cc989f4203a6e1517f2291d66f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd32a35972b347b89ceaaad2455aef5a" is_pre_election: true
I20260130 02:59:14.194240 22391 raft_consensus.cc:2468] T 476f8fd240bc48f1a9c47a0351ad5d27 P dd32a35972b347b89ceaaad2455aef5a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 111215cc989f4203a6e1517f2291d66f in term 0.
W20260130 02:59:14.196411 21960 catalog_manager.cc:5487] Ignoring report from unknown tablet 476f8fd240bc48f1a9c47a0351ad5d27
I20260130 02:59:14.196411 21961 master_service.cc:954] ReplaceTablet: received request to replace tablet 010faaeaf5e049a7893e4bf97b3f73e8 from {username='slave'} at 127.0.0.1:57858
I20260130 02:59:14.196285 22511 tablet_service.cc:1505] Processing CreateTablet for tablet 670a199ebd03458cb3dae713581506f7 (DEFAULT_TABLE table=default.table_d6f2cd9cd3b54168b14f1cacabbc74f0 [id=db113a9d4bf545faa72c7c94427d6dba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:59:14.178145 22861 consensus_queue.cc:260] T 7e4c634c4cbf4dfbb7caf795e9f9e5c2 P 789dbb4a27204ffbb68d7ae40f8f5e4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111215cc989f4203a6e1517f2291d66f" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 39751 } } peers { permanent_uuid: "789dbb4a27204ffbb68d7ae40f8f5e4a" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 41825 } } peers { permanent_uuid: "dd32a35972b347b89ceaaad2455aef5a" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 39781 } }
I20260130 02:59:14.204365 22180 leader_election.cc:304] T 476f8fd240bc48f1a9c47a0351ad5d27 P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 111215cc989f4203a6e1517f2291d66f, dd32a35972b347b89ceaaad2455aef5a; no voters: 789dbb4a27204ffbb68d7ae40f8f5e4a
I20260130 02:59:14.205808 22868 raft_consensus.cc:2804] T 476f8fd240bc48f1a9c47a0351ad5d27 P 111215cc989f4203a6e1517f2291d66f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260130 02:59:14.205897 22861 ts_tablet_manager.cc:1434] T 7e4c634c4cbf4dfbb7caf795e9f9e5c2 P 789dbb4a27204ffbb68d7ae40f8f5e4a: Time spent starting tablet: real 0.039s user 0.004s sys 0.001s
I20260130 02:59:14.206140 22868 raft_consensus.cc:493] T 476f8fd240bc48f1a9c47a0351ad5d27 P 111215cc989f4203a6e1517f2291d66f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260130 02:59:14.206444 22868 raft_consensus.cc:3060] T 476f8fd240bc48f1a9c47a0351ad5d27 P 111215cc989f4203a6e1517f2291d66f [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:59:14.197338 22511 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 670a199ebd03458cb3dae713581506f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:59:14.211546 21946 catalog_manager.cc:4985] TS dd32a35972b347b89ceaaad2455aef5a (127.18.100.66:39781): tablet 5cc07f1d50624e03b1f5498b12ea680b (table default.table_62e153a4e7844d97863f32a6e87da70b [id=dee95603eab3439b81355a006bc9f376]) successfully deleted
@ 0x7ffaf852069b kudu::Thread::SuperviseThread() at ??:0
@ 0x474190 __tsan_thread_start_func at ??:?
I20260130 02:59:14.212972 22861 tablet_bootstrap.cc:492] T 010faaeaf5e049a7893e4bf97b3f73e8 P 789dbb4a27204ffbb68d7ae40f8f5e4a: Bootstrap starting.
I20260130 02:59:14.222645 21961 catalog_manager.cc:5941] T 00000000000000000000000000000000 P b01057b6c5ff4fc4b07253801881fe13: Sending DeleteTablet for 3 replicas of tablet 010faaeaf5e049a7893e4bf97b3f73e8
I20260130 02:59:14.224234 21961 catalog_manager.cc:6724] ReplaceTablet: tablet 010faaeaf5e049a7893e4bf97b3f73e8 deleted and replaced by tablet 563d05e06b1f48b4be140219da5c9dbd
I20260130 02:59:14.224316 22226 tablet_service.cc:1552] Processing DeleteTablet for tablet 010faaeaf5e049a7893e4bf97b3f73e8 with delete_type TABLET_DATA_DELETED (replaced by tablet 563d05e06b1f48b4be140219da5c9dbd) from {username='slave'} at 127.0.0.1:53820
I20260130 02:59:14.224488 22369 tablet_service.cc:1552] Processing DeleteTablet for tablet 010faaeaf5e049a7893e4bf97b3f73e8 with delete_type TABLET_DATA_DELETED (replaced by tablet 563d05e06b1f48b4be140219da5c9dbd) from {username='slave'} at 127.0.0.1:44088
I20260130 02:59:14.225904 22514 tablet_service.cc:1552] Processing DeleteTablet for tablet 010faaeaf5e049a7893e4bf97b3f73e8 with delete_type TABLET_DATA_DELETED (replaced by tablet 563d05e06b1f48b4be140219da5c9dbd) from {username='slave'} at 127.0.0.1:53092
W20260130 02:59:14.226681 21946 catalog_manager.cc:4952] TS dd32a35972b347b89ceaaad2455aef5a (127.18.100.66:39781): delete failed for tablet 010faaeaf5e049a7893e4bf97b3f73e8 because the tablet was not found. No further retry: Not found: Tablet not found: 010faaeaf5e049a7893e4bf97b3f73e8
W20260130 02:59:14.227003 21945 catalog_manager.cc:4965] TS 111215cc989f4203a6e1517f2291d66f (127.18.100.65:39751): delete failed for tablet 010faaeaf5e049a7893e4bf97b3f73e8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 010faaeaf5e049a7893e4bf97b3f73e8 already in progress: creating tablet
I20260130 02:59:14.235251 22861 tablet_bootstrap.cc:654] T 010faaeaf5e049a7893e4bf97b3f73e8 P 789dbb4a27204ffbb68d7ae40f8f5e4a: Neither blocks nor log segments found. Creating new log.
I20260130 02:59:14.240190 22860 tablet_bootstrap.cc:654] T 010faaeaf5e049a7893e4bf97b3f73e8 P 111215cc989f4203a6e1517f2291d66f: Neither blocks nor log segments found. Creating new log.
I20260130 02:59:14.243304 22857 tablet_bootstrap.cc:492] T 010faaeaf5e049a7893e4bf97b3f73e8 P dd32a35972b347b89ceaaad2455aef5a: Bootstrap starting.
I20260130 02:59:14.223500 22868 raft_consensus.cc:515] T 476f8fd240bc48f1a9c47a0351ad5d27 P 111215cc989f4203a6e1517f2291d66f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111215cc989f4203a6e1517f2291d66f" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 39751 } } peers { permanent_uuid: "dd32a35972b347b89ceaaad2455aef5a" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 39781 } } peers { permanent_uuid: "789dbb4a27204ffbb68d7ae40f8f5e4a" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 41825 } }
I20260130 02:59:14.249248 22857 tablet_bootstrap.cc:654] T 010faaeaf5e049a7893e4bf97b3f73e8 P dd32a35972b347b89ceaaad2455aef5a: Neither blocks nor log segments found. Creating new log.
I20260130 02:59:14.251582 22391 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "476f8fd240bc48f1a9c47a0351ad5d27" candidate_uuid: "111215cc989f4203a6e1517f2291d66f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd32a35972b347b89ceaaad2455aef5a"
I20260130 02:59:14.252112 22391 raft_consensus.cc:3060] T 476f8fd240bc48f1a9c47a0351ad5d27 P dd32a35972b347b89ceaaad2455aef5a [term 0 FOLLOWER]: Advancing to term 1
W20260130 02:59:14.255173 22179 leader_election.cc:343] T 476f8fd240bc48f1a9c47a0351ad5d27 P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 789dbb4a27204ffbb68d7ae40f8f5e4a (127.18.100.67:41825): Not found: Tablet not found: 476f8fd240bc48f1a9c47a0351ad5d27
I20260130 02:59:14.256847 22859 ts_tablet_manager.cc:1929] T 5cc07f1d50624e03b1f5498b12ea680b P 789dbb4a27204ffbb68d7ae40f8f5e4a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260130 02:59:14.257225 22859 log.cc:1199] T 5cc07f1d50624e03b1f5498b12ea680b P 789dbb4a27204ffbb68d7ae40f8f5e4a: Deleting WAL directory at /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1769741859011789-18833-0/minicluster-data/ts-2/wal/wals/5cc07f1d50624e03b1f5498b12ea680b
I20260130 02:59:14.257866 22859 ts_tablet_manager.cc:1950] T 5cc07f1d50624e03b1f5498b12ea680b P 789dbb4a27204ffbb68d7ae40f8f5e4a: Deleting consensus metadata
I20260130 02:59:14.259790 21961 master_service.cc:954] ReplaceTablet: received request to replace tablet 7e4c634c4cbf4dfbb7caf795e9f9e5c2 from {username='slave'} at 127.0.0.1:57858
I20260130 02:59:14.259994 22868 leader_election.cc:290] T 476f8fd240bc48f1a9c47a0351ad5d27 P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 election: Requested vote from peers dd32a35972b347b89ceaaad2455aef5a (127.18.100.66:39781), 789dbb4a27204ffbb68d7ae40f8f5e4a (127.18.100.67:41825)
I20260130 02:59:14.263150 22391 raft_consensus.cc:2468] T 476f8fd240bc48f1a9c47a0351ad5d27 P dd32a35972b347b89ceaaad2455aef5a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 111215cc989f4203a6e1517f2291d66f in term 1.
I20260130 02:59:14.264693 22861 tablet_bootstrap.cc:492] T 010faaeaf5e049a7893e4bf97b3f73e8 P 789dbb4a27204ffbb68d7ae40f8f5e4a: No bootstrap required, opened a new log
I20260130 02:59:14.265118 22861 ts_tablet_manager.cc:1403] T 010faaeaf5e049a7893e4bf97b3f73e8 P 789dbb4a27204ffbb68d7ae40f8f5e4a: Time spent bootstrapping tablet: real 0.052s user 0.001s sys 0.011s
I20260130 02:59:14.272567 21945 catalog_manager.cc:4985] TS 789dbb4a27204ffbb68d7ae40f8f5e4a (127.18.100.67:41825): tablet 5cc07f1d50624e03b1f5498b12ea680b (table default.table_62e153a4e7844d97863f32a6e87da70b [id=dee95603eab3439b81355a006bc9f376]) successfully deleted
I20260130 02:59:14.275120 22180 leader_election.cc:304] T 476f8fd240bc48f1a9c47a0351ad5d27 P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 111215cc989f4203a6e1517f2291d66f, dd32a35972b347b89ceaaad2455aef5a; no voters: 789dbb4a27204ffbb68d7ae40f8f5e4a
I20260130 02:59:14.275849 22857 tablet_bootstrap.cc:492] T 010faaeaf5e049a7893e4bf97b3f73e8 P dd32a35972b347b89ceaaad2455aef5a: No bootstrap required, opened a new log
I20260130 02:59:14.276286 22857 ts_tablet_manager.cc:1403] T 010faaeaf5e049a7893e4bf97b3f73e8 P dd32a35972b347b89ceaaad2455aef5a: Time spent bootstrapping tablet: real 0.033s user 0.011s sys 0.000s
I20260130 02:59:14.276373 22868 raft_consensus.cc:2804] T 476f8fd240bc48f1a9c47a0351ad5d27 P 111215cc989f4203a6e1517f2291d66f [term 1 FOLLOWER]: Leader election won for term 1
I20260130 02:59:14.276791 22868 raft_consensus.cc:697] T 476f8fd240bc48f1a9c47a0351ad5d27 P 111215cc989f4203a6e1517f2291d66f [term 1 LEADER]: Becoming Leader. State: Replica: 111215cc989f4203a6e1517f2291d66f, State: Running, Role: LEADER
I20260130 02:59:14.278273 22868 consensus_queue.cc:237] T 476f8fd240bc48f1a9c47a0351ad5d27 P 111215cc989f4203a6e1517f2291d66f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111215cc989f4203a6e1517f2291d66f" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 39751 } } peers { permanent_uuid: "dd32a35972b347b89ceaaad2455aef5a" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 39781 } } peers { permanent_uuid: "789dbb4a27204ffbb68d7ae40f8f5e4a" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 41825 } }
I20260130 02:59:14.280972 22857 raft_consensus.cc:359] T 010faaeaf5e049a7893e4bf97b3f73e8 P dd32a35972b347b89ceaaad2455aef5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111215cc989f4203a6e1517f2291d66f" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 39751 } } peers { permanent_uuid: "dd32a35972b347b89ceaaad2455aef5a" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 39781 } } peers { permanent_uuid: "789dbb4a27204ffbb68d7ae40f8f5e4a" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 41825 } }
I20260130 02:59:14.283051 22857 raft_consensus.cc:385] T 010faaeaf5e049a7893e4bf97b3f73e8 P dd32a35972b347b89ceaaad2455aef5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:59:14.283340 22857 raft_consensus.cc:740] T 010faaeaf5e049a7893e4bf97b3f73e8 P dd32a35972b347b89ceaaad2455aef5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd32a35972b347b89ceaaad2455aef5a, State: Initialized, Role: FOLLOWER
W20260130 02:59:14.284744 21945 catalog_manager.cc:4965] TS 789dbb4a27204ffbb68d7ae40f8f5e4a (127.18.100.67:41825): delete failed for tablet 010faaeaf5e049a7893e4bf97b3f73e8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 010faaeaf5e049a7893e4bf97b3f73e8 already in progress: creating tablet
I20260130 02:59:14.285425 21961 catalog_manager.cc:5941] T 00000000000000000000000000000000 P b01057b6c5ff4fc4b07253801881fe13: Sending DeleteTablet for 3 replicas of tablet 7e4c634c4cbf4dfbb7caf795e9f9e5c2
I20260130 02:59:14.267545 22861 raft_consensus.cc:359] T 010faaeaf5e049a7893e4bf97b3f73e8 P 789dbb4a27204ffbb68d7ae40f8f5e4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111215cc989f4203a6e1517f2291d66f" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 39751 } } peers { permanent_uuid: "dd32a35972b347b89ceaaad2455aef5a" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 39781 } } peers { permanent_uuid: "789dbb4a27204ffbb68d7ae40f8f5e4a" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 41825 } }
I20260130 02:59:14.285892 22861 raft_consensus.cc:385] T 010faaeaf5e049a7893e4bf97b3f73e8 P 789dbb4a27204ffbb68d7ae40f8f5e4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:59:14.286176 22861 raft_consensus.cc:740] T 010faaeaf5e049a7893e4bf97b3f73e8 P 789dbb4a27204ffbb68d7ae40f8f5e4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 789dbb4a27204ffbb68d7ae40f8f5e4a, State: Initialized, Role: FOLLOWER
I20260130 02:59:14.286763 21961 catalog_manager.cc:6724] ReplaceTablet: tablet 7e4c634c4cbf4dfbb7caf795e9f9e5c2 deleted and replaced by tablet ce91a17790324cd884a0521ce67dde67
@ 0x7ffaf9c2b6db start_thread at ??:0
I20260130 02:59:14.291455 22511 tablet_service.cc:1552] Processing DeleteTablet for tablet 7e4c634c4cbf4dfbb7caf795e9f9e5c2 with delete_type TABLET_DATA_DELETED (replaced by tablet ce91a17790324cd884a0521ce67dde67) from {username='slave'} at 127.0.0.1:53092
I20260130 02:59:14.292196 22859 tablet_replica.cc:333] T 7e4c634c4cbf4dfbb7caf795e9f9e5c2 P 789dbb4a27204ffbb68d7ae40f8f5e4a: stopping tablet replica
I20260130 02:59:14.292526 22368 tablet_service.cc:1552] Processing DeleteTablet for tablet 7e4c634c4cbf4dfbb7caf795e9f9e5c2 with delete_type TABLET_DATA_DELETED (replaced by tablet ce91a17790324cd884a0521ce67dde67) from {username='slave'} at 127.0.0.1:44088
I20260130 02:59:14.292822 22859 raft_consensus.cc:2243] T 7e4c634c4cbf4dfbb7caf795e9f9e5c2 P 789dbb4a27204ffbb68d7ae40f8f5e4a [term 0 FOLLOWER]: Raft consensus shutting down.
I20260130 02:59:14.293203 22854 tablet_replica.cc:333] T 7e4c634c4cbf4dfbb7caf795e9f9e5c2 P dd32a35972b347b89ceaaad2455aef5a: stopping tablet replica
I20260130 02:59:14.293270 22859 raft_consensus.cc:2272] T 7e4c634c4cbf4dfbb7caf795e9f9e5c2 P 789dbb4a27204ffbb68d7ae40f8f5e4a [term 0 FOLLOWER]: Raft consensus is shut down!
I20260130 02:59:14.293877 22854 raft_consensus.cc:2243] T 7e4c634c4cbf4dfbb7caf795e9f9e5c2 P dd32a35972b347b89ceaaad2455aef5a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260130 02:59:14.294332 22854 raft_consensus.cc:2272] T 7e4c634c4cbf4dfbb7caf795e9f9e5c2 P dd32a35972b347b89ceaaad2455aef5a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260130 02:59:14.293996 22857 consensus_queue.cc:260] T 010faaeaf5e049a7893e4bf97b3f73e8 P dd32a35972b347b89ceaaad2455aef5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111215cc989f4203a6e1517f2291d66f" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 39751 } } peers { permanent_uuid: "dd32a35972b347b89ceaaad2455aef5a" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 39781 } } peers { permanent_uuid: "789dbb4a27204ffbb68d7ae40f8f5e4a" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 41825 } }
I20260130 02:59:14.295964 22860 tablet_bootstrap.cc:492] T 010faaeaf5e049a7893e4bf97b3f73e8 P 111215cc989f4203a6e1517f2291d66f: No bootstrap required, opened a new log
I20260130 02:59:14.296377 22860 ts_tablet_manager.cc:1403] T 010faaeaf5e049a7893e4bf97b3f73e8 P 111215cc989f4203a6e1517f2291d66f: Time spent bootstrapping tablet: real 0.112s user 0.006s sys 0.005s
I20260130 02:59:14.298043 22854 ts_tablet_manager.cc:1916] T 7e4c634c4cbf4dfbb7caf795e9f9e5c2 P dd32a35972b347b89ceaaad2455aef5a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260130 02:59:14.298841 22860 raft_consensus.cc:359] T 010faaeaf5e049a7893e4bf97b3f73e8 P 111215cc989f4203a6e1517f2291d66f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111215cc989f4203a6e1517f2291d66f" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 39751 } } peers { permanent_uuid: "dd32a35972b347b89ceaaad2455aef5a" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 39781 } } peers { permanent_uuid: "789dbb4a27204ffbb68d7ae40f8f5e4a" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 41825 } }
I20260130 02:59:14.299489 22860 raft_consensus.cc:385] T 010faaeaf5e049a7893e4bf97b3f73e8 P 111215cc989f4203a6e1517f2291d66f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:59:14.299731 22860 raft_consensus.cc:740] T 010faaeaf5e049a7893e4bf97b3f73e8 P 111215cc989f4203a6e1517f2291d66f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 111215cc989f4203a6e1517f2291d66f, State: Initialized, Role: FOLLOWER
I20260130 02:59:14.300412 22860 consensus_queue.cc:260] T 010faaeaf5e049a7893e4bf97b3f73e8 P 111215cc989f4203a6e1517f2291d66f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111215cc989f4203a6e1517f2291d66f" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 39751 } } peers { permanent_uuid: "dd32a35972b347b89ceaaad2455aef5a" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 39781 } } peers { permanent_uuid: "789dbb4a27204ffbb68d7ae40f8f5e4a" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 41825 } }
I20260130 02:59:14.302336 22859 ts_tablet_manager.cc:1916] T 7e4c634c4cbf4dfbb7caf795e9f9e5c2 P 789dbb4a27204ffbb68d7ae40f8f5e4a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260130 02:59:14.304738 22860 ts_tablet_manager.cc:1434] T 010faaeaf5e049a7893e4bf97b3f73e8 P 111215cc989f4203a6e1517f2291d66f: Time spent starting tablet: real 0.008s user 0.003s sys 0.004s
I20260130 02:59:14.305622 22860 tablet_bootstrap.cc:492] T 670a199ebd03458cb3dae713581506f7 P 111215cc989f4203a6e1517f2291d66f: Bootstrap starting.
I20260130 02:59:14.298905 22861 consensus_queue.cc:260] T 010faaeaf5e049a7893e4bf97b3f73e8 P 789dbb4a27204ffbb68d7ae40f8f5e4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111215cc989f4203a6e1517f2291d66f" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 39751 } } peers { permanent_uuid: "dd32a35972b347b89ceaaad2455aef5a" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 39781 } } peers { permanent_uuid: "789dbb4a27204ffbb68d7ae40f8f5e4a" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 41825 } }
I20260130 02:59:14.309630 22857 ts_tablet_manager.cc:1434] T 010faaeaf5e049a7893e4bf97b3f73e8 P dd32a35972b347b89ceaaad2455aef5a: Time spent starting tablet: real 0.033s user 0.008s sys 0.000s
I20260130 02:59:14.317236 22859 ts_tablet_manager.cc:1929] T 7e4c634c4cbf4dfbb7caf795e9f9e5c2 P 789dbb4a27204ffbb68d7ae40f8f5e4a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260130 02:59:14.317598 22859 log.cc:1199] T 7e4c634c4cbf4dfbb7caf795e9f9e5c2 P 789dbb4a27204ffbb68d7ae40f8f5e4a: Deleting WAL directory at /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1769741859011789-18833-0/minicluster-data/ts-2/wal/wals/7e4c634c4cbf4dfbb7caf795e9f9e5c2
I20260130 02:59:14.318423 22859 ts_tablet_manager.cc:1950] T 7e4c634c4cbf4dfbb7caf795e9f9e5c2 P 789dbb4a27204ffbb68d7ae40f8f5e4a: Deleting consensus metadata
I20260130 02:59:14.320564 22511 tablet_service.cc:1552] Processing DeleteTablet for tablet 010faaeaf5e049a7893e4bf97b3f73e8 with delete_type TABLET_DATA_DELETED (replaced by tablet 563d05e06b1f48b4be140219da5c9dbd) from {username='slave'} at 127.0.0.1:53092
I20260130 02:59:14.320721 21945 catalog_manager.cc:4985] TS 789dbb4a27204ffbb68d7ae40f8f5e4a (127.18.100.67:41825): tablet 7e4c634c4cbf4dfbb7caf795e9f9e5c2 (table default.table_e3b3683b36e144c2b33dd88b292bcb23 [id=318c872fbfef4f93b4c280fd078cabd9]) successfully deleted
W20260130 02:59:14.322364 21945 catalog_manager.cc:4965] TS 789dbb4a27204ffbb68d7ae40f8f5e4a (127.18.100.67:41825): delete failed for tablet 010faaeaf5e049a7893e4bf97b3f73e8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 010faaeaf5e049a7893e4bf97b3f73e8 already in progress: creating tablet
I20260130 02:59:14.323683 22861 ts_tablet_manager.cc:1434] T 010faaeaf5e049a7893e4bf97b3f73e8 P 789dbb4a27204ffbb68d7ae40f8f5e4a: Time spent starting tablet: real 0.058s user 0.003s sys 0.004s
I20260130 02:59:14.323957 22227 tablet_service.cc:1552] Processing DeleteTablet for tablet 7e4c634c4cbf4dfbb7caf795e9f9e5c2 with delete_type TABLET_DATA_DELETED (replaced by tablet ce91a17790324cd884a0521ce67dde67) from {username='slave'} at 127.0.0.1:53820
I20260130 02:59:14.324657 22853 tablet_replica.cc:333] T 7e4c634c4cbf4dfbb7caf795e9f9e5c2 P 111215cc989f4203a6e1517f2291d66f: stopping tablet replica
I20260130 02:59:14.325397 22853 raft_consensus.cc:2243] T 7e4c634c4cbf4dfbb7caf795e9f9e5c2 P 111215cc989f4203a6e1517f2291d66f [term 1 LEADER]: Raft consensus shutting down.
I20260130 02:59:14.326066 22853 pending_rounds.cc:70] T 7e4c634c4cbf4dfbb7caf795e9f9e5c2 P 111215cc989f4203a6e1517f2291d66f: Trying to abort 1 pending ops.
I20260130 02:59:14.327843 22853 pending_rounds.cc:77] T 7e4c634c4cbf4dfbb7caf795e9f9e5c2 P 111215cc989f4203a6e1517f2291d66f: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7248863043726774272 op_type: NO_OP noop_request { }
I20260130 02:59:14.328325 22853 raft_consensus.cc:2889] T 7e4c634c4cbf4dfbb7caf795e9f9e5c2 P 111215cc989f4203a6e1517f2291d66f [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20260130 02:59:14.328666 22853 raft_consensus.cc:2272] T 7e4c634c4cbf4dfbb7caf795e9f9e5c2 P 111215cc989f4203a6e1517f2291d66f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260130 02:59:14.330138 22368 tablet_service.cc:1552] Processing DeleteTablet for tablet 010faaeaf5e049a7893e4bf97b3f73e8 with delete_type TABLET_DATA_DELETED (replaced by tablet 563d05e06b1f48b4be140219da5c9dbd) from {username='slave'} at 127.0.0.1:44088
I20260130 02:59:14.334308 22853 ts_tablet_manager.cc:1916] T 7e4c634c4cbf4dfbb7caf795e9f9e5c2 P 111215cc989f4203a6e1517f2291d66f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260130 02:59:14.334616 22861 tablet_bootstrap.cc:492] T 670a199ebd03458cb3dae713581506f7 P 789dbb4a27204ffbb68d7ae40f8f5e4a: Bootstrap starting.
I20260130 02:59:14.337095 22227 tablet_service.cc:1505] Processing CreateTablet for tablet 563d05e06b1f48b4be140219da5c9dbd (DEFAULT_TABLE table=default.table_62e153a4e7844d97863f32a6e87da70b [id=dee95603eab3439b81355a006bc9f376]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:59:14.343393 22511 tablet_service.cc:1552] Processing DeleteTablet for tablet 010faaeaf5e049a7893e4bf97b3f73e8 with delete_type TABLET_DATA_DELETED (replaced by tablet 563d05e06b1f48b4be140219da5c9dbd) from {username='slave'} at 127.0.0.1:53092
I20260130 02:59:14.344146 22859 tablet_replica.cc:333] T 010faaeaf5e049a7893e4bf97b3f73e8 P 789dbb4a27204ffbb68d7ae40f8f5e4a: stopping tablet replica
I20260130 02:59:14.344975 22859 raft_consensus.cc:2243] T 010faaeaf5e049a7893e4bf97b3f73e8 P 789dbb4a27204ffbb68d7ae40f8f5e4a [term 0 FOLLOWER]: Raft consensus shutting down.
I20260130 02:59:14.345607 22859 raft_consensus.cc:2272] T 010faaeaf5e049a7893e4bf97b3f73e8 P 789dbb4a27204ffbb68d7ae40f8f5e4a [term 0 FOLLOWER]: Raft consensus is shut down!
I20260130 02:59:14.347855 22859 ts_tablet_manager.cc:1916] T 010faaeaf5e049a7893e4bf97b3f73e8 P 789dbb4a27204ffbb68d7ae40f8f5e4a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260130 02:59:14.347996 22227 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 563d05e06b1f48b4be140219da5c9dbd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:59:14.349243 22860 tablet_bootstrap.cc:654] T 670a199ebd03458cb3dae713581506f7 P 111215cc989f4203a6e1517f2291d66f: Neither blocks nor log segments found. Creating new log.
W20260130 02:59:14.350697 21960 catalog_manager.cc:5487] Ignoring report from unknown tablet 476f8fd240bc48f1a9c47a0351ad5d27
I20260130 02:59:14.353312 22853 ts_tablet_manager.cc:1929] T 7e4c634c4cbf4dfbb7caf795e9f9e5c2 P 111215cc989f4203a6e1517f2291d66f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260130 02:59:14.353667 22853 log.cc:1199] T 7e4c634c4cbf4dfbb7caf795e9f9e5c2 P 111215cc989f4203a6e1517f2291d66f: Deleting WAL directory at /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1769741859011789-18833-0/minicluster-data/ts-0/wal/wals/7e4c634c4cbf4dfbb7caf795e9f9e5c2
I20260130 02:59:14.354347 22853 ts_tablet_manager.cc:1950] T 7e4c634c4cbf4dfbb7caf795e9f9e5c2 P 111215cc989f4203a6e1517f2291d66f: Deleting consensus metadata
I20260130 02:59:14.356868 21945 catalog_manager.cc:4985] TS 111215cc989f4203a6e1517f2291d66f (127.18.100.65:39751): tablet 7e4c634c4cbf4dfbb7caf795e9f9e5c2 (table default.table_e3b3683b36e144c2b33dd88b292bcb23 [id=318c872fbfef4f93b4c280fd078cabd9]) successfully deleted
I20260130 02:59:14.358729 22226 tablet_service.cc:1552] Processing DeleteTablet for tablet 010faaeaf5e049a7893e4bf97b3f73e8 with delete_type TABLET_DATA_DELETED (replaced by tablet 563d05e06b1f48b4be140219da5c9dbd) from {username='slave'} at 127.0.0.1:53820
I20260130 02:59:14.359043 22857 tablet_bootstrap.cc:492] T 670a199ebd03458cb3dae713581506f7 P dd32a35972b347b89ceaaad2455aef5a: Bootstrap starting.
I20260130 02:59:14.360399 22853 tablet_replica.cc:333] T 010faaeaf5e049a7893e4bf97b3f73e8 P 111215cc989f4203a6e1517f2291d66f: stopping tablet replica
@ 0x7ffaf4fa371f clone at ??:0
I20260130 02:59:14.362445 22853 raft_consensus.cc:2243] T 010faaeaf5e049a7893e4bf97b3f73e8 P 111215cc989f4203a6e1517f2291d66f [term 0 FOLLOWER]: Raft consensus shutting down.
I20260130 02:59:14.362907 22853 raft_consensus.cc:2272] T 010faaeaf5e049a7893e4bf97b3f73e8 P 111215cc989f4203a6e1517f2291d66f [term 0 FOLLOWER]: Raft consensus is shut down!
I20260130 02:59:14.364776 22857 tablet_bootstrap.cc:654] T 670a199ebd03458cb3dae713581506f7 P dd32a35972b347b89ceaaad2455aef5a: Neither blocks nor log segments found. Creating new log.
I20260130 02:59:14.365412 22853 ts_tablet_manager.cc:1916] T 010faaeaf5e049a7893e4bf97b3f73e8 P 111215cc989f4203a6e1517f2291d66f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260130 02:59:14.376377 22860 tablet_bootstrap.cc:492] T 670a199ebd03458cb3dae713581506f7 P 111215cc989f4203a6e1517f2291d66f: No bootstrap required, opened a new log
I20260130 02:59:14.376847 22860 ts_tablet_manager.cc:1403] T 670a199ebd03458cb3dae713581506f7 P 111215cc989f4203a6e1517f2291d66f: Time spent bootstrapping tablet: real 0.071s user 0.009s sys 0.000s
I20260130 02:59:14.379495 22860 raft_consensus.cc:359] T 670a199ebd03458cb3dae713581506f7 P 111215cc989f4203a6e1517f2291d66f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111215cc989f4203a6e1517f2291d66f" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 39751 } } peers { permanent_uuid: "dd32a35972b347b89ceaaad2455aef5a" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 39781 } } peers { permanent_uuid: "789dbb4a27204ffbb68d7ae40f8f5e4a" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 41825 } }
I20260130 02:59:14.380129 22860 raft_consensus.cc:385] T 670a199ebd03458cb3dae713581506f7 P 111215cc989f4203a6e1517f2291d66f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:59:14.380070 22853 ts_tablet_manager.cc:1929] T 010faaeaf5e049a7893e4bf97b3f73e8 P 111215cc989f4203a6e1517f2291d66f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260130 02:59:14.386792 22853 log.cc:1199] T 010faaeaf5e049a7893e4bf97b3f73e8 P 111215cc989f4203a6e1517f2291d66f: Deleting WAL directory at /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1769741859011789-18833-0/minicluster-data/ts-0/wal/wals/010faaeaf5e049a7893e4bf97b3f73e8
I20260130 02:59:14.387549 22853 ts_tablet_manager.cc:1950] T 010faaeaf5e049a7893e4bf97b3f73e8 P 111215cc989f4203a6e1517f2291d66f: Deleting consensus metadata
I20260130 02:59:14.380460 22860 raft_consensus.cc:740] T 670a199ebd03458cb3dae713581506f7 P 111215cc989f4203a6e1517f2291d66f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 111215cc989f4203a6e1517f2291d66f, State: Initialized, Role: FOLLOWER
I20260130 02:59:14.391238 22860 consensus_queue.cc:260] T 670a199ebd03458cb3dae713581506f7 P 111215cc989f4203a6e1517f2291d66f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111215cc989f4203a6e1517f2291d66f" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 39751 } } peers { permanent_uuid: "dd32a35972b347b89ceaaad2455aef5a" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 39781 } } peers { permanent_uuid: "789dbb4a27204ffbb68d7ae40f8f5e4a" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 41825 } }
I20260130 02:59:14.393757 22861 tablet_bootstrap.cc:654] T 670a199ebd03458cb3dae713581506f7 P 789dbb4a27204ffbb68d7ae40f8f5e4a: Neither blocks nor log segments found. Creating new log.
I20260130 02:59:14.399979 22854 ts_tablet_manager.cc:1929] T 7e4c634c4cbf4dfbb7caf795e9f9e5c2 P dd32a35972b347b89ceaaad2455aef5a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260130 02:59:14.400360 22854 log.cc:1199] T 7e4c634c4cbf4dfbb7caf795e9f9e5c2 P dd32a35972b347b89ceaaad2455aef5a: Deleting WAL directory at /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1769741859011789-18833-0/minicluster-data/ts-1/wal/wals/7e4c634c4cbf4dfbb7caf795e9f9e5c2
I20260130 02:59:14.400949 22854 ts_tablet_manager.cc:1950] T 7e4c634c4cbf4dfbb7caf795e9f9e5c2 P dd32a35972b347b89ceaaad2455aef5a: Deleting consensus metadata
I20260130 02:59:14.407866 22860 ts_tablet_manager.cc:1434] T 670a199ebd03458cb3dae713581506f7 P 111215cc989f4203a6e1517f2291d66f: Time spent starting tablet: real 0.031s user 0.005s sys 0.002s
I20260130 02:59:14.409482 22859 ts_tablet_manager.cc:1929] T 010faaeaf5e049a7893e4bf97b3f73e8 P 789dbb4a27204ffbb68d7ae40f8f5e4a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260130 02:59:14.409858 22859 log.cc:1199] T 010faaeaf5e049a7893e4bf97b3f73e8 P 789dbb4a27204ffbb68d7ae40f8f5e4a: Deleting WAL directory at /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1769741859011789-18833-0/minicluster-data/ts-2/wal/wals/010faaeaf5e049a7893e4bf97b3f73e8
I20260130 02:59:14.410303 22860 tablet_bootstrap.cc:492] T 563d05e06b1f48b4be140219da5c9dbd P 111215cc989f4203a6e1517f2291d66f: Bootstrap starting.
I20260130 02:59:14.414199 22854 tablet_replica.cc:333] T 010faaeaf5e049a7893e4bf97b3f73e8 P dd32a35972b347b89ceaaad2455aef5a: stopping tablet replica
I20260130 02:59:14.415732 22854 raft_consensus.cc:2243] T 010faaeaf5e049a7893e4bf97b3f73e8 P dd32a35972b347b89ceaaad2455aef5a [term 0 FOLLOWER]: Raft consensus shutting down.
I20260130 02:59:14.416216 22854 raft_consensus.cc:2272] T 010faaeaf5e049a7893e4bf97b3f73e8 P dd32a35972b347b89ceaaad2455aef5a [term 0 FOLLOWER]: Raft consensus is shut down!
I20260130 02:59:14.419153 22857 tablet_bootstrap.cc:492] T 670a199ebd03458cb3dae713581506f7 P dd32a35972b347b89ceaaad2455aef5a: No bootstrap required, opened a new log
I20260130 02:59:14.419106 22854 ts_tablet_manager.cc:1916] T 010faaeaf5e049a7893e4bf97b3f73e8 P dd32a35972b347b89ceaaad2455aef5a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260130 02:59:14.419569 22857 ts_tablet_manager.cc:1403] T 670a199ebd03458cb3dae713581506f7 P dd32a35972b347b89ceaaad2455aef5a: Time spent bootstrapping tablet: real 0.061s user 0.004s sys 0.006s
I20260130 02:59:14.419775 22860 tablet_bootstrap.cc:654] T 563d05e06b1f48b4be140219da5c9dbd P 111215cc989f4203a6e1517f2291d66f: Neither blocks nor log segments found. Creating new log.
I20260130 02:59:14.421990 22857 raft_consensus.cc:359] T 670a199ebd03458cb3dae713581506f7 P dd32a35972b347b89ceaaad2455aef5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111215cc989f4203a6e1517f2291d66f" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 39751 } } peers { permanent_uuid: "dd32a35972b347b89ceaaad2455aef5a" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 39781 } } peers { permanent_uuid: "789dbb4a27204ffbb68d7ae40f8f5e4a" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 41825 } }
I20260130 02:59:14.431453 22861 tablet_bootstrap.cc:492] T 670a199ebd03458cb3dae713581506f7 P 789dbb4a27204ffbb68d7ae40f8f5e4a: No bootstrap required, opened a new log
I20260130 02:59:14.431895 22861 ts_tablet_manager.cc:1403] T 670a199ebd03458cb3dae713581506f7 P 789dbb4a27204ffbb68d7ae40f8f5e4a: Time spent bootstrapping tablet: real 0.097s user 0.010s sys 0.000s
I20260130 02:59:14.434527 22861 raft_consensus.cc:359] T 670a199ebd03458cb3dae713581506f7 P 789dbb4a27204ffbb68d7ae40f8f5e4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111215cc989f4203a6e1517f2291d66f" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 39751 } } peers { permanent_uuid: "dd32a35972b347b89ceaaad2455aef5a" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 39781 } } peers { permanent_uuid: "789dbb4a27204ffbb68d7ae40f8f5e4a" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 41825 } }
I20260130 02:59:14.435261 22861 raft_consensus.cc:385] T 670a199ebd03458cb3dae713581506f7 P 789dbb4a27204ffbb68d7ae40f8f5e4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:59:14.435530 22861 raft_consensus.cc:740] T 670a199ebd03458cb3dae713581506f7 P 789dbb4a27204ffbb68d7ae40f8f5e4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 789dbb4a27204ffbb68d7ae40f8f5e4a, State: Initialized, Role: FOLLOWER
I20260130 02:59:14.410470 22859 ts_tablet_manager.cc:1950] T 010faaeaf5e049a7893e4bf97b3f73e8 P 789dbb4a27204ffbb68d7ae40f8f5e4a: Deleting consensus metadata
I20260130 02:59:14.437666 22857 raft_consensus.cc:385] T 670a199ebd03458cb3dae713581506f7 P dd32a35972b347b89ceaaad2455aef5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:59:14.438038 22857 raft_consensus.cc:740] T 670a199ebd03458cb3dae713581506f7 P dd32a35972b347b89ceaaad2455aef5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd32a35972b347b89ceaaad2455aef5a, State: Initialized, Role: FOLLOWER
I20260130 02:59:14.438822 22857 consensus_queue.cc:260] T 670a199ebd03458cb3dae713581506f7 P dd32a35972b347b89ceaaad2455aef5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111215cc989f4203a6e1517f2291d66f" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 39751 } } peers { permanent_uuid: "dd32a35972b347b89ceaaad2455aef5a" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 39781 } } peers { permanent_uuid: "789dbb4a27204ffbb68d7ae40f8f5e4a" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 41825 } }
I20260130 02:59:14.441751 22854 ts_tablet_manager.cc:1929] T 010faaeaf5e049a7893e4bf97b3f73e8 P dd32a35972b347b89ceaaad2455aef5a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260130 02:59:14.442163 22854 log.cc:1199] T 010faaeaf5e049a7893e4bf97b3f73e8 P dd32a35972b347b89ceaaad2455aef5a: Deleting WAL directory at /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1769741859011789-18833-0/minicluster-data/ts-1/wal/wals/010faaeaf5e049a7893e4bf97b3f73e8
I20260130 02:59:14.442950 22854 ts_tablet_manager.cc:1950] T 010faaeaf5e049a7893e4bf97b3f73e8 P dd32a35972b347b89ceaaad2455aef5a: Deleting consensus metadata
I20260130 02:59:14.450899 22861 consensus_queue.cc:260] T 670a199ebd03458cb3dae713581506f7 P 789dbb4a27204ffbb68d7ae40f8f5e4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111215cc989f4203a6e1517f2291d66f" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 39751 } } peers { permanent_uuid: "dd32a35972b347b89ceaaad2455aef5a" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 39781 } } peers { permanent_uuid: "789dbb4a27204ffbb68d7ae40f8f5e4a" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 41825 } }
I20260130 02:59:14.452685 22857 ts_tablet_manager.cc:1434] T 670a199ebd03458cb3dae713581506f7 P dd32a35972b347b89ceaaad2455aef5a: Time spent starting tablet: real 0.033s user 0.004s sys 0.002s
I20260130 02:59:14.454420 22861 ts_tablet_manager.cc:1434] T 670a199ebd03458cb3dae713581506f7 P 789dbb4a27204ffbb68d7ae40f8f5e4a: Time spent starting tablet: real 0.022s user 0.006s sys 0.000s
I20260130 02:59:14.460625 22860 tablet_bootstrap.cc:492] T 563d05e06b1f48b4be140219da5c9dbd P 111215cc989f4203a6e1517f2291d66f: No bootstrap required, opened a new log
I20260130 02:59:14.461086 22860 ts_tablet_manager.cc:1403] T 563d05e06b1f48b4be140219da5c9dbd P 111215cc989f4203a6e1517f2291d66f: Time spent bootstrapping tablet: real 0.051s user 0.008s sys 0.002s
I20260130 02:59:14.463721 22860 raft_consensus.cc:359] T 563d05e06b1f48b4be140219da5c9dbd P 111215cc989f4203a6e1517f2291d66f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111215cc989f4203a6e1517f2291d66f" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 39751 } } peers { permanent_uuid: "dd32a35972b347b89ceaaad2455aef5a" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 39781 } } peers { permanent_uuid: "789dbb4a27204ffbb68d7ae40f8f5e4a" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 41825 } }
I20260130 02:59:14.464449 22860 raft_consensus.cc:385] T 563d05e06b1f48b4be140219da5c9dbd P 111215cc989f4203a6e1517f2291d66f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:59:14.464762 22860 raft_consensus.cc:740] T 563d05e06b1f48b4be140219da5c9dbd P 111215cc989f4203a6e1517f2291d66f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 111215cc989f4203a6e1517f2291d66f, State: Initialized, Role: FOLLOWER
I20260130 02:59:14.465430 22860 consensus_queue.cc:260] T 563d05e06b1f48b4be140219da5c9dbd P 111215cc989f4203a6e1517f2291d66f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111215cc989f4203a6e1517f2291d66f" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 39751 } } peers { permanent_uuid: "dd32a35972b347b89ceaaad2455aef5a" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 39781 } } peers { permanent_uuid: "789dbb4a27204ffbb68d7ae40f8f5e4a" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 41825 } }
I20260130 02:59:14.480847 22860 ts_tablet_manager.cc:1434] T 563d05e06b1f48b4be140219da5c9dbd P 111215cc989f4203a6e1517f2291d66f: Time spent starting tablet: real 0.019s user 0.006s sys 0.001s
I20260130 02:59:14.495904 22823 raft_consensus.cc:493] T 670a199ebd03458cb3dae713581506f7 P 789dbb4a27204ffbb68d7ae40f8f5e4a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:59:14.496372 22823 raft_consensus.cc:515] T 670a199ebd03458cb3dae713581506f7 P 789dbb4a27204ffbb68d7ae40f8f5e4a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111215cc989f4203a6e1517f2291d66f" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 39751 } } peers { permanent_uuid: "dd32a35972b347b89ceaaad2455aef5a" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 39781 } } peers { permanent_uuid: "789dbb4a27204ffbb68d7ae40f8f5e4a" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 41825 } }
I20260130 02:59:14.498159 22823 leader_election.cc:290] T 670a199ebd03458cb3dae713581506f7 P 789dbb4a27204ffbb68d7ae40f8f5e4a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 111215cc989f4203a6e1517f2291d66f (127.18.100.65:39751), dd32a35972b347b89ceaaad2455aef5a (127.18.100.66:39781)
I20260130 02:59:14.499707 22391 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "670a199ebd03458cb3dae713581506f7" candidate_uuid: "789dbb4a27204ffbb68d7ae40f8f5e4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd32a35972b347b89ceaaad2455aef5a" is_pre_election: true
I20260130 02:59:14.500293 22391 raft_consensus.cc:2468] T 670a199ebd03458cb3dae713581506f7 P dd32a35972b347b89ceaaad2455aef5a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 789dbb4a27204ffbb68d7ae40f8f5e4a in term 0.
I20260130 02:59:14.501768 22248 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "670a199ebd03458cb3dae713581506f7" candidate_uuid: "789dbb4a27204ffbb68d7ae40f8f5e4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "111215cc989f4203a6e1517f2291d66f" is_pre_election: true
I20260130 02:59:14.502348 22248 raft_consensus.cc:2468] T 670a199ebd03458cb3dae713581506f7 P 111215cc989f4203a6e1517f2291d66f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 789dbb4a27204ffbb68d7ae40f8f5e4a in term 0.
I20260130 02:59:14.503851 22468 leader_election.cc:304] T 670a199ebd03458cb3dae713581506f7 P 789dbb4a27204ffbb68d7ae40f8f5e4a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 111215cc989f4203a6e1517f2291d66f, 789dbb4a27204ffbb68d7ae40f8f5e4a; no voters:
I20260130 02:59:14.504598 22823 raft_consensus.cc:2804] T 670a199ebd03458cb3dae713581506f7 P 789dbb4a27204ffbb68d7ae40f8f5e4a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260130 02:59:14.504901 22823 raft_consensus.cc:493] T 670a199ebd03458cb3dae713581506f7 P 789dbb4a27204ffbb68d7ae40f8f5e4a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260130 02:59:14.505224 22823 raft_consensus.cc:3060] T 670a199ebd03458cb3dae713581506f7 P 789dbb4a27204ffbb68d7ae40f8f5e4a [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:59:14.518350 22823 raft_consensus.cc:515] T 670a199ebd03458cb3dae713581506f7 P 789dbb4a27204ffbb68d7ae40f8f5e4a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111215cc989f4203a6e1517f2291d66f" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 39751 } } peers { permanent_uuid: "dd32a35972b347b89ceaaad2455aef5a" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 39781 } } peers { permanent_uuid: "789dbb4a27204ffbb68d7ae40f8f5e4a" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 41825 } }
I20260130 02:59:14.521221 22248 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "670a199ebd03458cb3dae713581506f7" candidate_uuid: "789dbb4a27204ffbb68d7ae40f8f5e4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "111215cc989f4203a6e1517f2291d66f"
I20260130 02:59:14.521790 22248 raft_consensus.cc:3060] T 670a199ebd03458cb3dae713581506f7 P 111215cc989f4203a6e1517f2291d66f [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:59:14.528575 22248 raft_consensus.cc:2468] T 670a199ebd03458cb3dae713581506f7 P 111215cc989f4203a6e1517f2291d66f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 789dbb4a27204ffbb68d7ae40f8f5e4a in term 1.
I20260130 02:59:14.531774 22391 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "670a199ebd03458cb3dae713581506f7" candidate_uuid: "789dbb4a27204ffbb68d7ae40f8f5e4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd32a35972b347b89ceaaad2455aef5a"
I20260130 02:59:14.532347 22391 raft_consensus.cc:3060] T 670a199ebd03458cb3dae713581506f7 P dd32a35972b347b89ceaaad2455aef5a [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:59:14.540485 22391 raft_consensus.cc:2468] T 670a199ebd03458cb3dae713581506f7 P dd32a35972b347b89ceaaad2455aef5a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 789dbb4a27204ffbb68d7ae40f8f5e4a in term 1.
I20260130 02:59:14.541723 22469 leader_election.cc:304] T 670a199ebd03458cb3dae713581506f7 P 789dbb4a27204ffbb68d7ae40f8f5e4a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 789dbb4a27204ffbb68d7ae40f8f5e4a, dd32a35972b347b89ceaaad2455aef5a; no voters:
I20260130 02:59:14.547222 22823 leader_election.cc:290] T 670a199ebd03458cb3dae713581506f7 P 789dbb4a27204ffbb68d7ae40f8f5e4a [CANDIDATE]: Term 1 election: Requested vote from peers 111215cc989f4203a6e1517f2291d66f (127.18.100.65:39751), dd32a35972b347b89ceaaad2455aef5a (127.18.100.66:39781)
I20260130 02:59:14.549394 22823 raft_consensus.cc:2804] T 670a199ebd03458cb3dae713581506f7 P 789dbb4a27204ffbb68d7ae40f8f5e4a [term 1 FOLLOWER]: Leader election won for term 1
I20260130 02:59:14.555482 22823 raft_consensus.cc:697] T 670a199ebd03458cb3dae713581506f7 P 789dbb4a27204ffbb68d7ae40f8f5e4a [term 1 LEADER]: Becoming Leader. State: Replica: 789dbb4a27204ffbb68d7ae40f8f5e4a, State: Running, Role: LEADER
I20260130 02:59:14.558804 22823 consensus_queue.cc:237] T 670a199ebd03458cb3dae713581506f7 P 789dbb4a27204ffbb68d7ae40f8f5e4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111215cc989f4203a6e1517f2291d66f" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 39751 } } peers { permanent_uuid: "dd32a35972b347b89ceaaad2455aef5a" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 39781 } } peers { permanent_uuid: "789dbb4a27204ffbb68d7ae40f8f5e4a" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 41825 } }
W20260130 02:59:14.636806 22468 connection.cc:537] server connection from 127.0.0.1:53092 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260130 02:59:14.636916 22324 connection.cc:537] server connection from 127.0.0.1:44088 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260130 02:59:14.637466 22324 connection.cc:537] client connection to 127.18.100.125:39157 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260130 02:59:14.637498 22182 connection.cc:537] server connection from 127.0.0.1:53820 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260130 02:59:14.639407 22744 connection.cc:537] server connection from 127.0.0.1:60664 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260130 02:59:14.639839 22595 connection.cc:537] client connection to 127.18.100.125:39157 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260130 02:59:14.641407 22786 negotiation.cc:337] Failed RPC negotiation. Trace:
0130 02:59:14.639597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.100.125:39157 (local address 127.18.100.67:38819)
0130 02:59:14.640499 (+ 902us) negotiation.cc:107] Waiting for socket to connect
0130 02:59:14.640696 (+ 197us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.100.125:39157: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":305}
W20260130 02:59:14.641499 22595 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.18.100.125:39157)
W20260130 02:59:14.642633 22595 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.18.100.125:39157)
W20260130 02:59:14.643363 22182 connection.cc:537] client connection to 127.18.100.125:39157 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260130 02:59:14.644212 22595 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.18.100.125:39157)
W20260130 02:59:14.644896 22595 master_proxy_rpc.cc:203] Re-attempting DeleteTable request to leader Master (127.18.100.125:39157)
I20260130 02:59:14.761833 22868 raft_consensus.cc:493] T 5cc07f1d50624e03b1f5498b12ea680b P 111215cc989f4203a6e1517f2291d66f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:59:14.762358 22868 raft_consensus.cc:515] T 5cc07f1d50624e03b1f5498b12ea680b P 111215cc989f4203a6e1517f2291d66f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111215cc989f4203a6e1517f2291d66f" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 39751 } } peers { permanent_uuid: "dd32a35972b347b89ceaaad2455aef5a" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 39781 } } peers { permanent_uuid: "789dbb4a27204ffbb68d7ae40f8f5e4a" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 41825 } }
I20260130 02:59:14.764344 22868 leader_election.cc:290] T 5cc07f1d50624e03b1f5498b12ea680b P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dd32a35972b347b89ceaaad2455aef5a (127.18.100.66:39781), 789dbb4a27204ffbb68d7ae40f8f5e4a (127.18.100.67:41825)
W20260130 02:59:14.766081 22180 leader_election.cc:343] T 5cc07f1d50624e03b1f5498b12ea680b P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dd32a35972b347b89ceaaad2455aef5a (127.18.100.66:39781): Not found: Tablet not found: 5cc07f1d50624e03b1f5498b12ea680b
W20260130 02:59:14.766547 22179 leader_election.cc:343] T 5cc07f1d50624e03b1f5498b12ea680b P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 789dbb4a27204ffbb68d7ae40f8f5e4a (127.18.100.67:41825): Not found: Tablet not found: 5cc07f1d50624e03b1f5498b12ea680b
I20260130 02:59:14.766969 22179 leader_election.cc:304] T 5cc07f1d50624e03b1f5498b12ea680b P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 111215cc989f4203a6e1517f2291d66f; no voters: 789dbb4a27204ffbb68d7ae40f8f5e4a, dd32a35972b347b89ceaaad2455aef5a
I20260130 02:59:14.767998 22868 raft_consensus.cc:2749] T 5cc07f1d50624e03b1f5498b12ea680b P 111215cc989f4203a6e1517f2291d66f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260130 02:59:14.787914 22868 raft_consensus.cc:493] T 563d05e06b1f48b4be140219da5c9dbd P 111215cc989f4203a6e1517f2291d66f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:59:14.788435 22868 raft_consensus.cc:515] T 563d05e06b1f48b4be140219da5c9dbd P 111215cc989f4203a6e1517f2291d66f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111215cc989f4203a6e1517f2291d66f" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 39751 } } peers { permanent_uuid: "dd32a35972b347b89ceaaad2455aef5a" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 39781 } } peers { permanent_uuid: "789dbb4a27204ffbb68d7ae40f8f5e4a" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 41825 } }
I20260130 02:59:14.789927 22868 leader_election.cc:290] T 563d05e06b1f48b4be140219da5c9dbd P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dd32a35972b347b89ceaaad2455aef5a (127.18.100.66:39781), 789dbb4a27204ffbb68d7ae40f8f5e4a (127.18.100.67:41825)
W20260130 02:59:14.791332 22180 leader_election.cc:343] T 563d05e06b1f48b4be140219da5c9dbd P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dd32a35972b347b89ceaaad2455aef5a (127.18.100.66:39781): Not found: Tablet not found: 563d05e06b1f48b4be140219da5c9dbd
W20260130 02:59:14.791769 22179 leader_election.cc:343] T 563d05e06b1f48b4be140219da5c9dbd P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 789dbb4a27204ffbb68d7ae40f8f5e4a (127.18.100.67:41825): Not found: Tablet not found: 563d05e06b1f48b4be140219da5c9dbd
I20260130 02:59:14.792160 22179 leader_election.cc:304] T 563d05e06b1f48b4be140219da5c9dbd P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 111215cc989f4203a6e1517f2291d66f; no voters: 789dbb4a27204ffbb68d7ae40f8f5e4a, dd32a35972b347b89ceaaad2455aef5a
I20260130 02:59:14.792805 22868 raft_consensus.cc:2749] T 563d05e06b1f48b4be140219da5c9dbd P 111215cc989f4203a6e1517f2291d66f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260130 02:59:14.813975 22898 raft_consensus.cc:493] T 00000000000000000000000000000000 P 342be83f20a648fd849c7d23454fba08 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader b01057b6c5ff4fc4b07253801881fe13)
I20260130 02:59:14.814507 22898 raft_consensus.cc:515] T 00000000000000000000000000000000 P 342be83f20a648fd849c7d23454fba08 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22dedcf69e00438999d3a5fedde57324" member_type: VOTER last_known_addr { host: "127.18.100.126" port: 42077 } } peers { permanent_uuid: "b01057b6c5ff4fc4b07253801881fe13" member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39157 } } peers { permanent_uuid: "342be83f20a648fd849c7d23454fba08" member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34987 } }
I20260130 02:59:14.816078 22898 leader_election.cc:290] T 00000000000000000000000000000000 P 342be83f20a648fd849c7d23454fba08 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 22dedcf69e00438999d3a5fedde57324 (127.18.100.126:42077), b01057b6c5ff4fc4b07253801881fe13 (127.18.100.125:39157)
W20260130 02:59:14.822494 22743 leader_election.cc:336] T 00000000000000000000000000000000 P 342be83f20a648fd849c7d23454fba08 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 22dedcf69e00438999d3a5fedde57324 (127.18.100.126:42077): Network error: Client connection negotiation failed: client connection to 127.18.100.126:42077: connect: Connection refused (error 111)
W20260130 02:59:14.823206 22744 leader_election.cc:336] T 00000000000000000000000000000000 P 342be83f20a648fd849c7d23454fba08 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer b01057b6c5ff4fc4b07253801881fe13 (127.18.100.125:39157): Network error: Client connection negotiation failed: client connection to 127.18.100.125:39157: connect: Connection refused (error 111)
I20260130 02:59:14.823618 22744 leader_election.cc:304] T 00000000000000000000000000000000 P 342be83f20a648fd849c7d23454fba08 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 342be83f20a648fd849c7d23454fba08; no voters: 22dedcf69e00438999d3a5fedde57324, b01057b6c5ff4fc4b07253801881fe13
I20260130 02:59:14.824316 22898 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 342be83f20a648fd849c7d23454fba08 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260130 02:59:15.194928 22179 consensus_peers.cc:597] T 476f8fd240bc48f1a9c47a0351ad5d27 P 111215cc989f4203a6e1517f2291d66f -> Peer 789dbb4a27204ffbb68d7ae40f8f5e4a (127.18.100.67:41825): Couldn't send request to peer 789dbb4a27204ffbb68d7ae40f8f5e4a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 476f8fd240bc48f1a9c47a0351ad5d27. This is attempt 1: this message will repeat every 5th retry.
I20260130 02:59:15.367072 22902 raft_consensus.cc:493] T 00000000000000000000000000000000 P 342be83f20a648fd849c7d23454fba08 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader b01057b6c5ff4fc4b07253801881fe13)
I20260130 02:59:15.367688 22902 raft_consensus.cc:515] T 00000000000000000000000000000000 P 342be83f20a648fd849c7d23454fba08 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22dedcf69e00438999d3a5fedde57324" member_type: VOTER last_known_addr { host: "127.18.100.126" port: 42077 } } peers { permanent_uuid: "b01057b6c5ff4fc4b07253801881fe13" member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39157 } } peers { permanent_uuid: "342be83f20a648fd849c7d23454fba08" member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34987 } }
I20260130 02:59:15.375702 22902 leader_election.cc:290] T 00000000000000000000000000000000 P 342be83f20a648fd849c7d23454fba08 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 22dedcf69e00438999d3a5fedde57324 (127.18.100.126:42077), b01057b6c5ff4fc4b07253801881fe13 (127.18.100.125:39157)
W20260130 02:59:15.391875 22744 leader_election.cc:336] T 00000000000000000000000000000000 P 342be83f20a648fd849c7d23454fba08 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer b01057b6c5ff4fc4b07253801881fe13 (127.18.100.125:39157): Network error: Client connection negotiation failed: client connection to 127.18.100.125:39157: connect: Connection refused (error 111)
W20260130 02:59:15.392421 22743 leader_election.cc:336] T 00000000000000000000000000000000 P 342be83f20a648fd849c7d23454fba08 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 22dedcf69e00438999d3a5fedde57324 (127.18.100.126:42077): Network error: Client connection negotiation failed: client connection to 127.18.100.126:42077: connect: Connection refused (error 111)
I20260130 02:59:15.392768 22743 leader_election.cc:304] T 00000000000000000000000000000000 P 342be83f20a648fd849c7d23454fba08 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 342be83f20a648fd849c7d23454fba08; no voters: 22dedcf69e00438999d3a5fedde57324, b01057b6c5ff4fc4b07253801881fe13
I20260130 02:59:15.393328 22902 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 342be83f20a648fd849c7d23454fba08 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260130 02:59:15.440902 22868 consensus_queue.cc:1048] T 476f8fd240bc48f1a9c47a0351ad5d27 P 111215cc989f4203a6e1517f2291d66f [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd32a35972b347b89ceaaad2455aef5a" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 39781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260130 02:59:15.534988 22913 consensus_queue.cc:1048] T 670a199ebd03458cb3dae713581506f7 P 789dbb4a27204ffbb68d7ae40f8f5e4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "111215cc989f4203a6e1517f2291d66f" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 39751 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260130 02:59:15.566891 22920 consensus_queue.cc:1048] T 670a199ebd03458cb3dae713581506f7 P 789dbb4a27204ffbb68d7ae40f8f5e4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd32a35972b347b89ceaaad2455aef5a" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 39781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260130 02:59:15.896457 18833 external_mini_cluster.cc:1366] Running /tmp/dist-test-tasktGe2wI/build/tsan/bin/kudu
/tmp/dist-test-tasktGe2wI/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1769741859011789-18833-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1769741859011789-18833-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1769741859011789-18833-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1769741859011789-18833-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.100.126:42077
--webserver_interface=127.18.100.126
--webserver_port=44889
--builtin_ntp_servers=127.18.100.84:33971
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.100.126:42077,127.18.100.125:39157,127.18.100.124:34987
--hive_metastore_uris=thrift://127.0.0.1:42185
--hive_metastore_notification_log_poll_period_seconds=1
--enable_metadata_cleanup_for_deleted_tables_and_tablets=true
--catalog_manager_bg_task_wait_ms=10
--metadata_for_deleted_table_and_tablet_reserved_secs=0
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20260130 02:59:15.951987 22924 raft_consensus.cc:493] T 00000000000000000000000000000000 P 342be83f20a648fd849c7d23454fba08 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader b01057b6c5ff4fc4b07253801881fe13)
I20260130 02:59:15.952554 22924 raft_consensus.cc:515] T 00000000000000000000000000000000 P 342be83f20a648fd849c7d23454fba08 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22dedcf69e00438999d3a5fedde57324" member_type: VOTER last_known_addr { host: "127.18.100.126" port: 42077 } } peers { permanent_uuid: "b01057b6c5ff4fc4b07253801881fe13" member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39157 } } peers { permanent_uuid: "342be83f20a648fd849c7d23454fba08" member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34987 } }
I20260130 02:59:15.954058 22924 leader_election.cc:290] T 00000000000000000000000000000000 P 342be83f20a648fd849c7d23454fba08 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 22dedcf69e00438999d3a5fedde57324 (127.18.100.126:42077), b01057b6c5ff4fc4b07253801881fe13 (127.18.100.125:39157)
W20260130 02:59:15.959096 22743 leader_election.cc:336] T 00000000000000000000000000000000 P 342be83f20a648fd849c7d23454fba08 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 22dedcf69e00438999d3a5fedde57324 (127.18.100.126:42077): Network error: Client connection negotiation failed: client connection to 127.18.100.126:42077: connect: Connection refused (error 111)
W20260130 02:59:15.960999 22744 leader_election.cc:336] T 00000000000000000000000000000000 P 342be83f20a648fd849c7d23454fba08 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer b01057b6c5ff4fc4b07253801881fe13 (127.18.100.125:39157): Network error: Client connection negotiation failed: client connection to 127.18.100.125:39157: connect: Connection refused (error 111)
I20260130 02:59:15.961405 22744 leader_election.cc:304] T 00000000000000000000000000000000 P 342be83f20a648fd849c7d23454fba08 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 342be83f20a648fd849c7d23454fba08; no voters: 22dedcf69e00438999d3a5fedde57324, b01057b6c5ff4fc4b07253801881fe13
I20260130 02:59:15.962100 22924 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 342be83f20a648fd849c7d23454fba08 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260130 02:59:16.222291 22923 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260130 02:59:16.222931 22923 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260130 02:59:16.223464 22923 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260130 02:59:16.263652 22923 flags.cc:425] Enabled experimental flag: --enable_metadata_cleanup_for_deleted_tables_and_tablets=true
W20260130 02:59:16.263902 22923 flags.cc:425] Enabled experimental flag: --metadata_for_deleted_table_and_tablet_reserved_secs=0
W20260130 02:59:16.264186 22923 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260130 02:59:16.264442 22923 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260130 02:59:16.264670 22923 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260130 02:59:16.264899 22923 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260130 02:59:16.304320 22923 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.100.84:33971
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1769741859011789-18833-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1769741859011789-18833-0/minicluster-data/master-0/wal
--hive_metastore_uris=thrift://127.0.0.1:42185
--catalog_manager_bg_task_wait_ms=10
--enable_metadata_cleanup_for_deleted_tables_and_tablets=true
--metadata_for_deleted_table_and_tablet_reserved_secs=0
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.18.100.126:42077,127.18.100.125:39157,127.18.100.124:34987
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.100.126:42077
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1769741859011789-18833-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.100.126
--webserver_port=44889
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1769741859011789-18833-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type FASTDEBUG
built by None at 30 Jan 2026 02:43:21 UTC on 5fd53c4cbb9d
build id 10257
TSAN enabled
I20260130 02:59:16.306030 22923 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260130 02:59:16.308370 22923 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260130 02:59:16.319941 22938 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260130 02:59:16.519658 22944 raft_consensus.cc:493] T 00000000000000000000000000000000 P 342be83f20a648fd849c7d23454fba08 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader b01057b6c5ff4fc4b07253801881fe13)
I20260130 02:59:16.520056 22944 raft_consensus.cc:515] T 00000000000000000000000000000000 P 342be83f20a648fd849c7d23454fba08 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22dedcf69e00438999d3a5fedde57324" member_type: VOTER last_known_addr { host: "127.18.100.126" port: 42077 } } peers { permanent_uuid: "b01057b6c5ff4fc4b07253801881fe13" member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39157 } } peers { permanent_uuid: "342be83f20a648fd849c7d23454fba08" member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34987 } }
I20260130 02:59:16.521442 22944 leader_election.cc:290] T 00000000000000000000000000000000 P 342be83f20a648fd849c7d23454fba08 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 22dedcf69e00438999d3a5fedde57324 (127.18.100.126:42077), b01057b6c5ff4fc4b07253801881fe13 (127.18.100.125:39157)
W20260130 02:59:16.526516 22743 leader_election.cc:336] T 00000000000000000000000000000000 P 342be83f20a648fd849c7d23454fba08 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 22dedcf69e00438999d3a5fedde57324 (127.18.100.126:42077): Network error: Client connection negotiation failed: client connection to 127.18.100.126:42077: connect: Connection refused (error 111)
W20260130 02:59:16.527027 22744 leader_election.cc:336] T 00000000000000000000000000000000 P 342be83f20a648fd849c7d23454fba08 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer b01057b6c5ff4fc4b07253801881fe13 (127.18.100.125:39157): Network error: Client connection negotiation failed: client connection to 127.18.100.125:39157: connect: Connection refused (error 111)
I20260130 02:59:16.527326 22744 leader_election.cc:304] T 00000000000000000000000000000000 P 342be83f20a648fd849c7d23454fba08 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 342be83f20a648fd849c7d23454fba08; no voters: 22dedcf69e00438999d3a5fedde57324, b01057b6c5ff4fc4b07253801881fe13
I20260130 02:59:16.527884 22944 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 342be83f20a648fd849c7d23454fba08 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260130 02:59:16.320523 22939 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260130 02:59:16.324056 22941 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260130 02:59:17.481650 22940 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20260130 02:59:17.481762 22923 server_base.cc:1055] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20260130 02:59:17.486074 22923 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260130 02:59:17.489077 22923 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260130 02:59:17.490523 22923 hybrid_clock.cc:648] HybridClock initialized: now 1769741957490484 us; error 44 us; skew 500 ppm
I20260130 02:59:17.491355 22923 server_base.cc:860] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260130 02:59:17.497918 22923 webserver.cc:492] Webserver started at http://127.18.100.126:44889/ using document root <none> and password file <none>
I20260130 02:59:17.498916 22923 fs_manager.cc:362] Metadata directory not provided
I20260130 02:59:17.499156 22923 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260130 02:59:17.506770 22923 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.007s sys 0.001s
I20260130 02:59:17.511651 22966 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260130 02:59:17.512764 22923 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.002s
I20260130 02:59:17.513083 22923 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1769741859011789-18833-0/minicluster-data/master-0/data,/tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1769741859011789-18833-0/minicluster-data/master-0/wal
uuid: "22dedcf69e00438999d3a5fedde57324"
format_stamp: "Formatted at 2026-01-30 02:58:56 on dist-test-slave-mn07"
I20260130 02:59:17.515107 22923 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1769741859011789-18833-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1769741859011789-18833-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1769741859011789-18833-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260130 02:59:17.579938 22923 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260130 02:59:17.581377 22923 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260130 02:59:17.581779 22923 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260130 02:59:17.661016 22923 rpc_server.cc:307] RPC server started. Bound to: 127.18.100.126:42077
I20260130 02:59:17.661087 23023 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.100.126:42077 every 8 connection(s)
I20260130 02:59:17.663949 22923 server_base.cc:1192] Dumped server information to /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1769741859011789-18833-0/minicluster-data/master-0/data/info.pb
I20260130 02:59:17.670737 18833 external_mini_cluster.cc:1428] Started /tmp/dist-test-tasktGe2wI/build/tsan/bin/kudu as pid 22923
I20260130 02:59:17.696650 23024 sys_catalog.cc:263] Verifying existing consensus state
I20260130 02:59:17.705062 23024 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324: Bootstrap starting.
I20260130 02:59:17.707404 22582 heartbeater.cc:344] Connected to a master server at 127.18.100.126:42077
I20260130 02:59:17.724792 23024 log.cc:826] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324: Log is configured to *not* fsync() on all Append() calls
I20260130 02:59:17.768090 23024 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324: Bootstrap replayed 1/1 log segments. Stats: ops{read=16 overwritten=0 applied=16 ignored=0} inserts{seen=12 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260130 02:59:17.768987 23024 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324: Bootstrap complete.
I20260130 02:59:17.788976 23024 raft_consensus.cc:359] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22dedcf69e00438999d3a5fedde57324" member_type: VOTER last_known_addr { host: "127.18.100.126" port: 42077 } } peers { permanent_uuid: "b01057b6c5ff4fc4b07253801881fe13" member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39157 } } peers { permanent_uuid: "342be83f20a648fd849c7d23454fba08" member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34987 } }
I20260130 02:59:17.791507 23024 raft_consensus.cc:740] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22dedcf69e00438999d3a5fedde57324, State: Initialized, Role: FOLLOWER
I20260130 02:59:17.792292 23024 consensus_queue.cc:260] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 16, Last appended: 2.16, Last appended by leader: 16, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22dedcf69e00438999d3a5fedde57324" member_type: VOTER last_known_addr { host: "127.18.100.126" port: 42077 } } peers { permanent_uuid: "b01057b6c5ff4fc4b07253801881fe13" member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39157 } } peers { permanent_uuid: "342be83f20a648fd849c7d23454fba08" member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34987 } }
I20260130 02:59:17.794469 23032 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22dedcf69e00438999d3a5fedde57324" member_type: VOTER last_known_addr { host: "127.18.100.126" port: 42077 } } peers { permanent_uuid: "b01057b6c5ff4fc4b07253801881fe13" member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39157 } } peers { permanent_uuid: "342be83f20a648fd849c7d23454fba08" member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34987 } } }
I20260130 02:59:17.795135 23032 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [sys.catalog]: This master's current role is: FOLLOWER
I20260130 02:59:17.796027 23024 sys_catalog.cc:565] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [sys.catalog]: configured and running, proceeding with master startup.
I20260130 02:59:17.829918 23047 catalog_manager.cc:1269] Loaded cluster ID: 40eb42871f464e95bd641ceacb567979
I20260130 02:59:17.830298 23047 catalog_manager.cc:1562] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324: loading cluster ID for follower catalog manager: success
I20260130 02:59:17.839316 23047 catalog_manager.cc:1584] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324: acquiring CA information for follower catalog manager: success
I20260130 02:59:17.845747 23047 catalog_manager.cc:1612] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260130 02:59:17.836447 22923 master_runner.cc:186] Error getting master registration for 127.18.100.125:39157: Network error: Client connection negotiation failed: client connection to 127.18.100.125:39157: connect: Connection refused (error 111),
I20260130 02:59:17.862167 22923 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260130 02:59:17.987920 23050 raft_consensus.cc:493] T 563d05e06b1f48b4be140219da5c9dbd P 111215cc989f4203a6e1517f2291d66f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:59:17.988313 23050 raft_consensus.cc:515] T 563d05e06b1f48b4be140219da5c9dbd P 111215cc989f4203a6e1517f2291d66f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111215cc989f4203a6e1517f2291d66f" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 39751 } } peers { permanent_uuid: "dd32a35972b347b89ceaaad2455aef5a" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 39781 } } peers { permanent_uuid: "789dbb4a27204ffbb68d7ae40f8f5e4a" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 41825 } }
I20260130 02:59:17.991251 23050 leader_election.cc:290] T 563d05e06b1f48b4be140219da5c9dbd P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dd32a35972b347b89ceaaad2455aef5a (127.18.100.66:39781), 789dbb4a27204ffbb68d7ae40f8f5e4a (127.18.100.67:41825)
W20260130 02:59:17.992017 22180 leader_election.cc:343] T 563d05e06b1f48b4be140219da5c9dbd P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dd32a35972b347b89ceaaad2455aef5a (127.18.100.66:39781): Not found: Tablet not found: 563d05e06b1f48b4be140219da5c9dbd
W20260130 02:59:17.993304 22179 leader_election.cc:343] T 563d05e06b1f48b4be140219da5c9dbd P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 789dbb4a27204ffbb68d7ae40f8f5e4a (127.18.100.67:41825): Not found: Tablet not found: 563d05e06b1f48b4be140219da5c9dbd
I20260130 02:59:17.993753 22179 leader_election.cc:304] T 563d05e06b1f48b4be140219da5c9dbd P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 111215cc989f4203a6e1517f2291d66f; no voters: 789dbb4a27204ffbb68d7ae40f8f5e4a, dd32a35972b347b89ceaaad2455aef5a
I20260130 02:59:17.994917 23050 raft_consensus.cc:2749] T 563d05e06b1f48b4be140219da5c9dbd P 111215cc989f4203a6e1517f2291d66f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260130 02:59:18.014106 22646 master_runner.cc:186] Error getting master registration for 127.18.100.125:39157: Network error: Client connection negotiation failed: client connection to 127.18.100.125:39157: connect: Connection refused (error 111),
I20260130 02:59:18.014541 22646 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260130 02:59:18.041277 22298 heartbeater.cc:344] Connected to a master server at 127.18.100.126:42077
I20260130 02:59:18.047600 22988 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "111215cc989f4203a6e1517f2291d66f" instance_seqno: 1769741945288540) as {username='slave'} at 127.18.100.65:56087; Asking this server to re-register.
I20260130 02:59:18.048899 22298 heartbeater.cc:461] Registering TS with master...
I20260130 02:59:18.052078 22988 ts_manager.cc:194] Registered new tserver with Master: 111215cc989f4203a6e1517f2291d66f (127.18.100.65:39751)
I20260130 02:59:18.131207 22439 heartbeater.cc:344] Connected to a master server at 127.18.100.126:42077
I20260130 02:59:18.135267 22988 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "dd32a35972b347b89ceaaad2455aef5a" instance_seqno: 1769741947198445) as {username='slave'} at 127.18.100.66:41785; Asking this server to re-register.
I20260130 02:59:18.136675 22439 heartbeater.cc:461] Registering TS with master...
I20260130 02:59:18.140046 22988 ts_manager.cc:194] Registered new tserver with Master: dd32a35972b347b89ceaaad2455aef5a (127.18.100.66:39781)
I20260130 02:59:18.215682 22988 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "789dbb4a27204ffbb68d7ae40f8f5e4a" instance_seqno: 1769741949079638) as {username='slave'} at 127.18.100.67:51657; Asking this server to re-register.
I20260130 02:59:18.216964 22582 heartbeater.cc:461] Registering TS with master...
I20260130 02:59:18.219911 22988 ts_manager.cc:194] Registered new tserver with Master: 789dbb4a27204ffbb68d7ae40f8f5e4a (127.18.100.67:41825)
I20260130 02:59:18.240590 23032 raft_consensus.cc:493] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:59:18.241133 23032 raft_consensus.cc:515] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22dedcf69e00438999d3a5fedde57324" member_type: VOTER last_known_addr { host: "127.18.100.126" port: 42077 } } peers { permanent_uuid: "b01057b6c5ff4fc4b07253801881fe13" member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39157 } } peers { permanent_uuid: "342be83f20a648fd849c7d23454fba08" member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34987 } }
I20260130 02:59:18.244119 23032 leader_election.cc:290] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b01057b6c5ff4fc4b07253801881fe13 (127.18.100.125:39157), 342be83f20a648fd849c7d23454fba08 (127.18.100.124:34987)
W20260130 02:59:18.246924 22970 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.18.100.125:39157: connect: Connection refused (error 111)
W20260130 02:59:18.261508 22970 leader_election.cc:336] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b01057b6c5ff4fc4b07253801881fe13 (127.18.100.125:39157): Network error: Client connection negotiation failed: client connection to 127.18.100.125:39157: connect: Connection refused (error 111)
I20260130 02:59:18.272017 22768 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "22dedcf69e00438999d3a5fedde57324" candidate_term: 3 candidate_status { last_received { term: 2 index: 16 } } ignore_live_leader: false dest_uuid: "342be83f20a648fd849c7d23454fba08" is_pre_election: true
I20260130 02:59:18.272845 22768 raft_consensus.cc:2393] T 00000000000000000000000000000000 P 342be83f20a648fd849c7d23454fba08 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 22dedcf69e00438999d3a5fedde57324 in current term 3: Already voted for candidate b01057b6c5ff4fc4b07253801881fe13 in this term.
I20260130 02:59:18.274121 22968 leader_election.cc:304] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22dedcf69e00438999d3a5fedde57324; no voters: 342be83f20a648fd849c7d23454fba08, b01057b6c5ff4fc4b07253801881fe13
I20260130 02:59:18.275056 23032 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [term 2 FOLLOWER]: Advancing to term 3
I20260130 02:59:18.284475 23032 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260130 02:59:18.436986 23060 raft_consensus.cc:493] T 00000000000000000000000000000000 P 342be83f20a648fd849c7d23454fba08 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader b01057b6c5ff4fc4b07253801881fe13)
I20260130 02:59:18.437419 23060 raft_consensus.cc:515] T 00000000000000000000000000000000 P 342be83f20a648fd849c7d23454fba08 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22dedcf69e00438999d3a5fedde57324" member_type: VOTER last_known_addr { host: "127.18.100.126" port: 42077 } } peers { permanent_uuid: "b01057b6c5ff4fc4b07253801881fe13" member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39157 } } peers { permanent_uuid: "342be83f20a648fd849c7d23454fba08" member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34987 } }
I20260130 02:59:18.438936 23060 leader_election.cc:290] T 00000000000000000000000000000000 P 342be83f20a648fd849c7d23454fba08 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 22dedcf69e00438999d3a5fedde57324 (127.18.100.126:42077), b01057b6c5ff4fc4b07253801881fe13 (127.18.100.125:39157)
I20260130 02:59:18.440275 22999 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "342be83f20a648fd849c7d23454fba08" candidate_term: 4 candidate_status { last_received { term: 3 index: 30 } } ignore_live_leader: false dest_uuid: "22dedcf69e00438999d3a5fedde57324" is_pre_election: true
W20260130 02:59:18.440516 22744 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.18.100.125:39157: connect: Connection refused (error 111) [suppressed 8 similar messages]
I20260130 02:59:18.441082 22999 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 342be83f20a648fd849c7d23454fba08 in term 3.
I20260130 02:59:18.442287 22743 leader_election.cc:304] T 00000000000000000000000000000000 P 342be83f20a648fd849c7d23454fba08 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22dedcf69e00438999d3a5fedde57324, 342be83f20a648fd849c7d23454fba08; no voters:
I20260130 02:59:18.443096 23060 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 342be83f20a648fd849c7d23454fba08 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260130 02:59:18.443429 23060 raft_consensus.cc:493] T 00000000000000000000000000000000 P 342be83f20a648fd849c7d23454fba08 [term 3 FOLLOWER]: Starting leader election (detected failure of leader b01057b6c5ff4fc4b07253801881fe13)
I20260130 02:59:18.443703 23060 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 342be83f20a648fd849c7d23454fba08 [term 3 FOLLOWER]: Advancing to term 4
W20260130 02:59:18.444830 22744 leader_election.cc:336] T 00000000000000000000000000000000 P 342be83f20a648fd849c7d23454fba08 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer b01057b6c5ff4fc4b07253801881fe13 (127.18.100.125:39157): Network error: Client connection negotiation failed: client connection to 127.18.100.125:39157: connect: Connection refused (error 111)
I20260130 02:59:18.448619 23060 raft_consensus.cc:515] T 00000000000000000000000000000000 P 342be83f20a648fd849c7d23454fba08 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22dedcf69e00438999d3a5fedde57324" member_type: VOTER last_known_addr { host: "127.18.100.126" port: 42077 } } peers { permanent_uuid: "b01057b6c5ff4fc4b07253801881fe13" member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39157 } } peers { permanent_uuid: "342be83f20a648fd849c7d23454fba08" member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34987 } }
I20260130 02:59:18.450017 23060 leader_election.cc:290] T 00000000000000000000000000000000 P 342be83f20a648fd849c7d23454fba08 [CANDIDATE]: Term 4 election: Requested vote from peers 22dedcf69e00438999d3a5fedde57324 (127.18.100.126:42077), b01057b6c5ff4fc4b07253801881fe13 (127.18.100.125:39157)
I20260130 02:59:18.450979 22999 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "342be83f20a648fd849c7d23454fba08" candidate_term: 4 candidate_status { last_received { term: 3 index: 30 } } ignore_live_leader: false dest_uuid: "22dedcf69e00438999d3a5fedde57324"
I20260130 02:59:18.451563 22999 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [term 3 FOLLOWER]: Advancing to term 4
W20260130 02:59:18.453817 22744 leader_election.cc:336] T 00000000000000000000000000000000 P 342be83f20a648fd849c7d23454fba08 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer b01057b6c5ff4fc4b07253801881fe13 (127.18.100.125:39157): Network error: Client connection negotiation failed: client connection to 127.18.100.125:39157: connect: Connection refused (error 111)
I20260130 02:59:18.458178 22999 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 342be83f20a648fd849c7d23454fba08 in term 4.
I20260130 02:59:18.460247 22743 leader_election.cc:304] T 00000000000000000000000000000000 P 342be83f20a648fd849c7d23454fba08 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22dedcf69e00438999d3a5fedde57324, 342be83f20a648fd849c7d23454fba08; no voters: b01057b6c5ff4fc4b07253801881fe13
I20260130 02:59:18.461045 23060 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 342be83f20a648fd849c7d23454fba08 [term 4 FOLLOWER]: Leader election won for term 4
I20260130 02:59:18.462194 23060 raft_consensus.cc:697] T 00000000000000000000000000000000 P 342be83f20a648fd849c7d23454fba08 [term 4 LEADER]: Becoming Leader. State: Replica: 342be83f20a648fd849c7d23454fba08, State: Running, Role: LEADER
I20260130 02:59:18.463207 23060 consensus_queue.cc:237] T 00000000000000000000000000000000 P 342be83f20a648fd849c7d23454fba08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 3.30, Last appended by leader: 30, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22dedcf69e00438999d3a5fedde57324" member_type: VOTER last_known_addr { host: "127.18.100.126" port: 42077 } } peers { permanent_uuid: "b01057b6c5ff4fc4b07253801881fe13" member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39157 } } peers { permanent_uuid: "342be83f20a648fd849c7d23454fba08" member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34987 } }
I20260130 02:59:18.468539 23062 sys_catalog.cc:455] T 00000000000000000000000000000000 P 342be83f20a648fd849c7d23454fba08 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 342be83f20a648fd849c7d23454fba08. Latest consensus state: current_term: 4 leader_uuid: "342be83f20a648fd849c7d23454fba08" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22dedcf69e00438999d3a5fedde57324" member_type: VOTER last_known_addr { host: "127.18.100.126" port: 42077 } } peers { permanent_uuid: "b01057b6c5ff4fc4b07253801881fe13" member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39157 } } peers { permanent_uuid: "342be83f20a648fd849c7d23454fba08" member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34987 } } }
I20260130 02:59:18.469314 23062 sys_catalog.cc:458] T 00000000000000000000000000000000 P 342be83f20a648fd849c7d23454fba08 [sys.catalog]: This master's current role is: LEADER
I20260130 02:59:18.470958 23064 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260130 02:59:18.477100 23064 catalog_manager.cc:679] Loaded metadata for table default.table_e3b3683b36e144c2b33dd88b292bcb23 [id=318c872fbfef4f93b4c280fd078cabd9]
I20260130 02:59:18.478417 23064 catalog_manager.cc:679] Loaded metadata for table default.table_d6f2cd9cd3b54168b14f1cacabbc74f0 [id=db113a9d4bf545faa72c7c94427d6dba]
I20260130 02:59:18.479557 23064 catalog_manager.cc:679] Loaded metadata for table default.table_62e153a4e7844d97863f32a6e87da70b [id=dee95603eab3439b81355a006bc9f376]
I20260130 02:59:18.485212 23064 tablet_loader.cc:96] loaded metadata for tablet 563d05e06b1f48b4be140219da5c9dbd (table default.table_62e153a4e7844d97863f32a6e87da70b [id=dee95603eab3439b81355a006bc9f376])
I20260130 02:59:18.486119 23064 tablet_loader.cc:96] loaded metadata for tablet 670a199ebd03458cb3dae713581506f7 (table default.table_d6f2cd9cd3b54168b14f1cacabbc74f0 [id=db113a9d4bf545faa72c7c94427d6dba])
I20260130 02:59:18.487447 23064 tablet_loader.cc:96] loaded metadata for tablet ce91a17790324cd884a0521ce67dde67 (table default.table_e3b3683b36e144c2b33dd88b292bcb23 [id=318c872fbfef4f93b4c280fd078cabd9])
I20260130 02:59:18.488487 23064 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260130 02:59:18.492141 23064 catalog_manager.cc:1269] Loaded cluster ID: 40eb42871f464e95bd641ceacb567979
I20260130 02:59:18.492373 23064 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260130 02:59:18.496310 23064 catalog_manager.cc:1514] Loading token signing keys...
I20260130 02:59:18.499922 23064 catalog_manager.cc:6038] T 00000000000000000000000000000000 P 342be83f20a648fd849c7d23454fba08: Loaded TSK: 0
I20260130 02:59:18.500914 23064 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260130 02:59:18.504725 23064 catalog_manager.cc:1537] Loading latest processed Hive Metastore notification log event ID...
I20260130 02:59:18.506843 23050 raft_consensus.cc:493] T 5cc07f1d50624e03b1f5498b12ea680b P 111215cc989f4203a6e1517f2291d66f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:59:18.507360 23050 raft_consensus.cc:515] T 5cc07f1d50624e03b1f5498b12ea680b P 111215cc989f4203a6e1517f2291d66f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111215cc989f4203a6e1517f2291d66f" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 39751 } } peers { permanent_uuid: "dd32a35972b347b89ceaaad2455aef5a" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 39781 } } peers { permanent_uuid: "789dbb4a27204ffbb68d7ae40f8f5e4a" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 41825 } }
I20260130 02:59:18.508349 23064 catalog_manager.cc:5876] Last processed Hive Metastore notification event ID: -1
I20260130 02:59:18.509246 23050 leader_election.cc:290] T 5cc07f1d50624e03b1f5498b12ea680b P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dd32a35972b347b89ceaaad2455aef5a (127.18.100.66:39781), 789dbb4a27204ffbb68d7ae40f8f5e4a (127.18.100.67:41825)
W20260130 02:59:18.510987 22180 leader_election.cc:343] T 5cc07f1d50624e03b1f5498b12ea680b P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dd32a35972b347b89ceaaad2455aef5a (127.18.100.66:39781): Not found: Tablet not found: 5cc07f1d50624e03b1f5498b12ea680b
W20260130 02:59:18.511783 22179 leader_election.cc:343] T 5cc07f1d50624e03b1f5498b12ea680b P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 789dbb4a27204ffbb68d7ae40f8f5e4a (127.18.100.67:41825): Not found: Tablet not found: 5cc07f1d50624e03b1f5498b12ea680b
I20260130 02:59:18.512176 22179 leader_election.cc:304] T 5cc07f1d50624e03b1f5498b12ea680b P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 111215cc989f4203a6e1517f2291d66f; no voters: 789dbb4a27204ffbb68d7ae40f8f5e4a, dd32a35972b347b89ceaaad2455aef5a
I20260130 02:59:18.512869 23050 raft_consensus.cc:2749] T 5cc07f1d50624e03b1f5498b12ea680b P 111215cc989f4203a6e1517f2291d66f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260130 02:59:18.521790 22999 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [term 4 FOLLOWER]: Refusing update from remote peer 342be83f20a648fd849c7d23454fba08: Log matching property violated. Preceding OpId in replica: term: 2 index: 16. Preceding OpId from leader: term: 4 index: 32. (index mismatch)
I20260130 02:59:18.523099 23062 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 342be83f20a648fd849c7d23454fba08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22dedcf69e00438999d3a5fedde57324" member_type: VOTER last_known_addr { host: "127.18.100.126" port: 42077 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 16, Time since last communication: 0.000s
W20260130 02:59:18.524400 22744 consensus_peers.cc:597] T 00000000000000000000000000000000 P 342be83f20a648fd849c7d23454fba08 -> Peer b01057b6c5ff4fc4b07253801881fe13 (127.18.100.125:39157): Couldn't send request to peer b01057b6c5ff4fc4b07253801881fe13. Status: Network error: Client connection negotiation failed: client connection to 127.18.100.125:39157: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260130 02:59:18.571699 22756 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:60660:
table { table_name: "default.Table_e3b3683b36e144c2b33dd88b292bcb23" } new_table_name: "default.Table_1b5ff18b6b2146a98161fe3210afe549" modify_external_catalogs: true
I20260130 02:59:18.574534 22752 master_service.cc:954] ReplaceTablet: received request to replace tablet 476f8fd240bc48f1a9c47a0351ad5d27 from {username='slave'} at 127.0.0.1:60660
I20260130 02:59:18.577374 22750 master_service.cc:954] ReplaceTablet: received request to replace tablet 5cc07f1d50624e03b1f5498b12ea680b from {username='slave'} at 127.0.0.1:60660
I20260130 02:59:18.574230 22754 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60660:
name: "default.Table_91ef08c82eb0446fbb6164d207168099"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260130 02:59:18.578922 22750 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60660:
name: "default.Table_53c1c4a7b44f4bb095c3167a66cf3315"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260130 02:59:18.574997 22757 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:60660:
table { table_name: "default.Table_62e153a4e7844d97863f32a6e87da70b" } new_table_name: "default.Table_9fdc4fc1d8a442389a5c9201bcd4ab84" modify_external_catalogs: true
I20260130 02:59:18.576296 22751 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:60660:
table { table_name: "default.Table_d6f2cd9cd3b54168b14f1cacabbc74f0" } modify_external_catalogs: true
I20260130 02:59:18.580130 22751 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:60660:
table { table_name: "default.Table_d6f2cd9cd3b54168b14f1cacabbc74f0" } modify_external_catalogs: true
I20260130 02:59:18.576473 22755 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60660:
name: "default.Table_2dde477c96844e51bc37ab8d6bd13248"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260130 02:59:18.574553 22753 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60660:
name: "default.Table_66d14bc2eefe4c4cabbfe8ea4ca2a287"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
F20260130 02:59:18.587239 22816 hms_notification_log_listener.cc:287] non-monotonous event IDs from HMS: current 1, previous 1; dumping first 2 out of 4 received events: NotificationEvent(eventId=1, eventTime=1769741949, eventType=CREATE_TABLE, dbName=default, tableName=table_de96622ce9794752adfc14fb9ceb4559, message={"server":"","servicePrincipal":"","db":"default","table":"table_de96622ce9794752adfc14fb9ceb4559","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_de96622ce9794752adfc14fb9ceb4559\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1769741949},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1769741859011789-18833-0/minicluster-data/warehouse/table_de96622ce9794752adfc14fb9ceb4559\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_de96622ce9794752adfc14fb9ceb4559\",\"kudu.table_id\":\"e5276a565ee84745a424387f988a65e9\",\"transient_lastDdlTime\":\"1769741949\",\"comment\":\"\",\"kudu.master_addresses\":\"127.18.100.126:42077,127.18.100.125:39157,127.18.100.124:34987\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"40eb42871f464e95bd641ceacb567979\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1769741949,"files":[]}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=1, eventTime=1769741949, eventType=CREATE_TABLE, dbName=default, tableName=table_62e153a4e7844d97863f32a6e87da70b, message={"server":"","servicePrincipal":"","db":"default","table":"table_62e153a4e7844d97863f32a6e87da70b","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_62e153a4e7844d97863f32a6e87da70b\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1769741949},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1769741859011789-18833-0/minicluster-data/warehouse/table_62e153a4e7844d97863f32a6e87da70b\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_62e153a4e7844d97863f32a6e87da70b\",\"kudu.table_id\":\"dee95603eab3439b81355a006bc9f376\",\"transient_lastDdlTime\":\"1769741949\",\"comment\":\"\",\"kudu.master_addresses\":\"127.18.100.126:42077,127.18.100.125:39157,127.18.100.124:34987\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"40eb42871f464e95bd641ceacb567979\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1769741949,"files":[]}, messageFormat=json-0.2, catName=hive);
I20260130 02:59:18.590862 23032 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 342be83f20a648fd849c7d23454fba08. Latest consensus state: current_term: 4 leader_uuid: "342be83f20a648fd849c7d23454fba08" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22dedcf69e00438999d3a5fedde57324" member_type: VOTER last_known_addr { host: "127.18.100.126" port: 42077 } } peers { permanent_uuid: "b01057b6c5ff4fc4b07253801881fe13" member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39157 } } peers { permanent_uuid: "342be83f20a648fd849c7d23454fba08" member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34987 } } }
I20260130 02:59:18.591571 23032 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [sys.catalog]: This master's current role is: FOLLOWER
*** Check failure stack trace: ***
I20260130 02:59:18.597916 23032 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 4 leader_uuid: "342be83f20a648fd849c7d23454fba08" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22dedcf69e00438999d3a5fedde57324" member_type: VOTER last_known_addr { host: "127.18.100.126" port: 42077 } } peers { permanent_uuid: "b01057b6c5ff4fc4b07253801881fe13" member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39157 } } peers { permanent_uuid: "342be83f20a648fd849c7d23454fba08" member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34987 } } }
I20260130 02:59:18.598717 23032 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [sys.catalog]: This master's current role is: FOLLOWER
I20260130 02:59:18.598649 23068 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 4 leader_uuid: "342be83f20a648fd849c7d23454fba08" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22dedcf69e00438999d3a5fedde57324" member_type: VOTER last_known_addr { host: "127.18.100.126" port: 42077 } } peers { permanent_uuid: "b01057b6c5ff4fc4b07253801881fe13" member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39157 } } peers { permanent_uuid: "342be83f20a648fd849c7d23454fba08" member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34987 } } }
I20260130 02:59:18.599722 23068 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [sys.catalog]: This master's current role is: FOLLOWER
I20260130 02:59:18.608574 23062 sys_catalog.cc:455] T 00000000000000000000000000000000 P 342be83f20a648fd849c7d23454fba08 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 4 leader_uuid: "342be83f20a648fd849c7d23454fba08" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22dedcf69e00438999d3a5fedde57324" member_type: VOTER last_known_addr { host: "127.18.100.126" port: 42077 } } peers { permanent_uuid: "b01057b6c5ff4fc4b07253801881fe13" member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39157 } } peers { permanent_uuid: "342be83f20a648fd849c7d23454fba08" member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34987 } } }
I20260130 02:59:18.609287 23062 sys_catalog.cc:458] T 00000000000000000000000000000000 P 342be83f20a648fd849c7d23454fba08 [sys.catalog]: This master's current role is: LEADER
I20260130 02:59:18.629462 23066 mvcc.cc:204] Tried to move back new op lower bound from 7248863062090637312 to 7248863041496182784. Current Snapshot: MvccSnapshot[applied={T|T < 7248863062090637312 or (T in {7248863062090637312})}]
I20260130 02:59:18.647219 22511 tablet_service.cc:1505] Processing CreateTablet for tablet ce91a17790324cd884a0521ce67dde67 (DEFAULT_TABLE table=default.table_e3b3683b36e144c2b33dd88b292bcb23 [id=318c872fbfef4f93b4c280fd078cabd9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:59:18.648551 22511 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ce91a17790324cd884a0521ce67dde67. 1 dirs total, 0 dirs full, 0 dirs failed
@ 0x7f02b81c8950 google::LogMessage::Flush() at ??:0
@ 0x7f02b81c86bf google::LogMessage::~LogMessage() at ??:0
I20260130 02:59:18.679307 23078 tablet_bootstrap.cc:492] T ce91a17790324cd884a0521ce67dde67 P 789dbb4a27204ffbb68d7ae40f8f5e4a: Bootstrap starting.
I20260130 02:59:18.688630 23078 tablet_bootstrap.cc:654] T ce91a17790324cd884a0521ce67dde67 P 789dbb4a27204ffbb68d7ae40f8f5e4a: Neither blocks nor log segments found. Creating new log.
I20260130 02:59:18.700990 22367 tablet_service.cc:1505] Processing CreateTablet for tablet ce91a17790324cd884a0521ce67dde67 (DEFAULT_TABLE table=default.table_e3b3683b36e144c2b33dd88b292bcb23 [id=318c872fbfef4f93b4c280fd078cabd9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:59:18.702052 23078 tablet_bootstrap.cc:492] T ce91a17790324cd884a0521ce67dde67 P 789dbb4a27204ffbb68d7ae40f8f5e4a: No bootstrap required, opened a new log
I20260130 02:59:18.702425 22367 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ce91a17790324cd884a0521ce67dde67. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:59:18.703513 23078 ts_tablet_manager.cc:1403] T ce91a17790324cd884a0521ce67dde67 P 789dbb4a27204ffbb68d7ae40f8f5e4a: Time spent bootstrapping tablet: real 0.025s user 0.007s sys 0.010s
@ 0x7f02c3267fdc kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
I20260130 02:59:18.706300 23078 raft_consensus.cc:359] T ce91a17790324cd884a0521ce67dde67 P 789dbb4a27204ffbb68d7ae40f8f5e4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "789dbb4a27204ffbb68d7ae40f8f5e4a" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 41825 } } peers { permanent_uuid: "dd32a35972b347b89ceaaad2455aef5a" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 39781 } } peers { permanent_uuid: "111215cc989f4203a6e1517f2291d66f" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 39751 } }
I20260130 02:59:18.707198 23078 raft_consensus.cc:385] T ce91a17790324cd884a0521ce67dde67 P 789dbb4a27204ffbb68d7ae40f8f5e4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:59:18.707475 23078 raft_consensus.cc:740] T ce91a17790324cd884a0521ce67dde67 P 789dbb4a27204ffbb68d7ae40f8f5e4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 789dbb4a27204ffbb68d7ae40f8f5e4a, State: Initialized, Role: FOLLOWER
I20260130 02:59:18.710047 23078 consensus_queue.cc:260] T ce91a17790324cd884a0521ce67dde67 P 789dbb4a27204ffbb68d7ae40f8f5e4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "789dbb4a27204ffbb68d7ae40f8f5e4a" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 41825 } } peers { permanent_uuid: "dd32a35972b347b89ceaaad2455aef5a" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 39781 } } peers { permanent_uuid: "111215cc989f4203a6e1517f2291d66f" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 39751 } }
I20260130 02:59:18.719574 23078 ts_tablet_manager.cc:1434] T ce91a17790324cd884a0521ce67dde67 P 789dbb4a27204ffbb68d7ae40f8f5e4a: Time spent starting tablet: real 0.016s user 0.004s sys 0.008s
I20260130 02:59:18.724071 22581 heartbeater.cc:499] Master 127.18.100.124:34987 was elected leader, sending a full tablet report...
@ 0x7f02c326779c kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
I20260130 02:59:18.731500 22752 catalog_manager.cc:5654] T 670a199ebd03458cb3dae713581506f7 P 789dbb4a27204ffbb68d7ae40f8f5e4a reported cstate change: term changed from 0 to 1, leader changed from <none> to 789dbb4a27204ffbb68d7ae40f8f5e4a (127.18.100.67). New cstate: current_term: 1 leader_uuid: "789dbb4a27204ffbb68d7ae40f8f5e4a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111215cc989f4203a6e1517f2291d66f" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 39751 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dd32a35972b347b89ceaaad2455aef5a" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 39781 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "789dbb4a27204ffbb68d7ae40f8f5e4a" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 41825 } health_report { overall_health: HEALTHY } } }
@ 0x7f02c326b812 kudu::master::HmsNotificationLogListenerTask::Init()::$_0::operator()() at ??:0
I20260130 02:59:18.753355 22227 tablet_service.cc:1505] Processing CreateTablet for tablet ce91a17790324cd884a0521ce67dde67 (DEFAULT_TABLE table=default.table_e3b3683b36e144c2b33dd88b292bcb23 [id=318c872fbfef4f93b4c280fd078cabd9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:59:18.754462 22227 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ce91a17790324cd884a0521ce67dde67. 1 dirs total, 0 dirs full, 0 dirs failed
@ 0x7f02c326b7ca _ZNSt3__18__invokeIRZN4kudu6master30HmsNotificationLogListenerTask4InitEvE3$_0JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
I20260130 02:59:18.776852 23087 tablet_bootstrap.cc:492] T ce91a17790324cd884a0521ce67dde67 P dd32a35972b347b89ceaaad2455aef5a: Bootstrap starting.
I20260130 02:59:18.787626 23089 tablet_bootstrap.cc:492] T ce91a17790324cd884a0521ce67dde67 P 111215cc989f4203a6e1517f2291d66f: Bootstrap starting.
I20260130 02:59:18.794214 23089 tablet_bootstrap.cc:654] T ce91a17790324cd884a0521ce67dde67 P 111215cc989f4203a6e1517f2291d66f: Neither blocks nor log segments found. Creating new log.
I20260130 02:59:18.794889 23087 tablet_bootstrap.cc:654] T ce91a17790324cd884a0521ce67dde67 P dd32a35972b347b89ceaaad2455aef5a: Neither blocks nor log segments found. Creating new log.
I20260130 02:59:18.794854 22752 master_service.cc:954] ReplaceTablet: received request to replace tablet 670a199ebd03458cb3dae713581506f7 from {username='slave'} at 127.0.0.1:60660
I20260130 02:59:18.794987 22749 master_service.cc:954] ReplaceTablet: received request to replace tablet ce91a17790324cd884a0521ce67dde67 from {username='slave'} at 127.0.0.1:60660
@ 0x7f02c326b75a std::__1::__invoke_void_return_wrapper<>::__call<>() at ??:0
@ 0x7f02c326b722 std::__1::__function::__alloc_func<>::operator()() at ??:0
I20260130 02:59:18.824716 22752 catalog_manager.cc:5941] T 00000000000000000000000000000000 P 342be83f20a648fd849c7d23454fba08: Sending DeleteTablet for 3 replicas of tablet 670a199ebd03458cb3dae713581506f7
I20260130 02:59:18.826242 22752 catalog_manager.cc:6724] ReplaceTablet: tablet 670a199ebd03458cb3dae713581506f7 deleted and replaced by tablet be72107f1e024d5788f5d2459c85015c
I20260130 02:59:18.827143 22511 tablet_service.cc:1552] Processing DeleteTablet for tablet 670a199ebd03458cb3dae713581506f7 with delete_type TABLET_DATA_DELETED (replaced by tablet be72107f1e024d5788f5d2459c85015c) from {username='slave'} at 127.0.0.1:55910
I20260130 02:59:18.827493 22227 tablet_service.cc:1552] Processing DeleteTablet for tablet 670a199ebd03458cb3dae713581506f7 with delete_type TABLET_DATA_DELETED (replaced by tablet be72107f1e024d5788f5d2459c85015c) from {username='slave'} at 127.0.0.1:35484
I20260130 02:59:18.827541 22367 tablet_service.cc:1552] Processing DeleteTablet for tablet 670a199ebd03458cb3dae713581506f7 with delete_type TABLET_DATA_DELETED (replaced by tablet be72107f1e024d5788f5d2459c85015c) from {username='slave'} at 127.0.0.1:54432
I20260130 02:59:18.841763 23093 tablet_replica.cc:333] T 670a199ebd03458cb3dae713581506f7 P 789dbb4a27204ffbb68d7ae40f8f5e4a: stopping tablet replica
I20260130 02:59:18.842489 23093 raft_consensus.cc:2243] T 670a199ebd03458cb3dae713581506f7 P 789dbb4a27204ffbb68d7ae40f8f5e4a [term 1 LEADER]: Raft consensus shutting down.
I20260130 02:59:18.843469 23093 raft_consensus.cc:2272] T 670a199ebd03458cb3dae713581506f7 P 789dbb4a27204ffbb68d7ae40f8f5e4a [term 1 FOLLOWER]: Raft consensus is shut down!
@ 0x7f02c326aa1e std::__1::__function::__func<>::operator()() at ??:0
I20260130 02:59:18.845500 23093 ts_tablet_manager.cc:1916] T 670a199ebd03458cb3dae713581506f7 P 789dbb4a27204ffbb68d7ae40f8f5e4a: Deleting tablet data with delete state TABLET_DATA_DELETED
@ 0x5aa2b5 std::__1::__function::__value_func<>::operator()() at ??:?
@ 0x5aa0ea std::__1::function<>::operator()() at ??:?
I20260130 02:59:18.849720 23094 tablet_replica.cc:333] T 670a199ebd03458cb3dae713581506f7 P 111215cc989f4203a6e1517f2291d66f: stopping tablet replica
I20260130 02:59:18.850466 23094 raft_consensus.cc:2243] T 670a199ebd03458cb3dae713581506f7 P 111215cc989f4203a6e1517f2291d66f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260130 02:59:18.850957 23094 raft_consensus.cc:2272] T 670a199ebd03458cb3dae713581506f7 P 111215cc989f4203a6e1517f2291d66f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260130 02:59:18.852823 23094 ts_tablet_manager.cc:1916] T 670a199ebd03458cb3dae713581506f7 P 111215cc989f4203a6e1517f2291d66f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260130 02:59:18.854655 22749 catalog_manager.cc:5941] T 00000000000000000000000000000000 P 342be83f20a648fd849c7d23454fba08: Sending DeleteTablet for 3 replicas of tablet ce91a17790324cd884a0521ce67dde67
I20260130 02:59:18.856527 22514 tablet_service.cc:1552] Processing DeleteTablet for tablet ce91a17790324cd884a0521ce67dde67 with delete_type TABLET_DATA_DELETED (replaced by tablet 113f519ba54548ef89c61d7c3bac58a5) from {username='slave'} at 127.0.0.1:55910
I20260130 02:59:18.856943 22368 tablet_service.cc:1552] Processing DeleteTablet for tablet ce91a17790324cd884a0521ce67dde67 with delete_type TABLET_DATA_DELETED (replaced by tablet 113f519ba54548ef89c61d7c3bac58a5) from {username='slave'} at 127.0.0.1:54432
I20260130 02:59:18.863404 22749 catalog_manager.cc:6724] ReplaceTablet: tablet ce91a17790324cd884a0521ce67dde67 deleted and replaced by tablet 113f519ba54548ef89c61d7c3bac58a5
I20260130 02:59:18.864778 22227 tablet_service.cc:1552] Processing DeleteTablet for tablet ce91a17790324cd884a0521ce67dde67 with delete_type TABLET_DATA_DELETED (replaced by tablet 113f519ba54548ef89c61d7c3bac58a5) from {username='slave'} at 127.0.0.1:35484
@ 0x7f02ba3ea69b kudu::Thread::SuperviseThread() at ??:0
@ 0x474190 __tsan_thread_start_func at ??:?
I20260130 02:59:18.870086 23095 tablet_replica.cc:333] T 670a199ebd03458cb3dae713581506f7 P dd32a35972b347b89ceaaad2455aef5a: stopping tablet replica
I20260130 02:59:18.874692 23095 raft_consensus.cc:2243] T 670a199ebd03458cb3dae713581506f7 P dd32a35972b347b89ceaaad2455aef5a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260130 02:59:18.879606 23095 raft_consensus.cc:2272] T 670a199ebd03458cb3dae713581506f7 P dd32a35972b347b89ceaaad2455aef5a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260130 02:59:18.881825 23095 ts_tablet_manager.cc:1916] T 670a199ebd03458cb3dae713581506f7 P dd32a35972b347b89ceaaad2455aef5a: Deleting tablet data with delete state TABLET_DATA_DELETED
@ 0x7f02bbaf56db start_thread at ??:0
I20260130 02:59:18.890990 22511 tablet_service.cc:1505] Processing CreateTablet for tablet be72107f1e024d5788f5d2459c85015c (DEFAULT_TABLE table=default.table_d6f2cd9cd3b54168b14f1cacabbc74f0 [id=db113a9d4bf545faa72c7c94427d6dba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260130 02:59:18.892086 22511 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet be72107f1e024d5788f5d2459c85015c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260130 02:59:18.895891 23094 ts_tablet_manager.cc:1929] T 670a199ebd03458cb3dae713581506f7 P 111215cc989f4203a6e1517f2291d66f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260130 02:59:18.896282 23094 log.cc:1199] T 670a199ebd03458cb3dae713581506f7 P 111215cc989f4203a6e1517f2291d66f: Deleting WAL directory at /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1769741859011789-18833-0/minicluster-data/ts-0/wal/wals/670a199ebd03458cb3dae713581506f7
I20260130 02:59:18.897022 23094 ts_tablet_manager.cc:1950] T 670a199ebd03458cb3dae713581506f7 P 111215cc989f4203a6e1517f2291d66f: Deleting consensus metadata
@ 0x7f02b6e6d71f clone at ??:0
I20260130 02:59:18.901417 23093 ts_tablet_manager.cc:1929] T 670a199ebd03458cb3dae713581506f7 P 789dbb4a27204ffbb68d7ae40f8f5e4a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260130 02:59:18.901822 23093 log.cc:1199] T 670a199ebd03458cb3dae713581506f7 P 789dbb4a27204ffbb68d7ae40f8f5e4a: Deleting WAL directory at /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1769741859011789-18833-0/minicluster-data/ts-2/wal/wals/670a199ebd03458cb3dae713581506f7
I20260130 02:59:18.902678 23093 ts_tablet_manager.cc:1950] T 670a199ebd03458cb3dae713581506f7 P 789dbb4a27204ffbb68d7ae40f8f5e4a: Deleting consensus metadata
I20260130 02:59:18.913970 23095 ts_tablet_manager.cc:1929] T 670a199ebd03458cb3dae713581506f7 P dd32a35972b347b89ceaaad2455aef5a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260130 02:59:18.914423 23095 log.cc:1199] T 670a199ebd03458cb3dae713581506f7 P dd32a35972b347b89ceaaad2455aef5a: Deleting WAL directory at /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1769741859011789-18833-0/minicluster-data/ts-1/wal/wals/670a199ebd03458cb3dae713581506f7
I20260130 02:59:18.914983 23093 tablet_replica.cc:333] T ce91a17790324cd884a0521ce67dde67 P 789dbb4a27204ffbb68d7ae40f8f5e4a: stopping tablet replica
I20260130 02:59:18.915341 23095 ts_tablet_manager.cc:1950] T 670a199ebd03458cb3dae713581506f7 P dd32a35972b347b89ceaaad2455aef5a: Deleting consensus metadata
I20260130 02:59:18.915652 23093 raft_consensus.cc:2243] T ce91a17790324cd884a0521ce67dde67 P 789dbb4a27204ffbb68d7ae40f8f5e4a [term 0 FOLLOWER]: Raft consensus shutting down.
I20260130 02:59:18.916085 23093 raft_consensus.cc:2272] T ce91a17790324cd884a0521ce67dde67 P 789dbb4a27204ffbb68d7ae40f8f5e4a [term 0 FOLLOWER]: Raft consensus is shut down!
I20260130 02:59:18.918339 23087 tablet_bootstrap.cc:492] T ce91a17790324cd884a0521ce67dde67 P dd32a35972b347b89ceaaad2455aef5a: No bootstrap required, opened a new log
I20260130 02:59:18.918778 23087 ts_tablet_manager.cc:1403] T ce91a17790324cd884a0521ce67dde67 P dd32a35972b347b89ceaaad2455aef5a: Time spent bootstrapping tablet: real 0.142s user 0.014s sys 0.026s
I20260130 02:59:18.920040 23093 ts_tablet_manager.cc:1916] T ce91a17790324cd884a0521ce67dde67 P 789dbb4a27204ffbb68d7ae40f8f5e4a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260130 02:59:18.921470 23087 raft_consensus.cc:359] T ce91a17790324cd884a0521ce67dde67 P dd32a35972b347b89ceaaad2455aef5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "789dbb4a27204ffbb68d7ae40f8f5e4a" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 41825 } } peers { permanent_uuid: "dd32a35972b347b89ceaaad2455aef5a" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 39781 } } peers { permanent_uuid: "111215cc989f4203a6e1517f2291d66f" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 39751 } }
I20260130 02:59:18.922161 23087 raft_consensus.cc:385] T ce91a17790324cd884a0521ce67dde67 P dd32a35972b347b89ceaaad2455aef5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:59:18.922433 23087 raft_consensus.cc:740] T ce91a17790324cd884a0521ce67dde67 P dd32a35972b347b89ceaaad2455aef5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd32a35972b347b89ceaaad2455aef5a, State: Initialized, Role: FOLLOWER
I20260130 02:59:18.923221 23089 tablet_bootstrap.cc:492] T ce91a17790324cd884a0521ce67dde67 P 111215cc989f4203a6e1517f2291d66f: No bootstrap required, opened a new log
I20260130 02:59:18.923647 23089 ts_tablet_manager.cc:1403] T ce91a17790324cd884a0521ce67dde67 P 111215cc989f4203a6e1517f2291d66f: Time spent bootstrapping tablet: real 0.136s user 0.017s sys 0.017s
I20260130 02:59:18.923173 23087 consensus_queue.cc:260] T ce91a17790324cd884a0521ce67dde67 P dd32a35972b347b89ceaaad2455aef5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "789dbb4a27204ffbb68d7ae40f8f5e4a" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 41825 } } peers { permanent_uuid: "dd32a35972b347b89ceaaad2455aef5a" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 39781 } } peers { permanent_uuid: "111215cc989f4203a6e1517f2291d66f" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 39751 } }
I20260130 02:59:18.928717 23078 tablet_bootstrap.cc:492] T be72107f1e024d5788f5d2459c85015c P 789dbb4a27204ffbb68d7ae40f8f5e4a: Bootstrap starting.
I20260130 02:59:18.934718 23078 tablet_bootstrap.cc:654] T be72107f1e024d5788f5d2459c85015c P 789dbb4a27204ffbb68d7ae40f8f5e4a: Neither blocks nor log segments found. Creating new log.
I20260130 02:59:18.926144 23089 raft_consensus.cc:359] T ce91a17790324cd884a0521ce67dde67 P 111215cc989f4203a6e1517f2291d66f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "789dbb4a27204ffbb68d7ae40f8f5e4a" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 41825 } } peers { permanent_uuid: "dd32a35972b347b89ceaaad2455aef5a" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 39781 } } peers { permanent_uuid: "111215cc989f4203a6e1517f2291d66f" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 39751 } }
I20260130 02:59:18.939198 23089 raft_consensus.cc:385] T ce91a17790324cd884a0521ce67dde67 P 111215cc989f4203a6e1517f2291d66f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:59:18.939520 23089 raft_consensus.cc:740] T ce91a17790324cd884a0521ce67dde67 P 111215cc989f4203a6e1517f2291d66f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 111215cc989f4203a6e1517f2291d66f, State: Initialized, Role: FOLLOWER
I20260130 02:59:18.940191 23089 consensus_queue.cc:260] T ce91a17790324cd884a0521ce67dde67 P 111215cc989f4203a6e1517f2291d66f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "789dbb4a27204ffbb68d7ae40f8f5e4a" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 41825 } } peers { permanent_uuid: "dd32a35972b347b89ceaaad2455aef5a" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 39781 } } peers { permanent_uuid: "111215cc989f4203a6e1517f2291d66f" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 39751 } }
I20260130 02:59:18.941625 22294 heartbeater.cc:499] Master 127.18.100.124:34987 was elected leader, sending a full tablet report...
I20260130 02:59:18.943534 23089 ts_tablet_manager.cc:1434] T ce91a17790324cd884a0521ce67dde67 P 111215cc989f4203a6e1517f2291d66f: Time spent starting tablet: real 0.020s user 0.007s sys 0.000s
I20260130 02:59:18.947952 22438 heartbeater.cc:499] Master 127.18.100.124:34987 was elected leader, sending a full tablet report...
I20260130 02:59:18.951391 23093 ts_tablet_manager.cc:1929] T ce91a17790324cd884a0521ce67dde67 P 789dbb4a27204ffbb68d7ae40f8f5e4a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260130 02:59:18.951761 23093 log.cc:1199] T ce91a17790324cd884a0521ce67dde67 P 789dbb4a27204ffbb68d7ae40f8f5e4a: Deleting WAL directory at /tmp/dist-test-tasktGe2wI/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1769741859011789-18833-0/minicluster-data/ts-2/wal/wals/ce91a17790324cd884a0521ce67dde67
I20260130 02:59:18.952391 23093 ts_tablet_manager.cc:1950] T ce91a17790324cd884a0521ce67dde67 P 789dbb4a27204ffbb68d7ae40f8f5e4a: Deleting consensus metadata
I20260130 02:59:18.961269 23087 ts_tablet_manager.cc:1434] T ce91a17790324cd884a0521ce67dde67 P dd32a35972b347b89ceaaad2455aef5a: Time spent starting tablet: real 0.042s user 0.011s sys 0.000s
I20260130 02:59:18.970521 23078 tablet_bootstrap.cc:492] T be72107f1e024d5788f5d2459c85015c P 789dbb4a27204ffbb68d7ae40f8f5e4a: No bootstrap required, opened a new log
I20260130 02:59:18.970978 23078 ts_tablet_manager.cc:1403] T be72107f1e024d5788f5d2459c85015c P 789dbb4a27204ffbb68d7ae40f8f5e4a: Time spent bootstrapping tablet: real 0.042s user 0.008s sys 0.004s
I20260130 02:59:18.973565 23078 raft_consensus.cc:359] T be72107f1e024d5788f5d2459c85015c P 789dbb4a27204ffbb68d7ae40f8f5e4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "789dbb4a27204ffbb68d7ae40f8f5e4a" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 41825 } } peers { permanent_uuid: "dd32a35972b347b89ceaaad2455aef5a" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 39781 } } peers { permanent_uuid: "111215cc989f4203a6e1517f2291d66f" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 39751 } }
I20260130 02:59:18.979144 23078 raft_consensus.cc:385] T be72107f1e024d5788f5d2459c85015c P 789dbb4a27204ffbb68d7ae40f8f5e4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260130 02:59:18.979468 23078 raft_consensus.cc:740] T be72107f1e024d5788f5d2459c85015c P 789dbb4a27204ffbb68d7ae40f8f5e4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 789dbb4a27204ffbb68d7ae40f8f5e4a, State: Initialized, Role: FOLLOWER
I20260130 02:59:18.980345 23078 consensus_queue.cc:260] T be72107f1e024d5788f5d2459c85015c P 789dbb4a27204ffbb68d7ae40f8f5e4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "789dbb4a27204ffbb68d7ae40f8f5e4a" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 41825 } } peers { permanent_uuid: "dd32a35972b347b89ceaaad2455aef5a" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 39781 } } peers { permanent_uuid: "111215cc989f4203a6e1517f2291d66f" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 39751 } }
I20260130 02:59:18.982362 23078 ts_tablet_manager.cc:1434] T be72107f1e024d5788f5d2459c85015c P 789dbb4a27204ffbb68d7ae40f8f5e4a: Time spent starting tablet: real 0.011s user 0.002s sys 0.003s
W20260130 02:59:19.159335 22180 connection.cc:537] server connection from 127.0.0.1:35484 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260130 02:59:19.159512 22322 connection.cc:537] client connection to 127.18.100.124:34987 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260130 02:59:19.159492 22324 connection.cc:537] server connection from 127.0.0.1:54432 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260130 02:59:19.160604 22468 connection.cc:537] server connection from 127.0.0.1:55910 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260130 02:59:19.162773 22180 connection.cc:537] client connection to 127.18.100.124:34987 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260130 02:59:19.164803 22469 connection.cc:537] client connection to 127.18.100.124:34987 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260130 02:59:19.165664 22592 connection.cc:537] client connection to 127.18.100.124:34987 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260130 02:59:19.167197 22592 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.18.100.124:34987)
W20260130 02:59:19.167855 22592 master_proxy_rpc.cc:203] Re-attempting DeleteTable request to leader Master (127.18.100.124:34987)
W20260130 02:59:19.169531 22592 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.18.100.124:34987)
W20260130 02:59:19.170077 22592 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.18.100.124:34987)
W20260130 02:59:19.180253 23077 negotiation.cc:337] Failed RPC negotiation. Trace:
0130 02:59:19.161650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.100.124:34987 (local address 127.18.100.66:55197)
0130 02:59:19.162404 (+ 754us) negotiation.cc:107] Waiting for socket to connect
0130 02:59:19.162456 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0130 02:59:19.164458 (+ 2002us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0130 02:59:19.179342 (+ 14884us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.100.124:34987: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":144}
I20260130 02:59:19.181092 23083 raft_consensus.cc:493] T be72107f1e024d5788f5d2459c85015c P 789dbb4a27204ffbb68d7ae40f8f5e4a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:59:19.181555 23083 raft_consensus.cc:515] T be72107f1e024d5788f5d2459c85015c P 789dbb4a27204ffbb68d7ae40f8f5e4a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "789dbb4a27204ffbb68d7ae40f8f5e4a" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 41825 } } peers { permanent_uuid: "dd32a35972b347b89ceaaad2455aef5a" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 39781 } } peers { permanent_uuid: "111215cc989f4203a6e1517f2291d66f" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 39751 } }
I20260130 02:59:19.184195 23083 leader_election.cc:290] T be72107f1e024d5788f5d2459c85015c P 789dbb4a27204ffbb68d7ae40f8f5e4a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dd32a35972b347b89ceaaad2455aef5a (127.18.100.66:39781), 111215cc989f4203a6e1517f2291d66f (127.18.100.65:39751)
W20260130 02:59:19.184670 22973 negotiation.cc:337] Failed RPC negotiation. Trace:
0130 02:59:19.165074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.100.124:34987 (local address 127.18.100.65:52571)
0130 02:59:19.165964 (+ 890us) negotiation.cc:107] Waiting for socket to connect
0130 02:59:19.166006 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0130 02:59:19.166229 (+ 223us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0130 02:59:19.183765 (+ 17536us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.100.124:34987: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":294}
W20260130 02:59:19.186507 23085 negotiation.cc:337] Failed RPC negotiation. Trace:
0130 02:59:19.171073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.100.124:34987 (local address 127.18.100.67:39763)
0130 02:59:19.171765 (+ 692us) negotiation.cc:107] Waiting for socket to connect
0130 02:59:19.171806 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0130 02:59:19.172060 (+ 254us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0130 02:59:19.185611 (+ 13551us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.100.124:34987: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":92}
W20260130 02:59:19.187882 22468 leader_election.cc:343] T be72107f1e024d5788f5d2459c85015c P 789dbb4a27204ffbb68d7ae40f8f5e4a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 111215cc989f4203a6e1517f2291d66f (127.18.100.65:39751): Not found: Tablet not found: be72107f1e024d5788f5d2459c85015c
W20260130 02:59:19.188681 22469 leader_election.cc:343] T be72107f1e024d5788f5d2459c85015c P 789dbb4a27204ffbb68d7ae40f8f5e4a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dd32a35972b347b89ceaaad2455aef5a (127.18.100.66:39781): Not found: Tablet not found: be72107f1e024d5788f5d2459c85015c
I20260130 02:59:19.189038 22469 leader_election.cc:304] T be72107f1e024d5788f5d2459c85015c P 789dbb4a27204ffbb68d7ae40f8f5e4a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 789dbb4a27204ffbb68d7ae40f8f5e4a; no voters: 111215cc989f4203a6e1517f2291d66f, dd32a35972b347b89ceaaad2455aef5a
W20260130 02:59:19.189936 23105 negotiation.cc:337] Failed RPC negotiation. Trace:
0130 02:59:19.173928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.100.124:34987 (local address 127.0.0.1:32896)
0130 02:59:19.178820 (+ 4892us) negotiation.cc:107] Waiting for socket to connect
0130 02:59:19.178865 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0130 02:59:19.179140 (+ 275us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0130 02:59:19.188206 (+ 9066us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.100.124:34987: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":4262,"thread_start_us":8908,"threads_started":1}
I20260130 02:59:19.190146 23083 raft_consensus.cc:2749] T be72107f1e024d5788f5d2459c85015c P 789dbb4a27204ffbb68d7ae40f8f5e4a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260130 02:59:19.323772 23097 raft_consensus.cc:493] T ce91a17790324cd884a0521ce67dde67 P dd32a35972b347b89ceaaad2455aef5a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:59:19.324275 23097 raft_consensus.cc:515] T ce91a17790324cd884a0521ce67dde67 P dd32a35972b347b89ceaaad2455aef5a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "789dbb4a27204ffbb68d7ae40f8f5e4a" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 41825 } } peers { permanent_uuid: "dd32a35972b347b89ceaaad2455aef5a" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 39781 } } peers { permanent_uuid: "111215cc989f4203a6e1517f2291d66f" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 39751 } }
I20260130 02:59:19.327397 23097 leader_election.cc:290] T ce91a17790324cd884a0521ce67dde67 P dd32a35972b347b89ceaaad2455aef5a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 789dbb4a27204ffbb68d7ae40f8f5e4a (127.18.100.67:41825), 111215cc989f4203a6e1517f2291d66f (127.18.100.65:39751)
W20260130 02:59:19.340277 22321 leader_election.cc:343] T ce91a17790324cd884a0521ce67dde67 P dd32a35972b347b89ceaaad2455aef5a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 789dbb4a27204ffbb68d7ae40f8f5e4a (127.18.100.67:41825): Not found: Tablet not found: ce91a17790324cd884a0521ce67dde67
I20260130 02:59:19.341791 22248 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ce91a17790324cd884a0521ce67dde67" candidate_uuid: "dd32a35972b347b89ceaaad2455aef5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "111215cc989f4203a6e1517f2291d66f" is_pre_election: true
I20260130 02:59:19.342394 22248 raft_consensus.cc:2468] T ce91a17790324cd884a0521ce67dde67 P 111215cc989f4203a6e1517f2291d66f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd32a35972b347b89ceaaad2455aef5a in term 0.
I20260130 02:59:19.343740 22321 leader_election.cc:304] T ce91a17790324cd884a0521ce67dde67 P dd32a35972b347b89ceaaad2455aef5a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 111215cc989f4203a6e1517f2291d66f, dd32a35972b347b89ceaaad2455aef5a; no voters: 789dbb4a27204ffbb68d7ae40f8f5e4a
I20260130 02:59:19.344909 23097 raft_consensus.cc:2804] T ce91a17790324cd884a0521ce67dde67 P dd32a35972b347b89ceaaad2455aef5a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260130 02:59:19.345258 23097 raft_consensus.cc:493] T ce91a17790324cd884a0521ce67dde67 P dd32a35972b347b89ceaaad2455aef5a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260130 02:59:19.345567 23097 raft_consensus.cc:3060] T ce91a17790324cd884a0521ce67dde67 P dd32a35972b347b89ceaaad2455aef5a [term 0 FOLLOWER]: Advancing to term 1
I20260130 02:59:19.353329 23097 raft_consensus.cc:515] T ce91a17790324cd884a0521ce67dde67 P dd32a35972b347b89ceaaad2455aef5a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "789dbb4a27204ffbb68d7ae40f8f5e4a" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 41825 } } peers { permanent_uuid: "dd32a35972b347b89ceaaad2455aef5a" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 39781 } } peers { permanent_uuid: "111215cc989f4203a6e1517f2291d66f" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 39751 } }
I20260130 02:59:19.355235 23097 leader_election.cc:290] T ce91a17790324cd884a0521ce67dde67 P dd32a35972b347b89ceaaad2455aef5a [CANDIDATE]: Term 1 election: Requested vote from peers 789dbb4a27204ffbb68d7ae40f8f5e4a (127.18.100.67:41825), 111215cc989f4203a6e1517f2291d66f (127.18.100.65:39751)
I20260130 02:59:19.356827 22248 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ce91a17790324cd884a0521ce67dde67" candidate_uuid: "dd32a35972b347b89ceaaad2455aef5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "111215cc989f4203a6e1517f2291d66f"
I20260130 02:59:19.357398 22248 raft_consensus.cc:3060] T ce91a17790324cd884a0521ce67dde67 P 111215cc989f4203a6e1517f2291d66f [term 0 FOLLOWER]: Advancing to term 1
W20260130 02:59:19.360010 22321 leader_election.cc:343] T ce91a17790324cd884a0521ce67dde67 P dd32a35972b347b89ceaaad2455aef5a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 789dbb4a27204ffbb68d7ae40f8f5e4a (127.18.100.67:41825): Not found: Tablet not found: ce91a17790324cd884a0521ce67dde67
I20260130 02:59:19.364387 22248 raft_consensus.cc:2468] T ce91a17790324cd884a0521ce67dde67 P 111215cc989f4203a6e1517f2291d66f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd32a35972b347b89ceaaad2455aef5a in term 1.
I20260130 02:59:19.365438 22321 leader_election.cc:304] T ce91a17790324cd884a0521ce67dde67 P dd32a35972b347b89ceaaad2455aef5a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 111215cc989f4203a6e1517f2291d66f, dd32a35972b347b89ceaaad2455aef5a; no voters: 789dbb4a27204ffbb68d7ae40f8f5e4a
I20260130 02:59:19.366187 23097 raft_consensus.cc:2804] T ce91a17790324cd884a0521ce67dde67 P dd32a35972b347b89ceaaad2455aef5a [term 1 FOLLOWER]: Leader election won for term 1
I20260130 02:59:19.371733 23097 raft_consensus.cc:697] T ce91a17790324cd884a0521ce67dde67 P dd32a35972b347b89ceaaad2455aef5a [term 1 LEADER]: Becoming Leader. State: Replica: dd32a35972b347b89ceaaad2455aef5a, State: Running, Role: LEADER
I20260130 02:59:19.375555 23097 consensus_queue.cc:237] T ce91a17790324cd884a0521ce67dde67 P dd32a35972b347b89ceaaad2455aef5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "789dbb4a27204ffbb68d7ae40f8f5e4a" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 41825 } } peers { permanent_uuid: "dd32a35972b347b89ceaaad2455aef5a" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 39781 } } peers { permanent_uuid: "111215cc989f4203a6e1517f2291d66f" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 39751 } }
I20260130 02:59:19.386544 23112 raft_consensus.cc:493] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 342be83f20a648fd849c7d23454fba08)
I20260130 02:59:19.387056 23112 raft_consensus.cc:515] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22dedcf69e00438999d3a5fedde57324" member_type: VOTER last_known_addr { host: "127.18.100.126" port: 42077 } } peers { permanent_uuid: "b01057b6c5ff4fc4b07253801881fe13" member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39157 } } peers { permanent_uuid: "342be83f20a648fd849c7d23454fba08" member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34987 } }
I20260130 02:59:19.395375 23112 leader_election.cc:290] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers b01057b6c5ff4fc4b07253801881fe13 (127.18.100.125:39157), 342be83f20a648fd849c7d23454fba08 (127.18.100.124:34987)
W20260130 02:59:19.415056 22968 leader_election.cc:336] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 342be83f20a648fd849c7d23454fba08 (127.18.100.124:34987): Network error: Client connection negotiation failed: client connection to 127.18.100.124:34987: connect: Connection refused (error 111)
W20260130 02:59:19.415693 22970 leader_election.cc:336] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer b01057b6c5ff4fc4b07253801881fe13 (127.18.100.125:39157): Network error: Client connection negotiation failed: client connection to 127.18.100.125:39157: connect: Connection refused (error 111)
I20260130 02:59:19.416111 22970 leader_election.cc:304] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22dedcf69e00438999d3a5fedde57324; no voters: 342be83f20a648fd849c7d23454fba08, b01057b6c5ff4fc4b07253801881fe13
I20260130 02:59:19.417047 23112 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260130 02:59:20.077179 23124 raft_consensus.cc:493] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 342be83f20a648fd849c7d23454fba08)
I20260130 02:59:20.077625 23124 raft_consensus.cc:515] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22dedcf69e00438999d3a5fedde57324" member_type: VOTER last_known_addr { host: "127.18.100.126" port: 42077 } } peers { permanent_uuid: "b01057b6c5ff4fc4b07253801881fe13" member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39157 } } peers { permanent_uuid: "342be83f20a648fd849c7d23454fba08" member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34987 } }
I20260130 02:59:20.079077 23124 leader_election.cc:290] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers b01057b6c5ff4fc4b07253801881fe13 (127.18.100.125:39157), 342be83f20a648fd849c7d23454fba08 (127.18.100.124:34987)
W20260130 02:59:20.083410 22970 leader_election.cc:336] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer b01057b6c5ff4fc4b07253801881fe13 (127.18.100.125:39157): Network error: Client connection negotiation failed: client connection to 127.18.100.125:39157: connect: Connection refused (error 111)
W20260130 02:59:20.083761 22968 leader_election.cc:336] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 342be83f20a648fd849c7d23454fba08 (127.18.100.124:34987): Network error: Client connection negotiation failed: client connection to 127.18.100.124:34987: connect: Connection refused (error 111)
I20260130 02:59:20.084061 22968 leader_election.cc:304] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22dedcf69e00438999d3a5fedde57324; no voters: 342be83f20a648fd849c7d23454fba08, b01057b6c5ff4fc4b07253801881fe13
I20260130 02:59:20.084620 23124 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260130 02:59:20.204411 18833 master-stress-test.cc:577] Killing test thread 1/10
W20260130 02:59:20.205359 22321 consensus_peers.cc:597] T ce91a17790324cd884a0521ce67dde67 P dd32a35972b347b89ceaaad2455aef5a -> Peer 789dbb4a27204ffbb68d7ae40f8f5e4a (127.18.100.67:41825): Couldn't send request to peer 789dbb4a27204ffbb68d7ae40f8f5e4a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ce91a17790324cd884a0521ce67dde67. This is attempt 1: this message will repeat every 5th retry.
I20260130 02:59:20.250408 23128 consensus_queue.cc:1048] T ce91a17790324cd884a0521ce67dde67 P dd32a35972b347b89ceaaad2455aef5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "111215cc989f4203a6e1517f2291d66f" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 39751 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260130 02:59:21.065487 23139 raft_consensus.cc:493] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 342be83f20a648fd849c7d23454fba08)
I20260130 02:59:21.065965 23139 raft_consensus.cc:515] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22dedcf69e00438999d3a5fedde57324" member_type: VOTER last_known_addr { host: "127.18.100.126" port: 42077 } } peers { permanent_uuid: "b01057b6c5ff4fc4b07253801881fe13" member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39157 } } peers { permanent_uuid: "342be83f20a648fd849c7d23454fba08" member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34987 } }
I20260130 02:59:21.067716 23139 leader_election.cc:290] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers b01057b6c5ff4fc4b07253801881fe13 (127.18.100.125:39157), 342be83f20a648fd849c7d23454fba08 (127.18.100.124:34987)
W20260130 02:59:21.072808 22970 leader_election.cc:336] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer b01057b6c5ff4fc4b07253801881fe13 (127.18.100.125:39157): Network error: Client connection negotiation failed: client connection to 127.18.100.125:39157: connect: Connection refused (error 111)
W20260130 02:59:21.073303 22968 leader_election.cc:336] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 342be83f20a648fd849c7d23454fba08 (127.18.100.124:34987): Network error: Client connection negotiation failed: client connection to 127.18.100.124:34987: connect: Connection refused (error 111)
I20260130 02:59:21.073678 22968 leader_election.cc:304] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22dedcf69e00438999d3a5fedde57324; no voters: 342be83f20a648fd849c7d23454fba08, b01057b6c5ff4fc4b07253801881fe13
I20260130 02:59:21.074312 23139 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260130 02:59:21.767540 23154 raft_consensus.cc:493] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 342be83f20a648fd849c7d23454fba08)
I20260130 02:59:21.768069 23154 raft_consensus.cc:515] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22dedcf69e00438999d3a5fedde57324" member_type: VOTER last_known_addr { host: "127.18.100.126" port: 42077 } } peers { permanent_uuid: "b01057b6c5ff4fc4b07253801881fe13" member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39157 } } peers { permanent_uuid: "342be83f20a648fd849c7d23454fba08" member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34987 } }
I20260130 02:59:21.769721 23154 leader_election.cc:290] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers b01057b6c5ff4fc4b07253801881fe13 (127.18.100.125:39157), 342be83f20a648fd849c7d23454fba08 (127.18.100.124:34987)
W20260130 02:59:21.775314 22968 leader_election.cc:336] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 342be83f20a648fd849c7d23454fba08 (127.18.100.124:34987): Network error: Client connection negotiation failed: client connection to 127.18.100.124:34987: connect: Connection refused (error 111)
W20260130 02:59:21.775772 22970 leader_election.cc:336] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer b01057b6c5ff4fc4b07253801881fe13 (127.18.100.125:39157): Network error: Client connection negotiation failed: client connection to 127.18.100.125:39157: connect: Connection refused (error 111)
I20260130 02:59:21.776139 22970 leader_election.cc:304] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22dedcf69e00438999d3a5fedde57324; no voters: 342be83f20a648fd849c7d23454fba08, b01057b6c5ff4fc4b07253801881fe13
I20260130 02:59:21.777002 23154 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260130 02:59:22.053004 23148 raft_consensus.cc:493] T 5cc07f1d50624e03b1f5498b12ea680b P 111215cc989f4203a6e1517f2291d66f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:59:22.053426 23148 raft_consensus.cc:515] T 5cc07f1d50624e03b1f5498b12ea680b P 111215cc989f4203a6e1517f2291d66f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111215cc989f4203a6e1517f2291d66f" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 39751 } } peers { permanent_uuid: "dd32a35972b347b89ceaaad2455aef5a" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 39781 } } peers { permanent_uuid: "789dbb4a27204ffbb68d7ae40f8f5e4a" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 41825 } }
I20260130 02:59:22.055004 23148 leader_election.cc:290] T 5cc07f1d50624e03b1f5498b12ea680b P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dd32a35972b347b89ceaaad2455aef5a (127.18.100.66:39781), 789dbb4a27204ffbb68d7ae40f8f5e4a (127.18.100.67:41825)
W20260130 02:59:22.057104 22180 leader_election.cc:343] T 5cc07f1d50624e03b1f5498b12ea680b P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dd32a35972b347b89ceaaad2455aef5a (127.18.100.66:39781): Not found: Tablet not found: 5cc07f1d50624e03b1f5498b12ea680b
W20260130 02:59:22.057643 22179 leader_election.cc:343] T 5cc07f1d50624e03b1f5498b12ea680b P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 789dbb4a27204ffbb68d7ae40f8f5e4a (127.18.100.67:41825): Not found: Tablet not found: 5cc07f1d50624e03b1f5498b12ea680b
I20260130 02:59:22.057979 22179 leader_election.cc:304] T 5cc07f1d50624e03b1f5498b12ea680b P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 111215cc989f4203a6e1517f2291d66f; no voters: 789dbb4a27204ffbb68d7ae40f8f5e4a, dd32a35972b347b89ceaaad2455aef5a
I20260130 02:59:22.058717 23148 raft_consensus.cc:2749] T 5cc07f1d50624e03b1f5498b12ea680b P 111215cc989f4203a6e1517f2291d66f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260130 02:59:22.731261 23167 raft_consensus.cc:493] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 342be83f20a648fd849c7d23454fba08)
I20260130 02:59:22.731683 23167 raft_consensus.cc:515] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22dedcf69e00438999d3a5fedde57324" member_type: VOTER last_known_addr { host: "127.18.100.126" port: 42077 } } peers { permanent_uuid: "b01057b6c5ff4fc4b07253801881fe13" member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39157 } } peers { permanent_uuid: "342be83f20a648fd849c7d23454fba08" member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34987 } }
I20260130 02:59:22.733220 23167 leader_election.cc:290] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers b01057b6c5ff4fc4b07253801881fe13 (127.18.100.125:39157), 342be83f20a648fd849c7d23454fba08 (127.18.100.124:34987)
W20260130 02:59:22.737846 22970 leader_election.cc:336] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer b01057b6c5ff4fc4b07253801881fe13 (127.18.100.125:39157): Network error: Client connection negotiation failed: client connection to 127.18.100.125:39157: connect: Connection refused (error 111)
W20260130 02:59:22.738341 22968 leader_election.cc:336] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 342be83f20a648fd849c7d23454fba08 (127.18.100.124:34987): Network error: Client connection negotiation failed: client connection to 127.18.100.124:34987: connect: Connection refused (error 111)
I20260130 02:59:22.738698 22968 leader_election.cc:304] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22dedcf69e00438999d3a5fedde57324; no voters: 342be83f20a648fd849c7d23454fba08, b01057b6c5ff4fc4b07253801881fe13
I20260130 02:59:22.739214 23167 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260130 02:59:22.866775 22923 master_runner.cc:186] Error getting master registration for 127.18.100.125:39157: Network error: Client connection negotiation failed: client connection to 127.18.100.125:39157: connect: Connection refused (error 111),
I20260130 02:59:22.867111 22923 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260130 02:59:23.061147 23177 raft_consensus.cc:493] T be72107f1e024d5788f5d2459c85015c P 789dbb4a27204ffbb68d7ae40f8f5e4a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:59:23.061585 23177 raft_consensus.cc:515] T be72107f1e024d5788f5d2459c85015c P 789dbb4a27204ffbb68d7ae40f8f5e4a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "789dbb4a27204ffbb68d7ae40f8f5e4a" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 41825 } } peers { permanent_uuid: "dd32a35972b347b89ceaaad2455aef5a" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 39781 } } peers { permanent_uuid: "111215cc989f4203a6e1517f2291d66f" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 39751 } }
I20260130 02:59:23.063038 23177 leader_election.cc:290] T be72107f1e024d5788f5d2459c85015c P 789dbb4a27204ffbb68d7ae40f8f5e4a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dd32a35972b347b89ceaaad2455aef5a (127.18.100.66:39781), 111215cc989f4203a6e1517f2291d66f (127.18.100.65:39751)
W20260130 02:59:23.064603 22469 leader_election.cc:343] T be72107f1e024d5788f5d2459c85015c P 789dbb4a27204ffbb68d7ae40f8f5e4a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dd32a35972b347b89ceaaad2455aef5a (127.18.100.66:39781): Not found: Tablet not found: be72107f1e024d5788f5d2459c85015c
W20260130 02:59:23.065097 22468 leader_election.cc:343] T be72107f1e024d5788f5d2459c85015c P 789dbb4a27204ffbb68d7ae40f8f5e4a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 111215cc989f4203a6e1517f2291d66f (127.18.100.65:39751): Not found: Tablet not found: be72107f1e024d5788f5d2459c85015c
I20260130 02:59:23.065433 22468 leader_election.cc:304] T be72107f1e024d5788f5d2459c85015c P 789dbb4a27204ffbb68d7ae40f8f5e4a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 789dbb4a27204ffbb68d7ae40f8f5e4a; no voters: 111215cc989f4203a6e1517f2291d66f, dd32a35972b347b89ceaaad2455aef5a
I20260130 02:59:23.066002 23177 raft_consensus.cc:2749] T be72107f1e024d5788f5d2459c85015c P 789dbb4a27204ffbb68d7ae40f8f5e4a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260130 02:59:24.486284 23193 raft_consensus.cc:493] T 563d05e06b1f48b4be140219da5c9dbd P 111215cc989f4203a6e1517f2291d66f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:59:24.486961 23193 raft_consensus.cc:515] T 563d05e06b1f48b4be140219da5c9dbd P 111215cc989f4203a6e1517f2291d66f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111215cc989f4203a6e1517f2291d66f" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 39751 } } peers { permanent_uuid: "dd32a35972b347b89ceaaad2455aef5a" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 39781 } } peers { permanent_uuid: "789dbb4a27204ffbb68d7ae40f8f5e4a" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 41825 } }
I20260130 02:59:24.489054 23193 leader_election.cc:290] T 563d05e06b1f48b4be140219da5c9dbd P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dd32a35972b347b89ceaaad2455aef5a (127.18.100.66:39781), 789dbb4a27204ffbb68d7ae40f8f5e4a (127.18.100.67:41825)
W20260130 02:59:24.490968 22180 leader_election.cc:343] T 563d05e06b1f48b4be140219da5c9dbd P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dd32a35972b347b89ceaaad2455aef5a (127.18.100.66:39781): Not found: Tablet not found: 563d05e06b1f48b4be140219da5c9dbd
W20260130 02:59:24.491459 22179 leader_election.cc:343] T 563d05e06b1f48b4be140219da5c9dbd P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 789dbb4a27204ffbb68d7ae40f8f5e4a (127.18.100.67:41825): Not found: Tablet not found: 563d05e06b1f48b4be140219da5c9dbd
I20260130 02:59:24.491812 22179 leader_election.cc:304] T 563d05e06b1f48b4be140219da5c9dbd P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 111215cc989f4203a6e1517f2291d66f; no voters: 789dbb4a27204ffbb68d7ae40f8f5e4a, dd32a35972b347b89ceaaad2455aef5a
I20260130 02:59:24.492420 23193 raft_consensus.cc:2749] T 563d05e06b1f48b4be140219da5c9dbd P 111215cc989f4203a6e1517f2291d66f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260130 02:59:24.664757 23194 raft_consensus.cc:493] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 342be83f20a648fd849c7d23454fba08)
I20260130 02:59:24.665304 23194 raft_consensus.cc:515] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22dedcf69e00438999d3a5fedde57324" member_type: VOTER last_known_addr { host: "127.18.100.126" port: 42077 } } peers { permanent_uuid: "b01057b6c5ff4fc4b07253801881fe13" member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39157 } } peers { permanent_uuid: "342be83f20a648fd849c7d23454fba08" member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34987 } }
I20260130 02:59:24.667408 23194 leader_election.cc:290] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers b01057b6c5ff4fc4b07253801881fe13 (127.18.100.125:39157), 342be83f20a648fd849c7d23454fba08 (127.18.100.124:34987)
W20260130 02:59:24.669823 22970 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.18.100.125:39157: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260130 02:59:24.674031 22970 leader_election.cc:336] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer b01057b6c5ff4fc4b07253801881fe13 (127.18.100.125:39157): Network error: Client connection negotiation failed: client connection to 127.18.100.125:39157: connect: Connection refused (error 111)
W20260130 02:59:24.674626 22968 leader_election.cc:336] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 342be83f20a648fd849c7d23454fba08 (127.18.100.124:34987): Network error: Client connection negotiation failed: client connection to 127.18.100.124:34987: connect: Connection refused (error 111)
I20260130 02:59:24.675020 22968 leader_election.cc:304] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22dedcf69e00438999d3a5fedde57324; no voters: 342be83f20a648fd849c7d23454fba08, b01057b6c5ff4fc4b07253801881fe13
I20260130 02:59:24.675876 23194 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260130 02:59:24.898434 22179 consensus_peers.cc:597] T 476f8fd240bc48f1a9c47a0351ad5d27 P 111215cc989f4203a6e1517f2291d66f -> Peer 789dbb4a27204ffbb68d7ae40f8f5e4a (127.18.100.67:41825): Couldn't send request to peer 789dbb4a27204ffbb68d7ae40f8f5e4a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 476f8fd240bc48f1a9c47a0351ad5d27. This is attempt 6: this message will repeat every 5th retry.
I20260130 02:59:27.081774 23237 raft_consensus.cc:493] T be72107f1e024d5788f5d2459c85015c P 789dbb4a27204ffbb68d7ae40f8f5e4a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:59:27.082216 23237 raft_consensus.cc:515] T be72107f1e024d5788f5d2459c85015c P 789dbb4a27204ffbb68d7ae40f8f5e4a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "789dbb4a27204ffbb68d7ae40f8f5e4a" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 41825 } } peers { permanent_uuid: "dd32a35972b347b89ceaaad2455aef5a" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 39781 } } peers { permanent_uuid: "111215cc989f4203a6e1517f2291d66f" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 39751 } }
I20260130 02:59:27.084408 23237 leader_election.cc:290] T be72107f1e024d5788f5d2459c85015c P 789dbb4a27204ffbb68d7ae40f8f5e4a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dd32a35972b347b89ceaaad2455aef5a (127.18.100.66:39781), 111215cc989f4203a6e1517f2291d66f (127.18.100.65:39751)
W20260130 02:59:27.086297 22468 leader_election.cc:343] T be72107f1e024d5788f5d2459c85015c P 789dbb4a27204ffbb68d7ae40f8f5e4a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 111215cc989f4203a6e1517f2291d66f (127.18.100.65:39751): Not found: Tablet not found: be72107f1e024d5788f5d2459c85015c
W20260130 02:59:27.086815 22469 leader_election.cc:343] T be72107f1e024d5788f5d2459c85015c P 789dbb4a27204ffbb68d7ae40f8f5e4a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dd32a35972b347b89ceaaad2455aef5a (127.18.100.66:39781): Not found: Tablet not found: be72107f1e024d5788f5d2459c85015c
I20260130 02:59:27.087147 22469 leader_election.cc:304] T be72107f1e024d5788f5d2459c85015c P 789dbb4a27204ffbb68d7ae40f8f5e4a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 789dbb4a27204ffbb68d7ae40f8f5e4a; no voters: 111215cc989f4203a6e1517f2291d66f, dd32a35972b347b89ceaaad2455aef5a
I20260130 02:59:27.087754 23237 raft_consensus.cc:2749] T be72107f1e024d5788f5d2459c85015c P 789dbb4a27204ffbb68d7ae40f8f5e4a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260130 02:59:27.875200 22923 master_runner.cc:186] Error getting master registration for 127.18.100.125:39157: Network error: Client connection negotiation failed: client connection to 127.18.100.125:39157: connect: Connection refused (error 111),
I20260130 02:59:27.875840 22923 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260130 02:59:28.405917 23252 raft_consensus.cc:493] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 342be83f20a648fd849c7d23454fba08)
I20260130 02:59:28.406456 23252 raft_consensus.cc:515] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22dedcf69e00438999d3a5fedde57324" member_type: VOTER last_known_addr { host: "127.18.100.126" port: 42077 } } peers { permanent_uuid: "b01057b6c5ff4fc4b07253801881fe13" member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39157 } } peers { permanent_uuid: "342be83f20a648fd849c7d23454fba08" member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34987 } }
I20260130 02:59:28.408278 23252 leader_election.cc:290] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers b01057b6c5ff4fc4b07253801881fe13 (127.18.100.125:39157), 342be83f20a648fd849c7d23454fba08 (127.18.100.124:34987)
W20260130 02:59:28.413326 22970 leader_election.cc:336] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer b01057b6c5ff4fc4b07253801881fe13 (127.18.100.125:39157): Network error: Client connection negotiation failed: client connection to 127.18.100.125:39157: connect: Connection refused (error 111)
W20260130 02:59:28.413983 22968 leader_election.cc:336] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 342be83f20a648fd849c7d23454fba08 (127.18.100.124:34987): Network error: Client connection negotiation failed: client connection to 127.18.100.124:34987: connect: Connection refused (error 111)
I20260130 02:59:28.414301 22968 leader_election.cc:304] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22dedcf69e00438999d3a5fedde57324; no voters: 342be83f20a648fd849c7d23454fba08, b01057b6c5ff4fc4b07253801881fe13
I20260130 02:59:28.415047 23252 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260130 02:59:28.602952 23256 raft_consensus.cc:493] T 5cc07f1d50624e03b1f5498b12ea680b P 111215cc989f4203a6e1517f2291d66f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:59:28.603367 23256 raft_consensus.cc:515] T 5cc07f1d50624e03b1f5498b12ea680b P 111215cc989f4203a6e1517f2291d66f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111215cc989f4203a6e1517f2291d66f" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 39751 } } peers { permanent_uuid: "dd32a35972b347b89ceaaad2455aef5a" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 39781 } } peers { permanent_uuid: "789dbb4a27204ffbb68d7ae40f8f5e4a" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 41825 } }
I20260130 02:59:28.605579 23256 leader_election.cc:290] T 5cc07f1d50624e03b1f5498b12ea680b P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dd32a35972b347b89ceaaad2455aef5a (127.18.100.66:39781), 789dbb4a27204ffbb68d7ae40f8f5e4a (127.18.100.67:41825)
W20260130 02:59:28.607796 22180 leader_election.cc:343] T 5cc07f1d50624e03b1f5498b12ea680b P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dd32a35972b347b89ceaaad2455aef5a (127.18.100.66:39781): Not found: Tablet not found: 5cc07f1d50624e03b1f5498b12ea680b
W20260130 02:59:28.608243 22179 leader_election.cc:343] T 5cc07f1d50624e03b1f5498b12ea680b P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 789dbb4a27204ffbb68d7ae40f8f5e4a (127.18.100.67:41825): Not found: Tablet not found: 5cc07f1d50624e03b1f5498b12ea680b
I20260130 02:59:28.608639 22179 leader_election.cc:304] T 5cc07f1d50624e03b1f5498b12ea680b P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 111215cc989f4203a6e1517f2291d66f; no voters: 789dbb4a27204ffbb68d7ae40f8f5e4a, dd32a35972b347b89ceaaad2455aef5a
I20260130 02:59:28.609252 23256 raft_consensus.cc:2749] T 5cc07f1d50624e03b1f5498b12ea680b P 111215cc989f4203a6e1517f2291d66f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260130 02:59:29.732848 22321 consensus_peers.cc:597] T ce91a17790324cd884a0521ce67dde67 P dd32a35972b347b89ceaaad2455aef5a -> Peer 789dbb4a27204ffbb68d7ae40f8f5e4a (127.18.100.67:41825): Couldn't send request to peer 789dbb4a27204ffbb68d7ae40f8f5e4a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ce91a17790324cd884a0521ce67dde67. This is attempt 6: this message will repeat every 5th retry.
I20260130 02:59:30.728615 23279 raft_consensus.cc:493] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 342be83f20a648fd849c7d23454fba08)
I20260130 02:59:30.729027 23279 raft_consensus.cc:515] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22dedcf69e00438999d3a5fedde57324" member_type: VOTER last_known_addr { host: "127.18.100.126" port: 42077 } } peers { permanent_uuid: "b01057b6c5ff4fc4b07253801881fe13" member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39157 } } peers { permanent_uuid: "342be83f20a648fd849c7d23454fba08" member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34987 } }
I20260130 02:59:30.730623 23279 leader_election.cc:290] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers b01057b6c5ff4fc4b07253801881fe13 (127.18.100.125:39157), 342be83f20a648fd849c7d23454fba08 (127.18.100.124:34987)
W20260130 02:59:30.732904 22970 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.18.100.125:39157: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20260130 02:59:30.735786 22968 leader_election.cc:336] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 342be83f20a648fd849c7d23454fba08 (127.18.100.124:34987): Network error: Client connection negotiation failed: client connection to 127.18.100.124:34987: connect: Connection refused (error 111)
W20260130 02:59:30.736307 22970 leader_election.cc:336] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer b01057b6c5ff4fc4b07253801881fe13 (127.18.100.125:39157): Network error: Client connection negotiation failed: client connection to 127.18.100.125:39157: connect: Connection refused (error 111)
I20260130 02:59:30.736666 22970 leader_election.cc:304] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22dedcf69e00438999d3a5fedde57324; no voters: 342be83f20a648fd849c7d23454fba08, b01057b6c5ff4fc4b07253801881fe13
I20260130 02:59:30.737323 23279 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260130 02:59:31.473568 23289 raft_consensus.cc:493] T 563d05e06b1f48b4be140219da5c9dbd P 111215cc989f4203a6e1517f2291d66f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:59:31.473984 23289 raft_consensus.cc:515] T 563d05e06b1f48b4be140219da5c9dbd P 111215cc989f4203a6e1517f2291d66f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111215cc989f4203a6e1517f2291d66f" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 39751 } } peers { permanent_uuid: "dd32a35972b347b89ceaaad2455aef5a" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 39781 } } peers { permanent_uuid: "789dbb4a27204ffbb68d7ae40f8f5e4a" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 41825 } }
I20260130 02:59:31.475514 23289 leader_election.cc:290] T 563d05e06b1f48b4be140219da5c9dbd P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dd32a35972b347b89ceaaad2455aef5a (127.18.100.66:39781), 789dbb4a27204ffbb68d7ae40f8f5e4a (127.18.100.67:41825)
W20260130 02:59:31.477365 22180 leader_election.cc:343] T 563d05e06b1f48b4be140219da5c9dbd P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dd32a35972b347b89ceaaad2455aef5a (127.18.100.66:39781): Not found: Tablet not found: 563d05e06b1f48b4be140219da5c9dbd
W20260130 02:59:31.477880 22179 leader_election.cc:343] T 563d05e06b1f48b4be140219da5c9dbd P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 789dbb4a27204ffbb68d7ae40f8f5e4a (127.18.100.67:41825): Not found: Tablet not found: 563d05e06b1f48b4be140219da5c9dbd
I20260130 02:59:31.478243 22179 leader_election.cc:304] T 563d05e06b1f48b4be140219da5c9dbd P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 111215cc989f4203a6e1517f2291d66f; no voters: 789dbb4a27204ffbb68d7ae40f8f5e4a, dd32a35972b347b89ceaaad2455aef5a
I20260130 02:59:31.478969 23289 raft_consensus.cc:2749] T 563d05e06b1f48b4be140219da5c9dbd P 111215cc989f4203a6e1517f2291d66f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260130 02:59:31.938692 23299 raft_consensus.cc:493] T be72107f1e024d5788f5d2459c85015c P 789dbb4a27204ffbb68d7ae40f8f5e4a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:59:31.939309 23299 raft_consensus.cc:515] T be72107f1e024d5788f5d2459c85015c P 789dbb4a27204ffbb68d7ae40f8f5e4a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "789dbb4a27204ffbb68d7ae40f8f5e4a" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 41825 } } peers { permanent_uuid: "dd32a35972b347b89ceaaad2455aef5a" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 39781 } } peers { permanent_uuid: "111215cc989f4203a6e1517f2291d66f" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 39751 } }
I20260130 02:59:31.941181 23299 leader_election.cc:290] T be72107f1e024d5788f5d2459c85015c P 789dbb4a27204ffbb68d7ae40f8f5e4a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dd32a35972b347b89ceaaad2455aef5a (127.18.100.66:39781), 111215cc989f4203a6e1517f2291d66f (127.18.100.65:39751)
W20260130 02:59:31.942807 22469 leader_election.cc:343] T be72107f1e024d5788f5d2459c85015c P 789dbb4a27204ffbb68d7ae40f8f5e4a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dd32a35972b347b89ceaaad2455aef5a (127.18.100.66:39781): Not found: Tablet not found: be72107f1e024d5788f5d2459c85015c
W20260130 02:59:31.943441 22468 leader_election.cc:343] T be72107f1e024d5788f5d2459c85015c P 789dbb4a27204ffbb68d7ae40f8f5e4a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 111215cc989f4203a6e1517f2291d66f (127.18.100.65:39751): Not found: Tablet not found: be72107f1e024d5788f5d2459c85015c
I20260130 02:59:31.943737 22468 leader_election.cc:304] T be72107f1e024d5788f5d2459c85015c P 789dbb4a27204ffbb68d7ae40f8f5e4a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 789dbb4a27204ffbb68d7ae40f8f5e4a; no voters: 111215cc989f4203a6e1517f2291d66f, dd32a35972b347b89ceaaad2455aef5a
I20260130 02:59:31.944316 23299 raft_consensus.cc:2749] T be72107f1e024d5788f5d2459c85015c P 789dbb4a27204ffbb68d7ae40f8f5e4a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260130 02:59:32.594544 23307 raft_consensus.cc:493] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 342be83f20a648fd849c7d23454fba08)
I20260130 02:59:32.595032 23307 raft_consensus.cc:515] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22dedcf69e00438999d3a5fedde57324" member_type: VOTER last_known_addr { host: "127.18.100.126" port: 42077 } } peers { permanent_uuid: "b01057b6c5ff4fc4b07253801881fe13" member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39157 } } peers { permanent_uuid: "342be83f20a648fd849c7d23454fba08" member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34987 } }
I20260130 02:59:32.596876 23307 leader_election.cc:290] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers b01057b6c5ff4fc4b07253801881fe13 (127.18.100.125:39157), 342be83f20a648fd849c7d23454fba08 (127.18.100.124:34987)
W20260130 02:59:32.601694 22970 leader_election.cc:336] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer b01057b6c5ff4fc4b07253801881fe13 (127.18.100.125:39157): Network error: Client connection negotiation failed: client connection to 127.18.100.125:39157: connect: Connection refused (error 111)
W20260130 02:59:32.602216 22968 leader_election.cc:336] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 342be83f20a648fd849c7d23454fba08 (127.18.100.124:34987): Network error: Client connection negotiation failed: client connection to 127.18.100.124:34987: connect: Connection refused (error 111)
I20260130 02:59:32.602660 22968 leader_election.cc:304] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22dedcf69e00438999d3a5fedde57324; no voters: 342be83f20a648fd849c7d23454fba08, b01057b6c5ff4fc4b07253801881fe13
I20260130 02:59:32.603293 23307 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260130 02:59:32.881145 22923 master_runner.cc:186] Error getting master registration for 127.18.100.125:39157: Network error: Client connection negotiation failed: client connection to 127.18.100.125:39157: connect: Connection refused (error 111),
I20260130 02:59:32.881592 22923 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260130 02:59:34.029222 22179 consensus_peers.cc:597] T 476f8fd240bc48f1a9c47a0351ad5d27 P 111215cc989f4203a6e1517f2291d66f -> Peer 789dbb4a27204ffbb68d7ae40f8f5e4a (127.18.100.67:41825): Couldn't send request to peer 789dbb4a27204ffbb68d7ae40f8f5e4a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 476f8fd240bc48f1a9c47a0351ad5d27. This is attempt 11: this message will repeat every 5th retry.
I20260130 02:59:37.888242 22923 master_runner.cc:186] Error getting master registration for 127.18.100.125:39157: Network error: Client connection negotiation failed: client connection to 127.18.100.125:39157: connect: Connection refused (error 111),
I20260130 02:59:37.888674 22923 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260130 02:59:39.443919 22321 consensus_peers.cc:597] T ce91a17790324cd884a0521ce67dde67 P dd32a35972b347b89ceaaad2455aef5a -> Peer 789dbb4a27204ffbb68d7ae40f8f5e4a (127.18.100.67:41825): Couldn't send request to peer 789dbb4a27204ffbb68d7ae40f8f5e4a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ce91a17790324cd884a0521ce67dde67. This is attempt 11: this message will repeat every 5th retry.
I20260130 02:59:39.587157 23395 raft_consensus.cc:493] T 5cc07f1d50624e03b1f5498b12ea680b P 111215cc989f4203a6e1517f2291d66f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:59:39.587771 23395 raft_consensus.cc:515] T 5cc07f1d50624e03b1f5498b12ea680b P 111215cc989f4203a6e1517f2291d66f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111215cc989f4203a6e1517f2291d66f" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 39751 } } peers { permanent_uuid: "dd32a35972b347b89ceaaad2455aef5a" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 39781 } } peers { permanent_uuid: "789dbb4a27204ffbb68d7ae40f8f5e4a" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 41825 } }
I20260130 02:59:39.589686 23395 leader_election.cc:290] T 5cc07f1d50624e03b1f5498b12ea680b P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dd32a35972b347b89ceaaad2455aef5a (127.18.100.66:39781), 789dbb4a27204ffbb68d7ae40f8f5e4a (127.18.100.67:41825)
W20260130 02:59:39.591755 22180 leader_election.cc:343] T 5cc07f1d50624e03b1f5498b12ea680b P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dd32a35972b347b89ceaaad2455aef5a (127.18.100.66:39781): Not found: Tablet not found: 5cc07f1d50624e03b1f5498b12ea680b
W20260130 02:59:39.592211 22179 leader_election.cc:343] T 5cc07f1d50624e03b1f5498b12ea680b P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 789dbb4a27204ffbb68d7ae40f8f5e4a (127.18.100.67:41825): Not found: Tablet not found: 5cc07f1d50624e03b1f5498b12ea680b
I20260130 02:59:39.592547 22179 leader_election.cc:304] T 5cc07f1d50624e03b1f5498b12ea680b P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 111215cc989f4203a6e1517f2291d66f; no voters: 789dbb4a27204ffbb68d7ae40f8f5e4a, dd32a35972b347b89ceaaad2455aef5a
I20260130 02:59:39.593181 23395 raft_consensus.cc:2749] T 5cc07f1d50624e03b1f5498b12ea680b P 111215cc989f4203a6e1517f2291d66f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260130 02:59:40.743793 23417 raft_consensus.cc:493] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 342be83f20a648fd849c7d23454fba08)
I20260130 02:59:40.744468 23417 raft_consensus.cc:515] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22dedcf69e00438999d3a5fedde57324" member_type: VOTER last_known_addr { host: "127.18.100.126" port: 42077 } } peers { permanent_uuid: "b01057b6c5ff4fc4b07253801881fe13" member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39157 } } peers { permanent_uuid: "342be83f20a648fd849c7d23454fba08" member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34987 } }
I20260130 02:59:40.746229 23417 leader_election.cc:290] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers b01057b6c5ff4fc4b07253801881fe13 (127.18.100.125:39157), 342be83f20a648fd849c7d23454fba08 (127.18.100.124:34987)
W20260130 02:59:40.748359 22970 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.18.100.125:39157: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20260130 02:59:40.751467 22968 leader_election.cc:336] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 342be83f20a648fd849c7d23454fba08 (127.18.100.124:34987): Network error: Client connection negotiation failed: client connection to 127.18.100.124:34987: connect: Connection refused (error 111)
W20260130 02:59:40.751984 22970 leader_election.cc:336] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer b01057b6c5ff4fc4b07253801881fe13 (127.18.100.125:39157): Network error: Client connection negotiation failed: client connection to 127.18.100.125:39157: connect: Connection refused (error 111)
I20260130 02:59:40.752269 22970 leader_election.cc:304] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22dedcf69e00438999d3a5fedde57324; no voters: 342be83f20a648fd849c7d23454fba08, b01057b6c5ff4fc4b07253801881fe13
I20260130 02:59:40.752869 23417 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260130 02:59:40.946389 23414 raft_consensus.cc:493] T 563d05e06b1f48b4be140219da5c9dbd P 111215cc989f4203a6e1517f2291d66f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:59:40.947055 23414 raft_consensus.cc:515] T 563d05e06b1f48b4be140219da5c9dbd P 111215cc989f4203a6e1517f2291d66f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111215cc989f4203a6e1517f2291d66f" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 39751 } } peers { permanent_uuid: "dd32a35972b347b89ceaaad2455aef5a" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 39781 } } peers { permanent_uuid: "789dbb4a27204ffbb68d7ae40f8f5e4a" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 41825 } }
I20260130 02:59:40.949688 23414 leader_election.cc:290] T 563d05e06b1f48b4be140219da5c9dbd P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dd32a35972b347b89ceaaad2455aef5a (127.18.100.66:39781), 789dbb4a27204ffbb68d7ae40f8f5e4a (127.18.100.67:41825)
W20260130 02:59:40.951297 22180 leader_election.cc:343] T 563d05e06b1f48b4be140219da5c9dbd P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dd32a35972b347b89ceaaad2455aef5a (127.18.100.66:39781): Not found: Tablet not found: 563d05e06b1f48b4be140219da5c9dbd
W20260130 02:59:40.951870 22179 leader_election.cc:343] T 563d05e06b1f48b4be140219da5c9dbd P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 789dbb4a27204ffbb68d7ae40f8f5e4a (127.18.100.67:41825): Not found: Tablet not found: 563d05e06b1f48b4be140219da5c9dbd
I20260130 02:59:40.952258 22179 leader_election.cc:304] T 563d05e06b1f48b4be140219da5c9dbd P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 111215cc989f4203a6e1517f2291d66f; no voters: 789dbb4a27204ffbb68d7ae40f8f5e4a, dd32a35972b347b89ceaaad2455aef5a
I20260130 02:59:40.952994 23414 raft_consensus.cc:2749] T 563d05e06b1f48b4be140219da5c9dbd P 111215cc989f4203a6e1517f2291d66f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260130 02:59:42.894109 22923 master_runner.cc:186] Error getting master registration for 127.18.100.125:39157: Network error: Client connection negotiation failed: client connection to 127.18.100.125:39157: connect: Connection refused (error 111),
I20260130 02:59:42.894495 22923 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260130 02:59:44.574935 22179 consensus_peers.cc:597] T 476f8fd240bc48f1a9c47a0351ad5d27 P 111215cc989f4203a6e1517f2291d66f -> Peer 789dbb4a27204ffbb68d7ae40f8f5e4a (127.18.100.67:41825): Couldn't send request to peer 789dbb4a27204ffbb68d7ae40f8f5e4a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 476f8fd240bc48f1a9c47a0351ad5d27. This is attempt 16: this message will repeat every 5th retry.
I20260130 02:59:45.545563 23480 raft_consensus.cc:493] T be72107f1e024d5788f5d2459c85015c P 789dbb4a27204ffbb68d7ae40f8f5e4a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:59:45.546128 23480 raft_consensus.cc:515] T be72107f1e024d5788f5d2459c85015c P 789dbb4a27204ffbb68d7ae40f8f5e4a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "789dbb4a27204ffbb68d7ae40f8f5e4a" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 41825 } } peers { permanent_uuid: "dd32a35972b347b89ceaaad2455aef5a" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 39781 } } peers { permanent_uuid: "111215cc989f4203a6e1517f2291d66f" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 39751 } }
I20260130 02:59:45.548274 23480 leader_election.cc:290] T be72107f1e024d5788f5d2459c85015c P 789dbb4a27204ffbb68d7ae40f8f5e4a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dd32a35972b347b89ceaaad2455aef5a (127.18.100.66:39781), 111215cc989f4203a6e1517f2291d66f (127.18.100.65:39751)
W20260130 02:59:45.549922 22469 leader_election.cc:343] T be72107f1e024d5788f5d2459c85015c P 789dbb4a27204ffbb68d7ae40f8f5e4a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dd32a35972b347b89ceaaad2455aef5a (127.18.100.66:39781): Not found: Tablet not found: be72107f1e024d5788f5d2459c85015c
W20260130 02:59:45.550499 22468 leader_election.cc:343] T be72107f1e024d5788f5d2459c85015c P 789dbb4a27204ffbb68d7ae40f8f5e4a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 111215cc989f4203a6e1517f2291d66f (127.18.100.65:39751): Not found: Tablet not found: be72107f1e024d5788f5d2459c85015c
I20260130 02:59:45.550830 22468 leader_election.cc:304] T be72107f1e024d5788f5d2459c85015c P 789dbb4a27204ffbb68d7ae40f8f5e4a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 789dbb4a27204ffbb68d7ae40f8f5e4a; no voters: 111215cc989f4203a6e1517f2291d66f, dd32a35972b347b89ceaaad2455aef5a
I20260130 02:59:45.551434 23480 raft_consensus.cc:2749] T be72107f1e024d5788f5d2459c85015c P 789dbb4a27204ffbb68d7ae40f8f5e4a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260130 02:59:46.720356 23493 raft_consensus.cc:493] T 5cc07f1d50624e03b1f5498b12ea680b P 111215cc989f4203a6e1517f2291d66f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:59:46.721042 23493 raft_consensus.cc:515] T 5cc07f1d50624e03b1f5498b12ea680b P 111215cc989f4203a6e1517f2291d66f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111215cc989f4203a6e1517f2291d66f" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 39751 } } peers { permanent_uuid: "dd32a35972b347b89ceaaad2455aef5a" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 39781 } } peers { permanent_uuid: "789dbb4a27204ffbb68d7ae40f8f5e4a" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 41825 } }
I20260130 02:59:46.723742 23493 leader_election.cc:290] T 5cc07f1d50624e03b1f5498b12ea680b P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dd32a35972b347b89ceaaad2455aef5a (127.18.100.66:39781), 789dbb4a27204ffbb68d7ae40f8f5e4a (127.18.100.67:41825)
W20260130 02:59:46.725777 22180 leader_election.cc:343] T 5cc07f1d50624e03b1f5498b12ea680b P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dd32a35972b347b89ceaaad2455aef5a (127.18.100.66:39781): Not found: Tablet not found: 5cc07f1d50624e03b1f5498b12ea680b
W20260130 02:59:46.726271 22179 leader_election.cc:343] T 5cc07f1d50624e03b1f5498b12ea680b P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 789dbb4a27204ffbb68d7ae40f8f5e4a (127.18.100.67:41825): Not found: Tablet not found: 5cc07f1d50624e03b1f5498b12ea680b
I20260130 02:59:46.726711 22179 leader_election.cc:304] T 5cc07f1d50624e03b1f5498b12ea680b P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 111215cc989f4203a6e1517f2291d66f; no voters: 789dbb4a27204ffbb68d7ae40f8f5e4a, dd32a35972b347b89ceaaad2455aef5a
I20260130 02:59:46.727404 23493 raft_consensus.cc:2749] T 5cc07f1d50624e03b1f5498b12ea680b P 111215cc989f4203a6e1517f2291d66f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260130 02:59:47.901525 22923 master_runner.cc:186] Error getting master registration for 127.18.100.125:39157: Network error: Client connection negotiation failed: client connection to 127.18.100.125:39157: connect: Connection refused (error 111),
I20260130 02:59:47.902019 22923 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260130 02:59:49.164163 22321 consensus_peers.cc:597] T ce91a17790324cd884a0521ce67dde67 P dd32a35972b347b89ceaaad2455aef5a -> Peer 789dbb4a27204ffbb68d7ae40f8f5e4a (127.18.100.67:41825): Couldn't send request to peer 789dbb4a27204ffbb68d7ae40f8f5e4a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ce91a17790324cd884a0521ce67dde67. This is attempt 16: this message will repeat every 5th retry.
I20260130 02:59:50.022230 23527 raft_consensus.cc:493] T 5cc07f1d50624e03b1f5498b12ea680b P 111215cc989f4203a6e1517f2291d66f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:59:50.022738 23527 raft_consensus.cc:515] T 5cc07f1d50624e03b1f5498b12ea680b P 111215cc989f4203a6e1517f2291d66f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111215cc989f4203a6e1517f2291d66f" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 39751 } } peers { permanent_uuid: "dd32a35972b347b89ceaaad2455aef5a" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 39781 } } peers { permanent_uuid: "789dbb4a27204ffbb68d7ae40f8f5e4a" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 41825 } }
I20260130 02:59:50.024245 23527 leader_election.cc:290] T 5cc07f1d50624e03b1f5498b12ea680b P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dd32a35972b347b89ceaaad2455aef5a (127.18.100.66:39781), 789dbb4a27204ffbb68d7ae40f8f5e4a (127.18.100.67:41825)
W20260130 02:59:50.026343 22179 leader_election.cc:343] T 5cc07f1d50624e03b1f5498b12ea680b P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 789dbb4a27204ffbb68d7ae40f8f5e4a (127.18.100.67:41825): Not found: Tablet not found: 5cc07f1d50624e03b1f5498b12ea680b
W20260130 02:59:50.026861 22180 leader_election.cc:343] T 5cc07f1d50624e03b1f5498b12ea680b P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dd32a35972b347b89ceaaad2455aef5a (127.18.100.66:39781): Not found: Tablet not found: 5cc07f1d50624e03b1f5498b12ea680b
I20260130 02:59:50.027166 22180 leader_election.cc:304] T 5cc07f1d50624e03b1f5498b12ea680b P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 111215cc989f4203a6e1517f2291d66f; no voters: 789dbb4a27204ffbb68d7ae40f8f5e4a, dd32a35972b347b89ceaaad2455aef5a
I20260130 02:59:50.027751 23527 raft_consensus.cc:2749] T 5cc07f1d50624e03b1f5498b12ea680b P 111215cc989f4203a6e1517f2291d66f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260130 02:59:52.907658 22923 master_runner.cc:186] Error getting master registration for 127.18.100.125:39157: Network error: Client connection negotiation failed: client connection to 127.18.100.125:39157: connect: Connection refused (error 111),
I20260130 02:59:52.908031 22923 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260130 02:59:53.498862 23576 raft_consensus.cc:493] T 563d05e06b1f48b4be140219da5c9dbd P 111215cc989f4203a6e1517f2291d66f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:59:53.499692 23576 raft_consensus.cc:515] T 563d05e06b1f48b4be140219da5c9dbd P 111215cc989f4203a6e1517f2291d66f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111215cc989f4203a6e1517f2291d66f" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 39751 } } peers { permanent_uuid: "dd32a35972b347b89ceaaad2455aef5a" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 39781 } } peers { permanent_uuid: "789dbb4a27204ffbb68d7ae40f8f5e4a" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 41825 } }
I20260130 02:59:53.501679 23576 leader_election.cc:290] T 563d05e06b1f48b4be140219da5c9dbd P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dd32a35972b347b89ceaaad2455aef5a (127.18.100.66:39781), 789dbb4a27204ffbb68d7ae40f8f5e4a (127.18.100.67:41825)
W20260130 02:59:53.503135 22180 leader_election.cc:343] T 563d05e06b1f48b4be140219da5c9dbd P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dd32a35972b347b89ceaaad2455aef5a (127.18.100.66:39781): Not found: Tablet not found: 563d05e06b1f48b4be140219da5c9dbd
W20260130 02:59:53.503690 22179 leader_election.cc:343] T 563d05e06b1f48b4be140219da5c9dbd P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 789dbb4a27204ffbb68d7ae40f8f5e4a (127.18.100.67:41825): Not found: Tablet not found: 563d05e06b1f48b4be140219da5c9dbd
I20260130 02:59:53.504069 22179 leader_election.cc:304] T 563d05e06b1f48b4be140219da5c9dbd P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 111215cc989f4203a6e1517f2291d66f; no voters: 789dbb4a27204ffbb68d7ae40f8f5e4a, dd32a35972b347b89ceaaad2455aef5a
I20260130 02:59:53.504662 23576 raft_consensus.cc:2749] T 563d05e06b1f48b4be140219da5c9dbd P 111215cc989f4203a6e1517f2291d66f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260130 02:59:54.276050 22179 consensus_peers.cc:597] T 476f8fd240bc48f1a9c47a0351ad5d27 P 111215cc989f4203a6e1517f2291d66f -> Peer 789dbb4a27204ffbb68d7ae40f8f5e4a (127.18.100.67:41825): Couldn't send request to peer 789dbb4a27204ffbb68d7ae40f8f5e4a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 476f8fd240bc48f1a9c47a0351ad5d27. This is attempt 21: this message will repeat every 5th retry.
I20260130 02:59:54.951582 23598 raft_consensus.cc:493] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 342be83f20a648fd849c7d23454fba08)
I20260130 02:59:54.952077 23598 raft_consensus.cc:515] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22dedcf69e00438999d3a5fedde57324" member_type: VOTER last_known_addr { host: "127.18.100.126" port: 42077 } } peers { permanent_uuid: "b01057b6c5ff4fc4b07253801881fe13" member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39157 } } peers { permanent_uuid: "342be83f20a648fd849c7d23454fba08" member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34987 } }
I20260130 02:59:54.953658 23598 leader_election.cc:290] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers b01057b6c5ff4fc4b07253801881fe13 (127.18.100.125:39157), 342be83f20a648fd849c7d23454fba08 (127.18.100.124:34987)
W20260130 02:59:54.956187 22970 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.18.100.125:39157: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20260130 02:59:54.959764 22970 leader_election.cc:336] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer b01057b6c5ff4fc4b07253801881fe13 (127.18.100.125:39157): Network error: Client connection negotiation failed: client connection to 127.18.100.125:39157: connect: Connection refused (error 111)
W20260130 02:59:54.960386 22968 leader_election.cc:336] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 342be83f20a648fd849c7d23454fba08 (127.18.100.124:34987): Network error: Client connection negotiation failed: client connection to 127.18.100.124:34987: connect: Connection refused (error 111)
I20260130 02:59:54.960779 22968 leader_election.cc:304] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22dedcf69e00438999d3a5fedde57324; no voters: 342be83f20a648fd849c7d23454fba08, b01057b6c5ff4fc4b07253801881fe13
I20260130 02:59:54.961485 23598 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260130 02:59:55.352509 23608 raft_consensus.cc:493] T be72107f1e024d5788f5d2459c85015c P 789dbb4a27204ffbb68d7ae40f8f5e4a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 02:59:55.353160 23608 raft_consensus.cc:515] T be72107f1e024d5788f5d2459c85015c P 789dbb4a27204ffbb68d7ae40f8f5e4a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "789dbb4a27204ffbb68d7ae40f8f5e4a" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 41825 } } peers { permanent_uuid: "dd32a35972b347b89ceaaad2455aef5a" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 39781 } } peers { permanent_uuid: "111215cc989f4203a6e1517f2291d66f" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 39751 } }
I20260130 02:59:55.355123 23608 leader_election.cc:290] T be72107f1e024d5788f5d2459c85015c P 789dbb4a27204ffbb68d7ae40f8f5e4a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dd32a35972b347b89ceaaad2455aef5a (127.18.100.66:39781), 111215cc989f4203a6e1517f2291d66f (127.18.100.65:39751)
W20260130 02:59:55.357210 22469 leader_election.cc:343] T be72107f1e024d5788f5d2459c85015c P 789dbb4a27204ffbb68d7ae40f8f5e4a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dd32a35972b347b89ceaaad2455aef5a (127.18.100.66:39781): Not found: Tablet not found: be72107f1e024d5788f5d2459c85015c
W20260130 02:59:55.357878 22468 leader_election.cc:343] T be72107f1e024d5788f5d2459c85015c P 789dbb4a27204ffbb68d7ae40f8f5e4a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 111215cc989f4203a6e1517f2291d66f (127.18.100.65:39751): Not found: Tablet not found: be72107f1e024d5788f5d2459c85015c
I20260130 02:59:55.358242 22468 leader_election.cc:304] T be72107f1e024d5788f5d2459c85015c P 789dbb4a27204ffbb68d7ae40f8f5e4a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 789dbb4a27204ffbb68d7ae40f8f5e4a; no voters: 111215cc989f4203a6e1517f2291d66f, dd32a35972b347b89ceaaad2455aef5a
I20260130 02:59:55.358935 23608 raft_consensus.cc:2749] T be72107f1e024d5788f5d2459c85015c P 789dbb4a27204ffbb68d7ae40f8f5e4a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260130 02:59:57.914705 22923 master_runner.cc:186] Error getting master registration for 127.18.100.125:39157: Network error: Client connection negotiation failed: client connection to 127.18.100.125:39157: connect: Connection refused (error 111),
I20260130 02:59:57.915244 22923 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260130 02:59:59.663203 22321 consensus_peers.cc:597] T ce91a17790324cd884a0521ce67dde67 P dd32a35972b347b89ceaaad2455aef5a -> Peer 789dbb4a27204ffbb68d7ae40f8f5e4a (127.18.100.67:41825): Couldn't send request to peer 789dbb4a27204ffbb68d7ae40f8f5e4a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ce91a17790324cd884a0521ce67dde67. This is attempt 21: this message will repeat every 5th retry.
W20260130 03:00:01.342062 23020 debug-util.cc:398] Leaking SignalData structure 0x7b08000747a0 after lost signal to thread 22933
W20260130 03:00:01.342864 23020 debug-util.cc:398] Leaking SignalData structure 0x7b08000b1b80 after lost signal to thread 23023
I20260130 03:00:02.041766 23691 raft_consensus.cc:493] T 563d05e06b1f48b4be140219da5c9dbd P 111215cc989f4203a6e1517f2291d66f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 03:00:02.042310 23691 raft_consensus.cc:515] T 563d05e06b1f48b4be140219da5c9dbd P 111215cc989f4203a6e1517f2291d66f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111215cc989f4203a6e1517f2291d66f" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 39751 } } peers { permanent_uuid: "dd32a35972b347b89ceaaad2455aef5a" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 39781 } } peers { permanent_uuid: "789dbb4a27204ffbb68d7ae40f8f5e4a" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 41825 } }
I20260130 03:00:02.044376 23691 leader_election.cc:290] T 563d05e06b1f48b4be140219da5c9dbd P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dd32a35972b347b89ceaaad2455aef5a (127.18.100.66:39781), 789dbb4a27204ffbb68d7ae40f8f5e4a (127.18.100.67:41825)
W20260130 03:00:02.046312 22179 leader_election.cc:343] T 563d05e06b1f48b4be140219da5c9dbd P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 789dbb4a27204ffbb68d7ae40f8f5e4a (127.18.100.67:41825): Not found: Tablet not found: 563d05e06b1f48b4be140219da5c9dbd
W20260130 03:00:02.046800 22180 leader_election.cc:343] T 563d05e06b1f48b4be140219da5c9dbd P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dd32a35972b347b89ceaaad2455aef5a (127.18.100.66:39781): Not found: Tablet not found: 563d05e06b1f48b4be140219da5c9dbd
I20260130 03:00:02.047192 22180 leader_election.cc:304] T 563d05e06b1f48b4be140219da5c9dbd P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 111215cc989f4203a6e1517f2291d66f; no voters: 789dbb4a27204ffbb68d7ae40f8f5e4a, dd32a35972b347b89ceaaad2455aef5a
I20260130 03:00:02.047937 23691 raft_consensus.cc:2749] T 563d05e06b1f48b4be140219da5c9dbd P 111215cc989f4203a6e1517f2291d66f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260130 03:00:02.936414 22923 master_runner.cc:186] Error getting master registration for 127.18.100.125:39157: Network error: Client connection negotiation failed: client connection to 127.18.100.125:39157: connect: Connection refused (error 111),
I20260130 03:00:02.936751 22923 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260130 03:00:03.632838 23707 raft_consensus.cc:493] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 342be83f20a648fd849c7d23454fba08)
I20260130 03:00:03.633435 23707 raft_consensus.cc:515] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22dedcf69e00438999d3a5fedde57324" member_type: VOTER last_known_addr { host: "127.18.100.126" port: 42077 } } peers { permanent_uuid: "b01057b6c5ff4fc4b07253801881fe13" member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39157 } } peers { permanent_uuid: "342be83f20a648fd849c7d23454fba08" member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34987 } }
I20260130 03:00:03.635233 23707 leader_election.cc:290] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers b01057b6c5ff4fc4b07253801881fe13 (127.18.100.125:39157), 342be83f20a648fd849c7d23454fba08 (127.18.100.124:34987)
W20260130 03:00:03.637567 22970 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.18.100.125:39157: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20260130 03:00:03.640971 22970 leader_election.cc:336] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer b01057b6c5ff4fc4b07253801881fe13 (127.18.100.125:39157): Network error: Client connection negotiation failed: client connection to 127.18.100.125:39157: connect: Connection refused (error 111)
W20260130 03:00:03.641697 22968 leader_election.cc:336] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 342be83f20a648fd849c7d23454fba08 (127.18.100.124:34987): Network error: Client connection negotiation failed: client connection to 127.18.100.124:34987: connect: Connection refused (error 111)
I20260130 03:00:03.642089 22968 leader_election.cc:304] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22dedcf69e00438999d3a5fedde57324; no voters: 342be83f20a648fd849c7d23454fba08, b01057b6c5ff4fc4b07253801881fe13
I20260130 03:00:03.642989 23707 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260130 03:00:03.759053 22179 consensus_peers.cc:597] T 476f8fd240bc48f1a9c47a0351ad5d27 P 111215cc989f4203a6e1517f2291d66f -> Peer 789dbb4a27204ffbb68d7ae40f8f5e4a (127.18.100.67:41825): Couldn't send request to peer 789dbb4a27204ffbb68d7ae40f8f5e4a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 476f8fd240bc48f1a9c47a0351ad5d27. This is attempt 26: this message will repeat every 5th retry.
I20260130 03:00:07.944041 22923 master_runner.cc:186] Error getting master registration for 127.18.100.125:39157: Network error: Client connection negotiation failed: client connection to 127.18.100.125:39157: connect: Connection refused (error 111),
I20260130 03:00:07.944558 22923 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260130 03:00:08.562449 23780 raft_consensus.cc:493] T 5cc07f1d50624e03b1f5498b12ea680b P 111215cc989f4203a6e1517f2291d66f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 03:00:08.563195 23780 raft_consensus.cc:515] T 5cc07f1d50624e03b1f5498b12ea680b P 111215cc989f4203a6e1517f2291d66f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111215cc989f4203a6e1517f2291d66f" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 39751 } } peers { permanent_uuid: "dd32a35972b347b89ceaaad2455aef5a" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 39781 } } peers { permanent_uuid: "789dbb4a27204ffbb68d7ae40f8f5e4a" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 41825 } }
I20260130 03:00:08.565196 23780 leader_election.cc:290] T 5cc07f1d50624e03b1f5498b12ea680b P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dd32a35972b347b89ceaaad2455aef5a (127.18.100.66:39781), 789dbb4a27204ffbb68d7ae40f8f5e4a (127.18.100.67:41825)
W20260130 03:00:08.567369 22179 leader_election.cc:343] T 5cc07f1d50624e03b1f5498b12ea680b P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 789dbb4a27204ffbb68d7ae40f8f5e4a (127.18.100.67:41825): Not found: Tablet not found: 5cc07f1d50624e03b1f5498b12ea680b
W20260130 03:00:08.568086 22180 leader_election.cc:343] T 5cc07f1d50624e03b1f5498b12ea680b P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dd32a35972b347b89ceaaad2455aef5a (127.18.100.66:39781): Not found: Tablet not found: 5cc07f1d50624e03b1f5498b12ea680b
I20260130 03:00:08.568441 22180 leader_election.cc:304] T 5cc07f1d50624e03b1f5498b12ea680b P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 111215cc989f4203a6e1517f2291d66f; no voters: 789dbb4a27204ffbb68d7ae40f8f5e4a, dd32a35972b347b89ceaaad2455aef5a
I20260130 03:00:08.569129 23780 raft_consensus.cc:2749] T 5cc07f1d50624e03b1f5498b12ea680b P 111215cc989f4203a6e1517f2291d66f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260130 03:00:09.123636 22321 consensus_peers.cc:597] T ce91a17790324cd884a0521ce67dde67 P dd32a35972b347b89ceaaad2455aef5a -> Peer 789dbb4a27204ffbb68d7ae40f8f5e4a (127.18.100.67:41825): Couldn't send request to peer 789dbb4a27204ffbb68d7ae40f8f5e4a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ce91a17790324cd884a0521ce67dde67. This is attempt 26: this message will repeat every 5th retry.
W20260130 03:00:09.628422 22438 heartbeater.cc:646] Failed to heartbeat to 127.18.100.124:34987 (124 consecutive failures): Network error: Failed to ping master at 127.18.100.124:34987: Client connection negotiation failed: client connection to 127.18.100.124:34987: connect: Connection refused (error 111)
W20260130 03:00:09.738332 22580 heartbeater.cc:646] Failed to heartbeat to 127.18.100.125:39157 (115 consecutive failures): Network error: Failed to ping master at 127.18.100.125:39157: Client connection negotiation failed: client connection to 127.18.100.125:39157: connect: Connection refused (error 111)
W20260130 03:00:09.929906 22293 heartbeater.cc:646] Failed to heartbeat to 127.18.100.125:39157 (142 consecutive failures): Network error: Failed to ping master at 127.18.100.125:39157: Client connection negotiation failed: client connection to 127.18.100.125:39157: connect: Connection refused (error 111)
I20260130 03:00:10.348976 23792 raft_consensus.cc:493] T 563d05e06b1f48b4be140219da5c9dbd P 111215cc989f4203a6e1517f2291d66f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 03:00:10.349561 23792 raft_consensus.cc:515] T 563d05e06b1f48b4be140219da5c9dbd P 111215cc989f4203a6e1517f2291d66f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111215cc989f4203a6e1517f2291d66f" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 39751 } } peers { permanent_uuid: "dd32a35972b347b89ceaaad2455aef5a" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 39781 } } peers { permanent_uuid: "789dbb4a27204ffbb68d7ae40f8f5e4a" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 41825 } }
I20260130 03:00:10.351665 23792 leader_election.cc:290] T 563d05e06b1f48b4be140219da5c9dbd P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dd32a35972b347b89ceaaad2455aef5a (127.18.100.66:39781), 789dbb4a27204ffbb68d7ae40f8f5e4a (127.18.100.67:41825)
W20260130 03:00:10.353554 22180 leader_election.cc:343] T 563d05e06b1f48b4be140219da5c9dbd P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dd32a35972b347b89ceaaad2455aef5a (127.18.100.66:39781): Not found: Tablet not found: 563d05e06b1f48b4be140219da5c9dbd
W20260130 03:00:10.354023 22179 leader_election.cc:343] T 563d05e06b1f48b4be140219da5c9dbd P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 789dbb4a27204ffbb68d7ae40f8f5e4a (127.18.100.67:41825): Not found: Tablet not found: 563d05e06b1f48b4be140219da5c9dbd
I20260130 03:00:10.354424 22179 leader_election.cc:304] T 563d05e06b1f48b4be140219da5c9dbd P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 111215cc989f4203a6e1517f2291d66f; no voters: 789dbb4a27204ffbb68d7ae40f8f5e4a, dd32a35972b347b89ceaaad2455aef5a
I20260130 03:00:10.355266 23792 raft_consensus.cc:2749] T 563d05e06b1f48b4be140219da5c9dbd P 111215cc989f4203a6e1517f2291d66f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260130 03:00:11.500156 23814 raft_consensus.cc:493] T be72107f1e024d5788f5d2459c85015c P 789dbb4a27204ffbb68d7ae40f8f5e4a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 03:00:11.500685 23814 raft_consensus.cc:515] T be72107f1e024d5788f5d2459c85015c P 789dbb4a27204ffbb68d7ae40f8f5e4a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "789dbb4a27204ffbb68d7ae40f8f5e4a" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 41825 } } peers { permanent_uuid: "dd32a35972b347b89ceaaad2455aef5a" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 39781 } } peers { permanent_uuid: "111215cc989f4203a6e1517f2291d66f" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 39751 } }
I20260130 03:00:11.502755 23814 leader_election.cc:290] T be72107f1e024d5788f5d2459c85015c P 789dbb4a27204ffbb68d7ae40f8f5e4a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dd32a35972b347b89ceaaad2455aef5a (127.18.100.66:39781), 111215cc989f4203a6e1517f2291d66f (127.18.100.65:39751)
W20260130 03:00:11.504524 22469 leader_election.cc:343] T be72107f1e024d5788f5d2459c85015c P 789dbb4a27204ffbb68d7ae40f8f5e4a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dd32a35972b347b89ceaaad2455aef5a (127.18.100.66:39781): Not found: Tablet not found: be72107f1e024d5788f5d2459c85015c
W20260130 03:00:11.505115 22468 leader_election.cc:343] T be72107f1e024d5788f5d2459c85015c P 789dbb4a27204ffbb68d7ae40f8f5e4a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 111215cc989f4203a6e1517f2291d66f (127.18.100.65:39751): Not found: Tablet not found: be72107f1e024d5788f5d2459c85015c
I20260130 03:00:11.505414 22468 leader_election.cc:304] T be72107f1e024d5788f5d2459c85015c P 789dbb4a27204ffbb68d7ae40f8f5e4a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 789dbb4a27204ffbb68d7ae40f8f5e4a; no voters: 111215cc989f4203a6e1517f2291d66f, dd32a35972b347b89ceaaad2455aef5a
I20260130 03:00:11.506012 23814 raft_consensus.cc:2749] T be72107f1e024d5788f5d2459c85015c P 789dbb4a27204ffbb68d7ae40f8f5e4a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260130 03:00:12.950270 22923 master_runner.cc:186] Error getting master registration for 127.18.100.125:39157: Network error: Client connection negotiation failed: client connection to 127.18.100.125:39157: connect: Connection refused (error 111),
I20260130 03:00:12.950752 22923 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260130 03:00:14.129154 22179 consensus_peers.cc:597] T 476f8fd240bc48f1a9c47a0351ad5d27 P 111215cc989f4203a6e1517f2291d66f -> Peer 789dbb4a27204ffbb68d7ae40f8f5e4a (127.18.100.67:41825): Couldn't send request to peer 789dbb4a27204ffbb68d7ae40f8f5e4a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 476f8fd240bc48f1a9c47a0351ad5d27. This is attempt 31: this message will repeat every 5th retry.
I20260130 03:00:15.273769 23859 raft_consensus.cc:493] T 563d05e06b1f48b4be140219da5c9dbd P 111215cc989f4203a6e1517f2291d66f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 03:00:15.274362 23859 raft_consensus.cc:515] T 563d05e06b1f48b4be140219da5c9dbd P 111215cc989f4203a6e1517f2291d66f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111215cc989f4203a6e1517f2291d66f" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 39751 } } peers { permanent_uuid: "dd32a35972b347b89ceaaad2455aef5a" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 39781 } } peers { permanent_uuid: "789dbb4a27204ffbb68d7ae40f8f5e4a" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 41825 } }
I20260130 03:00:15.276340 23859 leader_election.cc:290] T 563d05e06b1f48b4be140219da5c9dbd P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dd32a35972b347b89ceaaad2455aef5a (127.18.100.66:39781), 789dbb4a27204ffbb68d7ae40f8f5e4a (127.18.100.67:41825)
W20260130 03:00:15.278416 22180 leader_election.cc:343] T 563d05e06b1f48b4be140219da5c9dbd P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dd32a35972b347b89ceaaad2455aef5a (127.18.100.66:39781): Not found: Tablet not found: 563d05e06b1f48b4be140219da5c9dbd
W20260130 03:00:15.278945 22179 leader_election.cc:343] T 563d05e06b1f48b4be140219da5c9dbd P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 789dbb4a27204ffbb68d7ae40f8f5e4a (127.18.100.67:41825): Not found: Tablet not found: 563d05e06b1f48b4be140219da5c9dbd
I20260130 03:00:15.279292 22179 leader_election.cc:304] T 563d05e06b1f48b4be140219da5c9dbd P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 111215cc989f4203a6e1517f2291d66f; no voters: 789dbb4a27204ffbb68d7ae40f8f5e4a, dd32a35972b347b89ceaaad2455aef5a
I20260130 03:00:15.279945 23859 raft_consensus.cc:2749] T 563d05e06b1f48b4be140219da5c9dbd P 111215cc989f4203a6e1517f2291d66f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260130 03:00:17.958076 22923 master_runner.cc:186] Error getting master registration for 127.18.100.125:39157: Network error: Client connection negotiation failed: client connection to 127.18.100.125:39157: connect: Connection refused (error 111),
I20260130 03:00:17.958629 22923 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260130 03:00:18.698843 23910 raft_consensus.cc:493] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 342be83f20a648fd849c7d23454fba08)
I20260130 03:00:18.699355 23910 raft_consensus.cc:515] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22dedcf69e00438999d3a5fedde57324" member_type: VOTER last_known_addr { host: "127.18.100.126" port: 42077 } } peers { permanent_uuid: "b01057b6c5ff4fc4b07253801881fe13" member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39157 } } peers { permanent_uuid: "342be83f20a648fd849c7d23454fba08" member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34987 } }
I20260130 03:00:18.701413 23910 leader_election.cc:290] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers b01057b6c5ff4fc4b07253801881fe13 (127.18.100.125:39157), 342be83f20a648fd849c7d23454fba08 (127.18.100.124:34987)
W20260130 03:00:18.703963 22970 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.18.100.125:39157: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20260130 03:00:18.707145 22968 leader_election.cc:336] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 342be83f20a648fd849c7d23454fba08 (127.18.100.124:34987): Network error: Client connection negotiation failed: client connection to 127.18.100.124:34987: connect: Connection refused (error 111)
W20260130 03:00:18.707641 22970 leader_election.cc:336] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer b01057b6c5ff4fc4b07253801881fe13 (127.18.100.125:39157): Network error: Client connection negotiation failed: client connection to 127.18.100.125:39157: connect: Connection refused (error 111)
I20260130 03:00:18.707979 22970 leader_election.cc:304] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22dedcf69e00438999d3a5fedde57324; no voters: 342be83f20a648fd849c7d23454fba08, b01057b6c5ff4fc4b07253801881fe13
I20260130 03:00:18.708544 23910 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260130 03:00:18.855110 22321 consensus_peers.cc:597] T ce91a17790324cd884a0521ce67dde67 P dd32a35972b347b89ceaaad2455aef5a -> Peer 789dbb4a27204ffbb68d7ae40f8f5e4a (127.18.100.67:41825): Couldn't send request to peer 789dbb4a27204ffbb68d7ae40f8f5e4a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ce91a17790324cd884a0521ce67dde67. This is attempt 31: this message will repeat every 5th retry.
I20260130 03:00:22.964653 22923 master_runner.cc:186] Error getting master registration for 127.18.100.125:39157: Network error: Client connection negotiation failed: client connection to 127.18.100.125:39157: connect: Connection refused (error 111),
I20260130 03:00:22.965032 22923 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260130 03:00:23.753427 23979 raft_consensus.cc:493] T 5cc07f1d50624e03b1f5498b12ea680b P 111215cc989f4203a6e1517f2291d66f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 03:00:23.754083 23979 raft_consensus.cc:515] T 5cc07f1d50624e03b1f5498b12ea680b P 111215cc989f4203a6e1517f2291d66f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111215cc989f4203a6e1517f2291d66f" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 39751 } } peers { permanent_uuid: "dd32a35972b347b89ceaaad2455aef5a" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 39781 } } peers { permanent_uuid: "789dbb4a27204ffbb68d7ae40f8f5e4a" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 41825 } }
I20260130 03:00:23.756345 23979 leader_election.cc:290] T 5cc07f1d50624e03b1f5498b12ea680b P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dd32a35972b347b89ceaaad2455aef5a (127.18.100.66:39781), 789dbb4a27204ffbb68d7ae40f8f5e4a (127.18.100.67:41825)
W20260130 03:00:23.758272 22180 leader_election.cc:343] T 5cc07f1d50624e03b1f5498b12ea680b P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dd32a35972b347b89ceaaad2455aef5a (127.18.100.66:39781): Not found: Tablet not found: 5cc07f1d50624e03b1f5498b12ea680b
W20260130 03:00:23.758839 22179 leader_election.cc:343] T 5cc07f1d50624e03b1f5498b12ea680b P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 789dbb4a27204ffbb68d7ae40f8f5e4a (127.18.100.67:41825): Not found: Tablet not found: 5cc07f1d50624e03b1f5498b12ea680b
I20260130 03:00:23.759217 22179 leader_election.cc:304] T 5cc07f1d50624e03b1f5498b12ea680b P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 111215cc989f4203a6e1517f2291d66f; no voters: 789dbb4a27204ffbb68d7ae40f8f5e4a, dd32a35972b347b89ceaaad2455aef5a
I20260130 03:00:23.759847 23979 raft_consensus.cc:2749] T 5cc07f1d50624e03b1f5498b12ea680b P 111215cc989f4203a6e1517f2291d66f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260130 03:00:23.892225 22179 consensus_peers.cc:597] T 476f8fd240bc48f1a9c47a0351ad5d27 P 111215cc989f4203a6e1517f2291d66f -> Peer 789dbb4a27204ffbb68d7ae40f8f5e4a (127.18.100.67:41825): Couldn't send request to peer 789dbb4a27204ffbb68d7ae40f8f5e4a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 476f8fd240bc48f1a9c47a0351ad5d27. This is attempt 36: this message will repeat every 5th retry.
I20260130 03:00:25.340080 23992 raft_consensus.cc:493] T 563d05e06b1f48b4be140219da5c9dbd P 111215cc989f4203a6e1517f2291d66f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 03:00:25.340590 23992 raft_consensus.cc:515] T 563d05e06b1f48b4be140219da5c9dbd P 111215cc989f4203a6e1517f2291d66f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111215cc989f4203a6e1517f2291d66f" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 39751 } } peers { permanent_uuid: "dd32a35972b347b89ceaaad2455aef5a" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 39781 } } peers { permanent_uuid: "789dbb4a27204ffbb68d7ae40f8f5e4a" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 41825 } }
I20260130 03:00:25.342201 23992 leader_election.cc:290] T 563d05e06b1f48b4be140219da5c9dbd P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dd32a35972b347b89ceaaad2455aef5a (127.18.100.66:39781), 789dbb4a27204ffbb68d7ae40f8f5e4a (127.18.100.67:41825)
W20260130 03:00:25.343716 22180 leader_election.cc:343] T 563d05e06b1f48b4be140219da5c9dbd P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dd32a35972b347b89ceaaad2455aef5a (127.18.100.66:39781): Not found: Tablet not found: 563d05e06b1f48b4be140219da5c9dbd
W20260130 03:00:25.344216 22179 leader_election.cc:343] T 563d05e06b1f48b4be140219da5c9dbd P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 789dbb4a27204ffbb68d7ae40f8f5e4a (127.18.100.67:41825): Not found: Tablet not found: 563d05e06b1f48b4be140219da5c9dbd
I20260130 03:00:25.344558 22179 leader_election.cc:304] T 563d05e06b1f48b4be140219da5c9dbd P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 111215cc989f4203a6e1517f2291d66f; no voters: 789dbb4a27204ffbb68d7ae40f8f5e4a, dd32a35972b347b89ceaaad2455aef5a
I20260130 03:00:25.345182 23992 raft_consensus.cc:2749] T 563d05e06b1f48b4be140219da5c9dbd P 111215cc989f4203a6e1517f2291d66f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260130 03:00:27.972182 22923 master_runner.cc:186] Error getting master registration for 127.18.100.125:39157: Network error: Client connection negotiation failed: client connection to 127.18.100.125:39157: connect: Connection refused (error 111),
I20260130 03:00:27.972744 22923 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260130 03:00:29.010624 24041 raft_consensus.cc:493] T 563d05e06b1f48b4be140219da5c9dbd P 111215cc989f4203a6e1517f2291d66f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 03:00:29.011087 24041 raft_consensus.cc:515] T 563d05e06b1f48b4be140219da5c9dbd P 111215cc989f4203a6e1517f2291d66f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111215cc989f4203a6e1517f2291d66f" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 39751 } } peers { permanent_uuid: "dd32a35972b347b89ceaaad2455aef5a" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 39781 } } peers { permanent_uuid: "789dbb4a27204ffbb68d7ae40f8f5e4a" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 41825 } }
I20260130 03:00:29.012563 24041 leader_election.cc:290] T 563d05e06b1f48b4be140219da5c9dbd P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dd32a35972b347b89ceaaad2455aef5a (127.18.100.66:39781), 789dbb4a27204ffbb68d7ae40f8f5e4a (127.18.100.67:41825)
W20260130 03:00:29.014495 22180 leader_election.cc:343] T 563d05e06b1f48b4be140219da5c9dbd P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dd32a35972b347b89ceaaad2455aef5a (127.18.100.66:39781): Not found: Tablet not found: 563d05e06b1f48b4be140219da5c9dbd
W20260130 03:00:29.015028 22179 leader_election.cc:343] T 563d05e06b1f48b4be140219da5c9dbd P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 789dbb4a27204ffbb68d7ae40f8f5e4a (127.18.100.67:41825): Not found: Tablet not found: 563d05e06b1f48b4be140219da5c9dbd
I20260130 03:00:29.015347 22179 leader_election.cc:304] T 563d05e06b1f48b4be140219da5c9dbd P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 111215cc989f4203a6e1517f2291d66f; no voters: 789dbb4a27204ffbb68d7ae40f8f5e4a, dd32a35972b347b89ceaaad2455aef5a
I20260130 03:00:29.015947 24041 raft_consensus.cc:2749] T 563d05e06b1f48b4be140219da5c9dbd P 111215cc989f4203a6e1517f2291d66f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260130 03:00:29.434396 22321 consensus_peers.cc:597] T ce91a17790324cd884a0521ce67dde67 P dd32a35972b347b89ceaaad2455aef5a -> Peer 789dbb4a27204ffbb68d7ae40f8f5e4a (127.18.100.67:41825): Couldn't send request to peer 789dbb4a27204ffbb68d7ae40f8f5e4a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ce91a17790324cd884a0521ce67dde67. This is attempt 36: this message will repeat every 5th retry.
I20260130 03:00:30.626396 24063 raft_consensus.cc:493] T be72107f1e024d5788f5d2459c85015c P 789dbb4a27204ffbb68d7ae40f8f5e4a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 03:00:30.627063 24063 raft_consensus.cc:515] T be72107f1e024d5788f5d2459c85015c P 789dbb4a27204ffbb68d7ae40f8f5e4a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "789dbb4a27204ffbb68d7ae40f8f5e4a" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 41825 } } peers { permanent_uuid: "dd32a35972b347b89ceaaad2455aef5a" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 39781 } } peers { permanent_uuid: "111215cc989f4203a6e1517f2291d66f" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 39751 } }
I20260130 03:00:30.629235 24063 leader_election.cc:290] T be72107f1e024d5788f5d2459c85015c P 789dbb4a27204ffbb68d7ae40f8f5e4a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dd32a35972b347b89ceaaad2455aef5a (127.18.100.66:39781), 111215cc989f4203a6e1517f2291d66f (127.18.100.65:39751)
W20260130 03:00:30.630890 22469 leader_election.cc:343] T be72107f1e024d5788f5d2459c85015c P 789dbb4a27204ffbb68d7ae40f8f5e4a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dd32a35972b347b89ceaaad2455aef5a (127.18.100.66:39781): Not found: Tablet not found: be72107f1e024d5788f5d2459c85015c
W20260130 03:00:30.631472 22468 leader_election.cc:343] T be72107f1e024d5788f5d2459c85015c P 789dbb4a27204ffbb68d7ae40f8f5e4a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 111215cc989f4203a6e1517f2291d66f (127.18.100.65:39751): Not found: Tablet not found: be72107f1e024d5788f5d2459c85015c
I20260130 03:00:30.631918 22468 leader_election.cc:304] T be72107f1e024d5788f5d2459c85015c P 789dbb4a27204ffbb68d7ae40f8f5e4a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 789dbb4a27204ffbb68d7ae40f8f5e4a; no voters: 111215cc989f4203a6e1517f2291d66f, dd32a35972b347b89ceaaad2455aef5a
I20260130 03:00:30.632547 24063 raft_consensus.cc:2749] T be72107f1e024d5788f5d2459c85015c P 789dbb4a27204ffbb68d7ae40f8f5e4a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260130 03:00:31.703699 23020 debug-util.cc:398] Leaking SignalData structure 0x7b080006fdc0 after lost signal to thread 22933
W20260130 03:00:31.704439 23020 debug-util.cc:398] Leaking SignalData structure 0x7b0800075820 after lost signal to thread 23023
I20260130 03:00:33.008986 22923 master_runner.cc:186] Error getting master registration for 127.18.100.125:39157: Network error: Client connection negotiation failed: client connection to 127.18.100.125:39157: connect: Connection refused (error 111),
I20260130 03:00:33.009410 22923 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260130 03:00:33.172696 24100 raft_consensus.cc:493] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 342be83f20a648fd849c7d23454fba08)
I20260130 03:00:33.173175 24100 raft_consensus.cc:515] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22dedcf69e00438999d3a5fedde57324" member_type: VOTER last_known_addr { host: "127.18.100.126" port: 42077 } } peers { permanent_uuid: "b01057b6c5ff4fc4b07253801881fe13" member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39157 } } peers { permanent_uuid: "342be83f20a648fd849c7d23454fba08" member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34987 } }
I20260130 03:00:33.174919 24100 leader_election.cc:290] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers b01057b6c5ff4fc4b07253801881fe13 (127.18.100.125:39157), 342be83f20a648fd849c7d23454fba08 (127.18.100.124:34987)
W20260130 03:00:33.176456 22970 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.18.100.125:39157: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20260130 03:00:33.179848 22970 leader_election.cc:336] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer b01057b6c5ff4fc4b07253801881fe13 (127.18.100.125:39157): Network error: Client connection negotiation failed: client connection to 127.18.100.125:39157: connect: Connection refused (error 111)
W20260130 03:00:33.180356 22968 leader_election.cc:336] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 342be83f20a648fd849c7d23454fba08 (127.18.100.124:34987): Network error: Client connection negotiation failed: client connection to 127.18.100.124:34987: connect: Connection refused (error 111)
I20260130 03:00:33.180627 22968 leader_election.cc:304] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22dedcf69e00438999d3a5fedde57324; no voters: 342be83f20a648fd849c7d23454fba08, b01057b6c5ff4fc4b07253801881fe13
I20260130 03:00:33.181154 24100 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260130 03:00:34.023931 22179 consensus_peers.cc:597] T 476f8fd240bc48f1a9c47a0351ad5d27 P 111215cc989f4203a6e1517f2291d66f -> Peer 789dbb4a27204ffbb68d7ae40f8f5e4a (127.18.100.67:41825): Couldn't send request to peer 789dbb4a27204ffbb68d7ae40f8f5e4a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 476f8fd240bc48f1a9c47a0351ad5d27. This is attempt 41: this message will repeat every 5th retry.
I20260130 03:00:35.707643 24132 raft_consensus.cc:493] T be72107f1e024d5788f5d2459c85015c P 789dbb4a27204ffbb68d7ae40f8f5e4a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 03:00:35.708133 24132 raft_consensus.cc:515] T be72107f1e024d5788f5d2459c85015c P 789dbb4a27204ffbb68d7ae40f8f5e4a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "789dbb4a27204ffbb68d7ae40f8f5e4a" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 41825 } } peers { permanent_uuid: "dd32a35972b347b89ceaaad2455aef5a" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 39781 } } peers { permanent_uuid: "111215cc989f4203a6e1517f2291d66f" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 39751 } }
I20260130 03:00:35.709751 24132 leader_election.cc:290] T be72107f1e024d5788f5d2459c85015c P 789dbb4a27204ffbb68d7ae40f8f5e4a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dd32a35972b347b89ceaaad2455aef5a (127.18.100.66:39781), 111215cc989f4203a6e1517f2291d66f (127.18.100.65:39751)
W20260130 03:00:35.711459 22469 leader_election.cc:343] T be72107f1e024d5788f5d2459c85015c P 789dbb4a27204ffbb68d7ae40f8f5e4a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dd32a35972b347b89ceaaad2455aef5a (127.18.100.66:39781): Not found: Tablet not found: be72107f1e024d5788f5d2459c85015c
W20260130 03:00:35.711910 22468 leader_election.cc:343] T be72107f1e024d5788f5d2459c85015c P 789dbb4a27204ffbb68d7ae40f8f5e4a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 111215cc989f4203a6e1517f2291d66f (127.18.100.65:39751): Not found: Tablet not found: be72107f1e024d5788f5d2459c85015c
I20260130 03:00:35.712251 22468 leader_election.cc:304] T be72107f1e024d5788f5d2459c85015c P 789dbb4a27204ffbb68d7ae40f8f5e4a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 789dbb4a27204ffbb68d7ae40f8f5e4a; no voters: 111215cc989f4203a6e1517f2291d66f, dd32a35972b347b89ceaaad2455aef5a
I20260130 03:00:35.712845 24132 raft_consensus.cc:2749] T be72107f1e024d5788f5d2459c85015c P 789dbb4a27204ffbb68d7ae40f8f5e4a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260130 03:00:37.182782 22289 debug-util.cc:398] Leaking SignalData structure 0x7b08000cabe0 after lost signal to thread 22155
W20260130 03:00:37.183717 22289 debug-util.cc:398] Leaking SignalData structure 0x7b080006d8a0 after lost signal to thread 22292
I20260130 03:00:38.015908 22923 master_runner.cc:186] Error getting master registration for 127.18.100.125:39157: Network error: Client connection negotiation failed: client connection to 127.18.100.125:39157: connect: Connection refused (error 111),
I20260130 03:00:38.016480 22923 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260130 03:00:38.654100 22321 consensus_peers.cc:597] T ce91a17790324cd884a0521ce67dde67 P dd32a35972b347b89ceaaad2455aef5a -> Peer 789dbb4a27204ffbb68d7ae40f8f5e4a (127.18.100.67:41825): Couldn't send request to peer 789dbb4a27204ffbb68d7ae40f8f5e4a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ce91a17790324cd884a0521ce67dde67. This is attempt 41: this message will repeat every 5th retry.
I20260130 03:00:41.451956 24203 raft_consensus.cc:493] T 5cc07f1d50624e03b1f5498b12ea680b P 111215cc989f4203a6e1517f2291d66f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 03:00:41.452548 24203 raft_consensus.cc:515] T 5cc07f1d50624e03b1f5498b12ea680b P 111215cc989f4203a6e1517f2291d66f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111215cc989f4203a6e1517f2291d66f" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 39751 } } peers { permanent_uuid: "dd32a35972b347b89ceaaad2455aef5a" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 39781 } } peers { permanent_uuid: "789dbb4a27204ffbb68d7ae40f8f5e4a" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 41825 } }
I20260130 03:00:41.455094 24203 leader_election.cc:290] T 5cc07f1d50624e03b1f5498b12ea680b P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dd32a35972b347b89ceaaad2455aef5a (127.18.100.66:39781), 789dbb4a27204ffbb68d7ae40f8f5e4a (127.18.100.67:41825)
W20260130 03:00:41.456804 22180 leader_election.cc:343] T 5cc07f1d50624e03b1f5498b12ea680b P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dd32a35972b347b89ceaaad2455aef5a (127.18.100.66:39781): Not found: Tablet not found: 5cc07f1d50624e03b1f5498b12ea680b
W20260130 03:00:41.457233 22179 leader_election.cc:343] T 5cc07f1d50624e03b1f5498b12ea680b P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 789dbb4a27204ffbb68d7ae40f8f5e4a (127.18.100.67:41825): Not found: Tablet not found: 5cc07f1d50624e03b1f5498b12ea680b
I20260130 03:00:41.457530 22179 leader_election.cc:304] T 5cc07f1d50624e03b1f5498b12ea680b P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 111215cc989f4203a6e1517f2291d66f; no voters: 789dbb4a27204ffbb68d7ae40f8f5e4a, dd32a35972b347b89ceaaad2455aef5a
I20260130 03:00:41.458091 24203 raft_consensus.cc:2749] T 5cc07f1d50624e03b1f5498b12ea680b P 111215cc989f4203a6e1517f2291d66f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260130 03:00:41.508785 24203 raft_consensus.cc:493] T 563d05e06b1f48b4be140219da5c9dbd P 111215cc989f4203a6e1517f2291d66f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 03:00:41.509378 24203 raft_consensus.cc:515] T 563d05e06b1f48b4be140219da5c9dbd P 111215cc989f4203a6e1517f2291d66f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111215cc989f4203a6e1517f2291d66f" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 39751 } } peers { permanent_uuid: "dd32a35972b347b89ceaaad2455aef5a" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 39781 } } peers { permanent_uuid: "789dbb4a27204ffbb68d7ae40f8f5e4a" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 41825 } }
I20260130 03:00:41.511543 24203 leader_election.cc:290] T 563d05e06b1f48b4be140219da5c9dbd P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dd32a35972b347b89ceaaad2455aef5a (127.18.100.66:39781), 789dbb4a27204ffbb68d7ae40f8f5e4a (127.18.100.67:41825)
W20260130 03:00:41.513221 22180 leader_election.cc:343] T 563d05e06b1f48b4be140219da5c9dbd P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dd32a35972b347b89ceaaad2455aef5a (127.18.100.66:39781): Not found: Tablet not found: 563d05e06b1f48b4be140219da5c9dbd
W20260130 03:00:41.513626 22179 leader_election.cc:343] T 563d05e06b1f48b4be140219da5c9dbd P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 789dbb4a27204ffbb68d7ae40f8f5e4a (127.18.100.67:41825): Not found: Tablet not found: 563d05e06b1f48b4be140219da5c9dbd
I20260130 03:00:41.514040 22179 leader_election.cc:304] T 563d05e06b1f48b4be140219da5c9dbd P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 111215cc989f4203a6e1517f2291d66f; no voters: 789dbb4a27204ffbb68d7ae40f8f5e4a, dd32a35972b347b89ceaaad2455aef5a
I20260130 03:00:41.514703 24203 raft_consensus.cc:2749] T 563d05e06b1f48b4be140219da5c9dbd P 111215cc989f4203a6e1517f2291d66f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260130 03:00:43.021915 22923 master_runner.cc:186] Error getting master registration for 127.18.100.125:39157: Network error: Client connection negotiation failed: client connection to 127.18.100.125:39157: connect: Connection refused (error 111),
I20260130 03:00:43.022354 22923 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260130 03:00:43.961537 24238 raft_consensus.cc:493] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 342be83f20a648fd849c7d23454fba08)
I20260130 03:00:43.962101 24238 raft_consensus.cc:515] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22dedcf69e00438999d3a5fedde57324" member_type: VOTER last_known_addr { host: "127.18.100.126" port: 42077 } } peers { permanent_uuid: "b01057b6c5ff4fc4b07253801881fe13" member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39157 } } peers { permanent_uuid: "342be83f20a648fd849c7d23454fba08" member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34987 } }
I20260130 03:00:43.963918 24238 leader_election.cc:290] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers b01057b6c5ff4fc4b07253801881fe13 (127.18.100.125:39157), 342be83f20a648fd849c7d23454fba08 (127.18.100.124:34987)
W20260130 03:00:43.966331 22970 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.18.100.125:39157: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20260130 03:00:43.969532 22970 leader_election.cc:336] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer b01057b6c5ff4fc4b07253801881fe13 (127.18.100.125:39157): Network error: Client connection negotiation failed: client connection to 127.18.100.125:39157: connect: Connection refused (error 111)
W20260130 03:00:43.970456 22968 leader_election.cc:336] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 342be83f20a648fd849c7d23454fba08 (127.18.100.124:34987): Network error: Client connection negotiation failed: client connection to 127.18.100.124:34987: connect: Connection refused (error 111)
I20260130 03:00:43.970880 22968 leader_election.cc:304] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22dedcf69e00438999d3a5fedde57324; no voters: 342be83f20a648fd849c7d23454fba08, b01057b6c5ff4fc4b07253801881fe13
I20260130 03:00:43.971992 24238 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260130 03:00:44.342427 24246 raft_consensus.cc:493] T be72107f1e024d5788f5d2459c85015c P 789dbb4a27204ffbb68d7ae40f8f5e4a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 03:00:44.343097 24246 raft_consensus.cc:515] T be72107f1e024d5788f5d2459c85015c P 789dbb4a27204ffbb68d7ae40f8f5e4a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "789dbb4a27204ffbb68d7ae40f8f5e4a" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 41825 } } peers { permanent_uuid: "dd32a35972b347b89ceaaad2455aef5a" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 39781 } } peers { permanent_uuid: "111215cc989f4203a6e1517f2291d66f" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 39751 } }
I20260130 03:00:44.345067 24246 leader_election.cc:290] T be72107f1e024d5788f5d2459c85015c P 789dbb4a27204ffbb68d7ae40f8f5e4a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dd32a35972b347b89ceaaad2455aef5a (127.18.100.66:39781), 111215cc989f4203a6e1517f2291d66f (127.18.100.65:39751)
W20260130 03:00:44.347151 22469 leader_election.cc:343] T be72107f1e024d5788f5d2459c85015c P 789dbb4a27204ffbb68d7ae40f8f5e4a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dd32a35972b347b89ceaaad2455aef5a (127.18.100.66:39781): Not found: Tablet not found: be72107f1e024d5788f5d2459c85015c
W20260130 03:00:44.347698 22468 leader_election.cc:343] T be72107f1e024d5788f5d2459c85015c P 789dbb4a27204ffbb68d7ae40f8f5e4a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 111215cc989f4203a6e1517f2291d66f (127.18.100.65:39751): Not found: Tablet not found: be72107f1e024d5788f5d2459c85015c
I20260130 03:00:44.347994 22468 leader_election.cc:304] T be72107f1e024d5788f5d2459c85015c P 789dbb4a27204ffbb68d7ae40f8f5e4a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 789dbb4a27204ffbb68d7ae40f8f5e4a; no voters: 111215cc989f4203a6e1517f2291d66f, dd32a35972b347b89ceaaad2455aef5a
I20260130 03:00:44.348663 24246 raft_consensus.cc:2749] T be72107f1e024d5788f5d2459c85015c P 789dbb4a27204ffbb68d7ae40f8f5e4a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260130 03:00:44.613266 22179 consensus_peers.cc:597] T 476f8fd240bc48f1a9c47a0351ad5d27 P 111215cc989f4203a6e1517f2291d66f -> Peer 789dbb4a27204ffbb68d7ae40f8f5e4a (127.18.100.67:41825): Couldn't send request to peer 789dbb4a27204ffbb68d7ae40f8f5e4a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 476f8fd240bc48f1a9c47a0351ad5d27. This is attempt 46: this message will repeat every 5th retry.
W20260130 03:00:44.622676 22433 debug-util.cc:398] Leaking SignalData structure 0x7b0800041960 after lost signal to thread 22304
W20260130 03:00:44.623759 22433 debug-util.cc:398] Leaking SignalData structure 0x7b08000c0a60 after lost signal to thread 22436
I20260130 03:00:47.282840 24284 raft_consensus.cc:493] T 563d05e06b1f48b4be140219da5c9dbd P 111215cc989f4203a6e1517f2291d66f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 03:00:47.283473 24284 raft_consensus.cc:515] T 563d05e06b1f48b4be140219da5c9dbd P 111215cc989f4203a6e1517f2291d66f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111215cc989f4203a6e1517f2291d66f" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 39751 } } peers { permanent_uuid: "dd32a35972b347b89ceaaad2455aef5a" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 39781 } } peers { permanent_uuid: "789dbb4a27204ffbb68d7ae40f8f5e4a" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 41825 } }
I20260130 03:00:47.285283 24284 leader_election.cc:290] T 563d05e06b1f48b4be140219da5c9dbd P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dd32a35972b347b89ceaaad2455aef5a (127.18.100.66:39781), 789dbb4a27204ffbb68d7ae40f8f5e4a (127.18.100.67:41825)
W20260130 03:00:47.287246 22179 leader_election.cc:343] T 563d05e06b1f48b4be140219da5c9dbd P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 789dbb4a27204ffbb68d7ae40f8f5e4a (127.18.100.67:41825): Not found: Tablet not found: 563d05e06b1f48b4be140219da5c9dbd
W20260130 03:00:47.287761 22180 leader_election.cc:343] T 563d05e06b1f48b4be140219da5c9dbd P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dd32a35972b347b89ceaaad2455aef5a (127.18.100.66:39781): Not found: Tablet not found: 563d05e06b1f48b4be140219da5c9dbd
I20260130 03:00:47.288030 22180 leader_election.cc:304] T 563d05e06b1f48b4be140219da5c9dbd P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 111215cc989f4203a6e1517f2291d66f; no voters: 789dbb4a27204ffbb68d7ae40f8f5e4a, dd32a35972b347b89ceaaad2455aef5a
I20260130 03:00:47.288596 24284 raft_consensus.cc:2749] T 563d05e06b1f48b4be140219da5c9dbd P 111215cc989f4203a6e1517f2291d66f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260130 03:00:48.028893 22923 master_runner.cc:186] Error getting master registration for 127.18.100.125:39157: Network error: Client connection negotiation failed: client connection to 127.18.100.125:39157: connect: Connection refused (error 111),
I20260130 03:00:48.029381 22923 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260130 03:00:48.956128 22321 consensus_peers.cc:597] T ce91a17790324cd884a0521ce67dde67 P dd32a35972b347b89ceaaad2455aef5a -> Peer 789dbb4a27204ffbb68d7ae40f8f5e4a (127.18.100.67:41825): Couldn't send request to peer 789dbb4a27204ffbb68d7ae40f8f5e4a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ce91a17790324cd884a0521ce67dde67. This is attempt 46: this message will repeat every 5th retry.
I20260130 03:00:50.780246 24328 raft_consensus.cc:493] T 5cc07f1d50624e03b1f5498b12ea680b P 111215cc989f4203a6e1517f2291d66f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 03:00:50.780750 24328 raft_consensus.cc:515] T 5cc07f1d50624e03b1f5498b12ea680b P 111215cc989f4203a6e1517f2291d66f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111215cc989f4203a6e1517f2291d66f" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 39751 } } peers { permanent_uuid: "dd32a35972b347b89ceaaad2455aef5a" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 39781 } } peers { permanent_uuid: "789dbb4a27204ffbb68d7ae40f8f5e4a" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 41825 } }
I20260130 03:00:50.782526 24328 leader_election.cc:290] T 5cc07f1d50624e03b1f5498b12ea680b P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dd32a35972b347b89ceaaad2455aef5a (127.18.100.66:39781), 789dbb4a27204ffbb68d7ae40f8f5e4a (127.18.100.67:41825)
W20260130 03:00:50.784229 22180 leader_election.cc:343] T 5cc07f1d50624e03b1f5498b12ea680b P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dd32a35972b347b89ceaaad2455aef5a (127.18.100.66:39781): Not found: Tablet not found: 5cc07f1d50624e03b1f5498b12ea680b
W20260130 03:00:50.784649 22179 leader_election.cc:343] T 5cc07f1d50624e03b1f5498b12ea680b P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 789dbb4a27204ffbb68d7ae40f8f5e4a (127.18.100.67:41825): Not found: Tablet not found: 5cc07f1d50624e03b1f5498b12ea680b
I20260130 03:00:50.784920 22179 leader_election.cc:304] T 5cc07f1d50624e03b1f5498b12ea680b P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 111215cc989f4203a6e1517f2291d66f; no voters: 789dbb4a27204ffbb68d7ae40f8f5e4a, dd32a35972b347b89ceaaad2455aef5a
I20260130 03:00:50.785430 24328 raft_consensus.cc:2749] T 5cc07f1d50624e03b1f5498b12ea680b P 111215cc989f4203a6e1517f2291d66f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260130 03:00:51.964615 24346 raft_consensus.cc:493] T 563d05e06b1f48b4be140219da5c9dbd P 111215cc989f4203a6e1517f2291d66f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 03:00:51.965112 24346 raft_consensus.cc:515] T 563d05e06b1f48b4be140219da5c9dbd P 111215cc989f4203a6e1517f2291d66f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111215cc989f4203a6e1517f2291d66f" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 39751 } } peers { permanent_uuid: "dd32a35972b347b89ceaaad2455aef5a" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 39781 } } peers { permanent_uuid: "789dbb4a27204ffbb68d7ae40f8f5e4a" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 41825 } }
I20260130 03:00:51.966884 24346 leader_election.cc:290] T 563d05e06b1f48b4be140219da5c9dbd P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dd32a35972b347b89ceaaad2455aef5a (127.18.100.66:39781), 789dbb4a27204ffbb68d7ae40f8f5e4a (127.18.100.67:41825)
W20260130 03:00:51.968492 22180 leader_election.cc:343] T 563d05e06b1f48b4be140219da5c9dbd P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dd32a35972b347b89ceaaad2455aef5a (127.18.100.66:39781): Not found: Tablet not found: 563d05e06b1f48b4be140219da5c9dbd
W20260130 03:00:51.968921 22179 leader_election.cc:343] T 563d05e06b1f48b4be140219da5c9dbd P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 789dbb4a27204ffbb68d7ae40f8f5e4a (127.18.100.67:41825): Not found: Tablet not found: 563d05e06b1f48b4be140219da5c9dbd
I20260130 03:00:51.969274 22179 leader_election.cc:304] T 563d05e06b1f48b4be140219da5c9dbd P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 111215cc989f4203a6e1517f2291d66f; no voters: 789dbb4a27204ffbb68d7ae40f8f5e4a, dd32a35972b347b89ceaaad2455aef5a
I20260130 03:00:51.969913 24346 raft_consensus.cc:2749] T 563d05e06b1f48b4be140219da5c9dbd P 111215cc989f4203a6e1517f2291d66f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260130 03:00:53.035094 22923 master_runner.cc:186] Error getting master registration for 127.18.100.125:39157: Network error: Client connection negotiation failed: client connection to 127.18.100.125:39157: connect: Connection refused (error 111),
I20260130 03:00:53.035463 22923 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260130 03:00:54.333870 22179 consensus_peers.cc:597] T 476f8fd240bc48f1a9c47a0351ad5d27 P 111215cc989f4203a6e1517f2291d66f -> Peer 789dbb4a27204ffbb68d7ae40f8f5e4a (127.18.100.67:41825): Couldn't send request to peer 789dbb4a27204ffbb68d7ae40f8f5e4a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 476f8fd240bc48f1a9c47a0351ad5d27. This is attempt 51: this message will repeat every 5th retry.
W20260130 03:00:56.064077 22576 debug-util.cc:398] Leaking SignalData structure 0x7b080006b900 after lost signal to thread 22448
W20260130 03:00:56.065218 22576 debug-util.cc:398] Leaking SignalData structure 0x7b080006bfe0 after lost signal to thread 22579
I20260130 03:00:58.042270 22923 master_runner.cc:186] Error getting master registration for 127.18.100.125:39157: Network error: Client connection negotiation failed: client connection to 127.18.100.125:39157: connect: Connection refused (error 111),
I20260130 03:00:58.042755 22923 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260130 03:00:58.846385 22321 consensus_peers.cc:597] T ce91a17790324cd884a0521ce67dde67 P dd32a35972b347b89ceaaad2455aef5a -> Peer 789dbb4a27204ffbb68d7ae40f8f5e4a (127.18.100.67:41825): Couldn't send request to peer 789dbb4a27204ffbb68d7ae40f8f5e4a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ce91a17790324cd884a0521ce67dde67. This is attempt 51: this message will repeat every 5th retry.
I20260130 03:01:03.048971 22923 master_runner.cc:186] Error getting master registration for 127.18.100.125:39157: Network error: Client connection negotiation failed: client connection to 127.18.100.125:39157: connect: Connection refused (error 111),
I20260130 03:01:03.049387 22923 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260130 03:01:03.494879 24497 raft_consensus.cc:493] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 342be83f20a648fd849c7d23454fba08)
I20260130 03:01:03.495496 24497 raft_consensus.cc:515] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22dedcf69e00438999d3a5fedde57324" member_type: VOTER last_known_addr { host: "127.18.100.126" port: 42077 } } peers { permanent_uuid: "b01057b6c5ff4fc4b07253801881fe13" member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39157 } } peers { permanent_uuid: "342be83f20a648fd849c7d23454fba08" member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34987 } }
I20260130 03:01:03.497757 24497 leader_election.cc:290] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers b01057b6c5ff4fc4b07253801881fe13 (127.18.100.125:39157), 342be83f20a648fd849c7d23454fba08 (127.18.100.124:34987)
W20260130 03:01:03.499466 22970 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.18.100.125:39157: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20260130 03:01:03.502758 22970 leader_election.cc:336] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer b01057b6c5ff4fc4b07253801881fe13 (127.18.100.125:39157): Network error: Client connection negotiation failed: client connection to 127.18.100.125:39157: connect: Connection refused (error 111)
W20260130 03:01:03.503466 22968 leader_election.cc:336] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 342be83f20a648fd849c7d23454fba08 (127.18.100.124:34987): Network error: Client connection negotiation failed: client connection to 127.18.100.124:34987: connect: Connection refused (error 111)
I20260130 03:01:03.503872 22968 leader_election.cc:304] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22dedcf69e00438999d3a5fedde57324; no voters: 342be83f20a648fd849c7d23454fba08, b01057b6c5ff4fc4b07253801881fe13
I20260130 03:01:03.504472 24497 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260130 03:01:03.674444 24500 raft_consensus.cc:493] T be72107f1e024d5788f5d2459c85015c P 789dbb4a27204ffbb68d7ae40f8f5e4a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 03:01:03.675329 24500 raft_consensus.cc:515] T be72107f1e024d5788f5d2459c85015c P 789dbb4a27204ffbb68d7ae40f8f5e4a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "789dbb4a27204ffbb68d7ae40f8f5e4a" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 41825 } } peers { permanent_uuid: "dd32a35972b347b89ceaaad2455aef5a" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 39781 } } peers { permanent_uuid: "111215cc989f4203a6e1517f2291d66f" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 39751 } }
I20260130 03:01:03.678278 24500 leader_election.cc:290] T be72107f1e024d5788f5d2459c85015c P 789dbb4a27204ffbb68d7ae40f8f5e4a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dd32a35972b347b89ceaaad2455aef5a (127.18.100.66:39781), 111215cc989f4203a6e1517f2291d66f (127.18.100.65:39751)
W20260130 03:01:03.680374 22469 leader_election.cc:343] T be72107f1e024d5788f5d2459c85015c P 789dbb4a27204ffbb68d7ae40f8f5e4a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dd32a35972b347b89ceaaad2455aef5a (127.18.100.66:39781): Not found: Tablet not found: be72107f1e024d5788f5d2459c85015c
W20260130 03:01:03.681001 22468 leader_election.cc:343] T be72107f1e024d5788f5d2459c85015c P 789dbb4a27204ffbb68d7ae40f8f5e4a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 111215cc989f4203a6e1517f2291d66f (127.18.100.65:39751): Not found: Tablet not found: be72107f1e024d5788f5d2459c85015c
I20260130 03:01:03.681370 22468 leader_election.cc:304] T be72107f1e024d5788f5d2459c85015c P 789dbb4a27204ffbb68d7ae40f8f5e4a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 789dbb4a27204ffbb68d7ae40f8f5e4a; no voters: 111215cc989f4203a6e1517f2291d66f, dd32a35972b347b89ceaaad2455aef5a
I20260130 03:01:03.682060 24500 raft_consensus.cc:2749] T be72107f1e024d5788f5d2459c85015c P 789dbb4a27204ffbb68d7ae40f8f5e4a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260130 03:01:05.222095 24519 raft_consensus.cc:493] T 5cc07f1d50624e03b1f5498b12ea680b P 111215cc989f4203a6e1517f2291d66f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 03:01:05.222852 24519 raft_consensus.cc:515] T 5cc07f1d50624e03b1f5498b12ea680b P 111215cc989f4203a6e1517f2291d66f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111215cc989f4203a6e1517f2291d66f" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 39751 } } peers { permanent_uuid: "dd32a35972b347b89ceaaad2455aef5a" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 39781 } } peers { permanent_uuid: "789dbb4a27204ffbb68d7ae40f8f5e4a" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 41825 } }
I20260130 03:01:05.225096 24519 leader_election.cc:290] T 5cc07f1d50624e03b1f5498b12ea680b P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dd32a35972b347b89ceaaad2455aef5a (127.18.100.66:39781), 789dbb4a27204ffbb68d7ae40f8f5e4a (127.18.100.67:41825)
W20260130 03:01:05.226835 22180 leader_election.cc:343] T 5cc07f1d50624e03b1f5498b12ea680b P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dd32a35972b347b89ceaaad2455aef5a (127.18.100.66:39781): Not found: Tablet not found: 5cc07f1d50624e03b1f5498b12ea680b
W20260130 03:01:05.227326 22179 leader_election.cc:343] T 5cc07f1d50624e03b1f5498b12ea680b P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 789dbb4a27204ffbb68d7ae40f8f5e4a (127.18.100.67:41825): Not found: Tablet not found: 5cc07f1d50624e03b1f5498b12ea680b
I20260130 03:01:05.227617 22179 leader_election.cc:304] T 5cc07f1d50624e03b1f5498b12ea680b P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 111215cc989f4203a6e1517f2291d66f; no voters: 789dbb4a27204ffbb68d7ae40f8f5e4a, dd32a35972b347b89ceaaad2455aef5a
I20260130 03:01:05.228222 24519 raft_consensus.cc:2749] T 5cc07f1d50624e03b1f5498b12ea680b P 111215cc989f4203a6e1517f2291d66f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260130 03:01:05.399755 22179 consensus_peers.cc:597] T 476f8fd240bc48f1a9c47a0351ad5d27 P 111215cc989f4203a6e1517f2291d66f -> Peer 789dbb4a27204ffbb68d7ae40f8f5e4a (127.18.100.67:41825): Couldn't send request to peer 789dbb4a27204ffbb68d7ae40f8f5e4a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 476f8fd240bc48f1a9c47a0351ad5d27. This is attempt 56: this message will repeat every 5th retry.
W20260130 03:01:05.650259 22289 debug-util.cc:398] Leaking SignalData structure 0x7b08000d0be0 after lost signal to thread 22155
W20260130 03:01:05.651753 22289 debug-util.cc:398] Leaking SignalData structure 0x7b080006cee0 after lost signal to thread 22292
I20260130 03:01:08.057045 22923 master_runner.cc:186] Error getting master registration for 127.18.100.125:39157: Network error: Client connection negotiation failed: client connection to 127.18.100.125:39157: connect: Connection refused (error 111),
I20260130 03:01:08.057809 22923 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260130 03:01:09.278858 24574 raft_consensus.cc:493] T 563d05e06b1f48b4be140219da5c9dbd P 111215cc989f4203a6e1517f2291d66f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 03:01:09.279354 24574 raft_consensus.cc:515] T 563d05e06b1f48b4be140219da5c9dbd P 111215cc989f4203a6e1517f2291d66f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111215cc989f4203a6e1517f2291d66f" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 39751 } } peers { permanent_uuid: "dd32a35972b347b89ceaaad2455aef5a" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 39781 } } peers { permanent_uuid: "789dbb4a27204ffbb68d7ae40f8f5e4a" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 41825 } }
I20260130 03:01:09.281109 24574 leader_election.cc:290] T 563d05e06b1f48b4be140219da5c9dbd P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dd32a35972b347b89ceaaad2455aef5a (127.18.100.66:39781), 789dbb4a27204ffbb68d7ae40f8f5e4a (127.18.100.67:41825)
W20260130 03:01:09.283488 22180 leader_election.cc:343] T 563d05e06b1f48b4be140219da5c9dbd P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dd32a35972b347b89ceaaad2455aef5a (127.18.100.66:39781): Not found: Tablet not found: 563d05e06b1f48b4be140219da5c9dbd
W20260130 03:01:09.284013 22179 leader_election.cc:343] T 563d05e06b1f48b4be140219da5c9dbd P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 789dbb4a27204ffbb68d7ae40f8f5e4a (127.18.100.67:41825): Not found: Tablet not found: 563d05e06b1f48b4be140219da5c9dbd
I20260130 03:01:09.284353 22179 leader_election.cc:304] T 563d05e06b1f48b4be140219da5c9dbd P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 111215cc989f4203a6e1517f2291d66f; no voters: 789dbb4a27204ffbb68d7ae40f8f5e4a, dd32a35972b347b89ceaaad2455aef5a
I20260130 03:01:09.284969 24574 raft_consensus.cc:2749] T 563d05e06b1f48b4be140219da5c9dbd P 111215cc989f4203a6e1517f2291d66f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260130 03:01:09.561877 22321 consensus_peers.cc:597] T ce91a17790324cd884a0521ce67dde67 P dd32a35972b347b89ceaaad2455aef5a -> Peer 789dbb4a27204ffbb68d7ae40f8f5e4a (127.18.100.67:41825): Couldn't send request to peer 789dbb4a27204ffbb68d7ae40f8f5e4a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ce91a17790324cd884a0521ce67dde67. This is attempt 56: this message will repeat every 5th retry.
W20260130 03:01:10.025054 22580 heartbeater.cc:646] Failed to heartbeat to 127.18.100.125:39157 (235 consecutive failures): Network error: Failed to ping master at 127.18.100.125:39157: Client connection negotiation failed: client connection to 127.18.100.125:39157: connect: Connection refused (error 111)
W20260130 03:01:10.063732 22437 heartbeater.cc:646] Failed to heartbeat to 127.18.100.125:39157 (288 consecutive failures): Network error: Failed to ping master at 127.18.100.125:39157: Client connection negotiation failed: client connection to 127.18.100.125:39157: connect: Connection refused (error 111)
W20260130 03:01:10.381771 22293 heartbeater.cc:646] Failed to heartbeat to 127.18.100.125:39157 (294 consecutive failures): Network error: Failed to ping master at 127.18.100.125:39157: Client connection negotiation failed: client connection to 127.18.100.125:39157: connect: Connection refused (error 111)
I20260130 03:01:13.063944 22923 master_runner.cc:186] Error getting master registration for 127.18.100.125:39157: Network error: Client connection negotiation failed: client connection to 127.18.100.125:39157: connect: Connection refused (error 111),
I20260130 03:01:13.064319 22923 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260130 03:01:14.911484 24645 raft_consensus.cc:493] T be72107f1e024d5788f5d2459c85015c P 789dbb4a27204ffbb68d7ae40f8f5e4a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 03:01:14.912192 24645 raft_consensus.cc:515] T be72107f1e024d5788f5d2459c85015c P 789dbb4a27204ffbb68d7ae40f8f5e4a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "789dbb4a27204ffbb68d7ae40f8f5e4a" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 41825 } } peers { permanent_uuid: "dd32a35972b347b89ceaaad2455aef5a" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 39781 } } peers { permanent_uuid: "111215cc989f4203a6e1517f2291d66f" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 39751 } }
I20260130 03:01:14.913968 24645 leader_election.cc:290] T be72107f1e024d5788f5d2459c85015c P 789dbb4a27204ffbb68d7ae40f8f5e4a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dd32a35972b347b89ceaaad2455aef5a (127.18.100.66:39781), 111215cc989f4203a6e1517f2291d66f (127.18.100.65:39751)
W20260130 03:01:14.915853 22469 leader_election.cc:343] T be72107f1e024d5788f5d2459c85015c P 789dbb4a27204ffbb68d7ae40f8f5e4a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dd32a35972b347b89ceaaad2455aef5a (127.18.100.66:39781): Not found: Tablet not found: be72107f1e024d5788f5d2459c85015c
W20260130 03:01:14.916442 22468 leader_election.cc:343] T be72107f1e024d5788f5d2459c85015c P 789dbb4a27204ffbb68d7ae40f8f5e4a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 111215cc989f4203a6e1517f2291d66f (127.18.100.65:39751): Not found: Tablet not found: be72107f1e024d5788f5d2459c85015c
I20260130 03:01:14.916862 22468 leader_election.cc:304] T be72107f1e024d5788f5d2459c85015c P 789dbb4a27204ffbb68d7ae40f8f5e4a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 789dbb4a27204ffbb68d7ae40f8f5e4a; no voters: 111215cc989f4203a6e1517f2291d66f, dd32a35972b347b89ceaaad2455aef5a
I20260130 03:01:14.917536 24645 raft_consensus.cc:2749] T be72107f1e024d5788f5d2459c85015c P 789dbb4a27204ffbb68d7ae40f8f5e4a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260130 03:01:15.204071 22179 consensus_peers.cc:597] T 476f8fd240bc48f1a9c47a0351ad5d27 P 111215cc989f4203a6e1517f2291d66f -> Peer 789dbb4a27204ffbb68d7ae40f8f5e4a (127.18.100.67:41825): Couldn't send request to peer 789dbb4a27204ffbb68d7ae40f8f5e4a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 476f8fd240bc48f1a9c47a0351ad5d27. This is attempt 61: this message will repeat every 5th retry.
I20260130 03:01:17.559718 24682 raft_consensus.cc:493] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 342be83f20a648fd849c7d23454fba08)
I20260130 03:01:17.560387 24682 raft_consensus.cc:515] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22dedcf69e00438999d3a5fedde57324" member_type: VOTER last_known_addr { host: "127.18.100.126" port: 42077 } } peers { permanent_uuid: "b01057b6c5ff4fc4b07253801881fe13" member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39157 } } peers { permanent_uuid: "342be83f20a648fd849c7d23454fba08" member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34987 } }
I20260130 03:01:17.562553 24682 leader_election.cc:290] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers b01057b6c5ff4fc4b07253801881fe13 (127.18.100.125:39157), 342be83f20a648fd849c7d23454fba08 (127.18.100.124:34987)
W20260130 03:01:17.565821 22970 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.18.100.125:39157: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20260130 03:01:17.569109 22968 leader_election.cc:336] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 342be83f20a648fd849c7d23454fba08 (127.18.100.124:34987): Network error: Client connection negotiation failed: client connection to 127.18.100.124:34987: connect: Connection refused (error 111)
W20260130 03:01:17.569581 22970 leader_election.cc:336] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer b01057b6c5ff4fc4b07253801881fe13 (127.18.100.125:39157): Network error: Client connection negotiation failed: client connection to 127.18.100.125:39157: connect: Connection refused (error 111)
I20260130 03:01:17.569897 22970 leader_election.cc:304] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22dedcf69e00438999d3a5fedde57324; no voters: 342be83f20a648fd849c7d23454fba08, b01057b6c5ff4fc4b07253801881fe13
I20260130 03:01:17.570477 24682 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260130 03:01:17.937896 22972 maintenance_manager.cc:419] P 22dedcf69e00438999d3a5fedde57324: Scheduling FlushMRSOp(00000000000000000000000000000000): perf score=0.033373
I20260130 03:01:17.949448 24691 raft_consensus.cc:493] T 563d05e06b1f48b4be140219da5c9dbd P 111215cc989f4203a6e1517f2291d66f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 03:01:17.949980 24691 raft_consensus.cc:515] T 563d05e06b1f48b4be140219da5c9dbd P 111215cc989f4203a6e1517f2291d66f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111215cc989f4203a6e1517f2291d66f" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 39751 } } peers { permanent_uuid: "dd32a35972b347b89ceaaad2455aef5a" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 39781 } } peers { permanent_uuid: "789dbb4a27204ffbb68d7ae40f8f5e4a" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 41825 } }
I20260130 03:01:17.951743 24691 leader_election.cc:290] T 563d05e06b1f48b4be140219da5c9dbd P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dd32a35972b347b89ceaaad2455aef5a (127.18.100.66:39781), 789dbb4a27204ffbb68d7ae40f8f5e4a (127.18.100.67:41825)
W20260130 03:01:17.953676 22180 leader_election.cc:343] T 563d05e06b1f48b4be140219da5c9dbd P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dd32a35972b347b89ceaaad2455aef5a (127.18.100.66:39781): Not found: Tablet not found: 563d05e06b1f48b4be140219da5c9dbd
W20260130 03:01:17.954248 22179 leader_election.cc:343] T 563d05e06b1f48b4be140219da5c9dbd P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 789dbb4a27204ffbb68d7ae40f8f5e4a (127.18.100.67:41825): Not found: Tablet not found: 563d05e06b1f48b4be140219da5c9dbd
I20260130 03:01:17.954627 22179 leader_election.cc:304] T 563d05e06b1f48b4be140219da5c9dbd P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 111215cc989f4203a6e1517f2291d66f; no voters: 789dbb4a27204ffbb68d7ae40f8f5e4a, dd32a35972b347b89ceaaad2455aef5a
I20260130 03:01:17.955274 24691 raft_consensus.cc:2749] T 563d05e06b1f48b4be140219da5c9dbd P 111215cc989f4203a6e1517f2291d66f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260130 03:01:18.034910 22971 maintenance_manager.cc:643] P 22dedcf69e00438999d3a5fedde57324: FlushMRSOp(00000000000000000000000000000000) complete. Timing: real 0.096s user 0.043s sys 0.021s Metrics: {"bytes_written":11819,"cfile_init":1,"drs_written":1,"lbm_read_time_us":224,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2371,"lbm_writes_lt_1ms":30,"peak_mem_usage":0,"rows_written":20,"thread_start_us":761,"threads_started":1,"wal-append.queue_time_us":1123}
I20260130 03:01:18.037286 22972 maintenance_manager.cc:419] P 22dedcf69e00438999d3a5fedde57324: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 1246 bytes on disk
I20260130 03:01:18.039191 22971 maintenance_manager.cc:643] P 22dedcf69e00438999d3a5fedde57324: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.001s user 0.001s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":250,"lbm_reads_lt_1ms":4}
I20260130 03:01:18.070432 22923 master_runner.cc:186] Error getting master registration for 127.18.100.125:39157: Network error: Client connection negotiation failed: client connection to 127.18.100.125:39157: connect: Connection refused (error 111),
I20260130 03:01:18.070884 22923 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260130 03:01:18.086903 24696 raft_consensus.cc:493] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 342be83f20a648fd849c7d23454fba08)
I20260130 03:01:18.087476 24696 raft_consensus.cc:515] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22dedcf69e00438999d3a5fedde57324" member_type: VOTER last_known_addr { host: "127.18.100.126" port: 42077 } } peers { permanent_uuid: "b01057b6c5ff4fc4b07253801881fe13" member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39157 } } peers { permanent_uuid: "342be83f20a648fd849c7d23454fba08" member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34987 } }
I20260130 03:01:18.089368 24696 leader_election.cc:290] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers b01057b6c5ff4fc4b07253801881fe13 (127.18.100.125:39157), 342be83f20a648fd849c7d23454fba08 (127.18.100.124:34987)
W20260130 03:01:18.093653 22970 leader_election.cc:336] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer b01057b6c5ff4fc4b07253801881fe13 (127.18.100.125:39157): Network error: Client connection negotiation failed: client connection to 127.18.100.125:39157: connect: Connection refused (error 111)
W20260130 03:01:18.094120 22968 leader_election.cc:336] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 342be83f20a648fd849c7d23454fba08 (127.18.100.124:34987): Network error: Client connection negotiation failed: client connection to 127.18.100.124:34987: connect: Connection refused (error 111)
I20260130 03:01:18.094489 22968 leader_election.cc:304] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22dedcf69e00438999d3a5fedde57324; no voters: 342be83f20a648fd849c7d23454fba08, b01057b6c5ff4fc4b07253801881fe13
I20260130 03:01:18.095110 24696 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260130 03:01:19.505072 24709 raft_consensus.cc:493] T 5cc07f1d50624e03b1f5498b12ea680b P 111215cc989f4203a6e1517f2291d66f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 03:01:19.505542 24709 raft_consensus.cc:515] T 5cc07f1d50624e03b1f5498b12ea680b P 111215cc989f4203a6e1517f2291d66f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111215cc989f4203a6e1517f2291d66f" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 39751 } } peers { permanent_uuid: "dd32a35972b347b89ceaaad2455aef5a" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 39781 } } peers { permanent_uuid: "789dbb4a27204ffbb68d7ae40f8f5e4a" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 41825 } }
I20260130 03:01:19.507097 24709 leader_election.cc:290] T 5cc07f1d50624e03b1f5498b12ea680b P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dd32a35972b347b89ceaaad2455aef5a (127.18.100.66:39781), 789dbb4a27204ffbb68d7ae40f8f5e4a (127.18.100.67:41825)
W20260130 03:01:19.508891 22180 leader_election.cc:343] T 5cc07f1d50624e03b1f5498b12ea680b P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dd32a35972b347b89ceaaad2455aef5a (127.18.100.66:39781): Not found: Tablet not found: 5cc07f1d50624e03b1f5498b12ea680b
W20260130 03:01:19.509392 22179 leader_election.cc:343] T 5cc07f1d50624e03b1f5498b12ea680b P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 789dbb4a27204ffbb68d7ae40f8f5e4a (127.18.100.67:41825): Not found: Tablet not found: 5cc07f1d50624e03b1f5498b12ea680b
I20260130 03:01:19.509711 22179 leader_election.cc:304] T 5cc07f1d50624e03b1f5498b12ea680b P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 111215cc989f4203a6e1517f2291d66f; no voters: 789dbb4a27204ffbb68d7ae40f8f5e4a, dd32a35972b347b89ceaaad2455aef5a
I20260130 03:01:19.510208 24709 raft_consensus.cc:2749] T 5cc07f1d50624e03b1f5498b12ea680b P 111215cc989f4203a6e1517f2291d66f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260130 03:01:20.355324 22321 consensus_peers.cc:597] T ce91a17790324cd884a0521ce67dde67 P dd32a35972b347b89ceaaad2455aef5a -> Peer 789dbb4a27204ffbb68d7ae40f8f5e4a (127.18.100.67:41825): Couldn't send request to peer 789dbb4a27204ffbb68d7ae40f8f5e4a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ce91a17790324cd884a0521ce67dde67. This is attempt 61: this message will repeat every 5th retry.
I20260130 03:01:21.205030 24737 raft_consensus.cc:493] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 342be83f20a648fd849c7d23454fba08)
I20260130 03:01:21.205545 24737 raft_consensus.cc:515] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22dedcf69e00438999d3a5fedde57324" member_type: VOTER last_known_addr { host: "127.18.100.126" port: 42077 } } peers { permanent_uuid: "b01057b6c5ff4fc4b07253801881fe13" member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39157 } } peers { permanent_uuid: "342be83f20a648fd849c7d23454fba08" member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34987 } }
I20260130 03:01:21.207264 24737 leader_election.cc:290] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers b01057b6c5ff4fc4b07253801881fe13 (127.18.100.125:39157), 342be83f20a648fd849c7d23454fba08 (127.18.100.124:34987)
W20260130 03:01:21.212944 22968 leader_election.cc:336] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 342be83f20a648fd849c7d23454fba08 (127.18.100.124:34987): Network error: Client connection negotiation failed: client connection to 127.18.100.124:34987: connect: Connection refused (error 111)
W20260130 03:01:21.213485 22970 leader_election.cc:336] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer b01057b6c5ff4fc4b07253801881fe13 (127.18.100.125:39157): Network error: Client connection negotiation failed: client connection to 127.18.100.125:39157: connect: Connection refused (error 111)
I20260130 03:01:21.213811 22970 leader_election.cc:304] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22dedcf69e00438999d3a5fedde57324; no voters: 342be83f20a648fd849c7d23454fba08, b01057b6c5ff4fc4b07253801881fe13
I20260130 03:01:21.214444 24737 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260130 03:01:23.077135 22923 master_runner.cc:186] Error getting master registration for 127.18.100.125:39157: Network error: Client connection negotiation failed: client connection to 127.18.100.125:39157: connect: Connection refused (error 111),
I20260130 03:01:23.077625 22923 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260130 03:01:25.140636 22179 consensus_peers.cc:597] T 476f8fd240bc48f1a9c47a0351ad5d27 P 111215cc989f4203a6e1517f2291d66f -> Peer 789dbb4a27204ffbb68d7ae40f8f5e4a (127.18.100.67:41825): Couldn't send request to peer 789dbb4a27204ffbb68d7ae40f8f5e4a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 476f8fd240bc48f1a9c47a0351ad5d27. This is attempt 66: this message will repeat every 5th retry.
I20260130 03:01:26.321108 24811 raft_consensus.cc:493] T be72107f1e024d5788f5d2459c85015c P 789dbb4a27204ffbb68d7ae40f8f5e4a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 03:01:26.321918 24811 raft_consensus.cc:515] T be72107f1e024d5788f5d2459c85015c P 789dbb4a27204ffbb68d7ae40f8f5e4a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "789dbb4a27204ffbb68d7ae40f8f5e4a" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 41825 } } peers { permanent_uuid: "dd32a35972b347b89ceaaad2455aef5a" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 39781 } } peers { permanent_uuid: "111215cc989f4203a6e1517f2291d66f" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 39751 } }
I20260130 03:01:26.324831 24811 leader_election.cc:290] T be72107f1e024d5788f5d2459c85015c P 789dbb4a27204ffbb68d7ae40f8f5e4a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dd32a35972b347b89ceaaad2455aef5a (127.18.100.66:39781), 111215cc989f4203a6e1517f2291d66f (127.18.100.65:39751)
W20260130 03:01:26.327219 22469 leader_election.cc:343] T be72107f1e024d5788f5d2459c85015c P 789dbb4a27204ffbb68d7ae40f8f5e4a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dd32a35972b347b89ceaaad2455aef5a (127.18.100.66:39781): Not found: Tablet not found: be72107f1e024d5788f5d2459c85015c
W20260130 03:01:26.327867 22468 leader_election.cc:343] T be72107f1e024d5788f5d2459c85015c P 789dbb4a27204ffbb68d7ae40f8f5e4a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 111215cc989f4203a6e1517f2291d66f (127.18.100.65:39751): Not found: Tablet not found: be72107f1e024d5788f5d2459c85015c
I20260130 03:01:26.328258 22468 leader_election.cc:304] T be72107f1e024d5788f5d2459c85015c P 789dbb4a27204ffbb68d7ae40f8f5e4a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 789dbb4a27204ffbb68d7ae40f8f5e4a; no voters: 111215cc989f4203a6e1517f2291d66f, dd32a35972b347b89ceaaad2455aef5a
I20260130 03:01:26.328917 24811 raft_consensus.cc:2749] T be72107f1e024d5788f5d2459c85015c P 789dbb4a27204ffbb68d7ae40f8f5e4a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260130 03:01:27.597172 24823 raft_consensus.cc:493] T 5cc07f1d50624e03b1f5498b12ea680b P 111215cc989f4203a6e1517f2291d66f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 03:01:27.597705 24823 raft_consensus.cc:515] T 5cc07f1d50624e03b1f5498b12ea680b P 111215cc989f4203a6e1517f2291d66f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111215cc989f4203a6e1517f2291d66f" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 39751 } } peers { permanent_uuid: "dd32a35972b347b89ceaaad2455aef5a" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 39781 } } peers { permanent_uuid: "789dbb4a27204ffbb68d7ae40f8f5e4a" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 41825 } }
I20260130 03:01:27.600466 24823 leader_election.cc:290] T 5cc07f1d50624e03b1f5498b12ea680b P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dd32a35972b347b89ceaaad2455aef5a (127.18.100.66:39781), 789dbb4a27204ffbb68d7ae40f8f5e4a (127.18.100.67:41825)
W20260130 03:01:27.602293 22180 leader_election.cc:343] T 5cc07f1d50624e03b1f5498b12ea680b P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dd32a35972b347b89ceaaad2455aef5a (127.18.100.66:39781): Not found: Tablet not found: 5cc07f1d50624e03b1f5498b12ea680b
W20260130 03:01:27.602747 22179 leader_election.cc:343] T 5cc07f1d50624e03b1f5498b12ea680b P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 789dbb4a27204ffbb68d7ae40f8f5e4a (127.18.100.67:41825): Not found: Tablet not found: 5cc07f1d50624e03b1f5498b12ea680b
I20260130 03:01:27.603124 22179 leader_election.cc:304] T 5cc07f1d50624e03b1f5498b12ea680b P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 111215cc989f4203a6e1517f2291d66f; no voters: 789dbb4a27204ffbb68d7ae40f8f5e4a, dd32a35972b347b89ceaaad2455aef5a
I20260130 03:01:27.603693 24823 raft_consensus.cc:2749] T 5cc07f1d50624e03b1f5498b12ea680b P 111215cc989f4203a6e1517f2291d66f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260130 03:01:28.084306 22923 master_runner.cc:186] Error getting master registration for 127.18.100.125:39157: Network error: Client connection negotiation failed: client connection to 127.18.100.125:39157: connect: Connection refused (error 111),
I20260130 03:01:28.084990 22923 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260130 03:01:29.891968 22321 consensus_peers.cc:597] T ce91a17790324cd884a0521ce67dde67 P dd32a35972b347b89ceaaad2455aef5a -> Peer 789dbb4a27204ffbb68d7ae40f8f5e4a (127.18.100.67:41825): Couldn't send request to peer 789dbb4a27204ffbb68d7ae40f8f5e4a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ce91a17790324cd884a0521ce67dde67. This is attempt 66: this message will repeat every 5th retry.
I20260130 03:01:31.466192 24898 raft_consensus.cc:493] T 563d05e06b1f48b4be140219da5c9dbd P 111215cc989f4203a6e1517f2291d66f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 03:01:31.466698 24898 raft_consensus.cc:515] T 563d05e06b1f48b4be140219da5c9dbd P 111215cc989f4203a6e1517f2291d66f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111215cc989f4203a6e1517f2291d66f" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 39751 } } peers { permanent_uuid: "dd32a35972b347b89ceaaad2455aef5a" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 39781 } } peers { permanent_uuid: "789dbb4a27204ffbb68d7ae40f8f5e4a" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 41825 } }
I20260130 03:01:31.468580 24898 leader_election.cc:290] T 563d05e06b1f48b4be140219da5c9dbd P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dd32a35972b347b89ceaaad2455aef5a (127.18.100.66:39781), 789dbb4a27204ffbb68d7ae40f8f5e4a (127.18.100.67:41825)
W20260130 03:01:31.470152 22180 leader_election.cc:343] T 563d05e06b1f48b4be140219da5c9dbd P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dd32a35972b347b89ceaaad2455aef5a (127.18.100.66:39781): Not found: Tablet not found: 563d05e06b1f48b4be140219da5c9dbd
W20260130 03:01:31.470600 22179 leader_election.cc:343] T 563d05e06b1f48b4be140219da5c9dbd P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 789dbb4a27204ffbb68d7ae40f8f5e4a (127.18.100.67:41825): Not found: Tablet not found: 563d05e06b1f48b4be140219da5c9dbd
I20260130 03:01:31.470955 22179 leader_election.cc:304] T 563d05e06b1f48b4be140219da5c9dbd P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 111215cc989f4203a6e1517f2291d66f; no voters: 789dbb4a27204ffbb68d7ae40f8f5e4a, dd32a35972b347b89ceaaad2455aef5a
I20260130 03:01:31.471699 24898 raft_consensus.cc:2749] T 563d05e06b1f48b4be140219da5c9dbd P 111215cc989f4203a6e1517f2291d66f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260130 03:01:33.091421 22923 master_runner.cc:186] Error getting master registration for 127.18.100.125:39157: Network error: Client connection negotiation failed: client connection to 127.18.100.125:39157: connect: Connection refused (error 111),
I20260130 03:01:33.091841 22923 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260130 03:01:33.272488 24932 raft_consensus.cc:493] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 342be83f20a648fd849c7d23454fba08)
I20260130 03:01:33.273121 24932 raft_consensus.cc:515] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22dedcf69e00438999d3a5fedde57324" member_type: VOTER last_known_addr { host: "127.18.100.126" port: 42077 } } peers { permanent_uuid: "b01057b6c5ff4fc4b07253801881fe13" member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39157 } } peers { permanent_uuid: "342be83f20a648fd849c7d23454fba08" member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34987 } }
I20260130 03:01:33.275008 24932 leader_election.cc:290] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers b01057b6c5ff4fc4b07253801881fe13 (127.18.100.125:39157), 342be83f20a648fd849c7d23454fba08 (127.18.100.124:34987)
W20260130 03:01:33.276533 22970 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.18.100.125:39157: connect: Connection refused (error 111) [suppressed 5 similar messages]
W20260130 03:01:33.280136 22970 leader_election.cc:336] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer b01057b6c5ff4fc4b07253801881fe13 (127.18.100.125:39157): Network error: Client connection negotiation failed: client connection to 127.18.100.125:39157: connect: Connection refused (error 111)
W20260130 03:01:33.280812 22968 leader_election.cc:336] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 342be83f20a648fd849c7d23454fba08 (127.18.100.124:34987): Network error: Client connection negotiation failed: client connection to 127.18.100.124:34987: connect: Connection refused (error 111)
I20260130 03:01:33.281131 22968 leader_election.cc:304] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22dedcf69e00438999d3a5fedde57324; no voters: 342be83f20a648fd849c7d23454fba08, b01057b6c5ff4fc4b07253801881fe13
I20260130 03:01:33.281756 24932 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260130 03:01:35.098532 22179 consensus_peers.cc:597] T 476f8fd240bc48f1a9c47a0351ad5d27 P 111215cc989f4203a6e1517f2291d66f -> Peer 789dbb4a27204ffbb68d7ae40f8f5e4a (127.18.100.67:41825): Couldn't send request to peer 789dbb4a27204ffbb68d7ae40f8f5e4a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 476f8fd240bc48f1a9c47a0351ad5d27. This is attempt 71: this message will repeat every 5th retry.
W20260130 03:01:35.148533 22433 debug-util.cc:398] Leaking SignalData structure 0x7b080003e900 after lost signal to thread 22304
W20260130 03:01:35.149379 22433 debug-util.cc:398] Leaking SignalData structure 0x7b08000cc6c0 after lost signal to thread 22436
I20260130 03:01:35.668984 24972 raft_consensus.cc:493] T 563d05e06b1f48b4be140219da5c9dbd P 111215cc989f4203a6e1517f2291d66f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 03:01:35.669689 24972 raft_consensus.cc:515] T 563d05e06b1f48b4be140219da5c9dbd P 111215cc989f4203a6e1517f2291d66f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111215cc989f4203a6e1517f2291d66f" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 39751 } } peers { permanent_uuid: "dd32a35972b347b89ceaaad2455aef5a" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 39781 } } peers { permanent_uuid: "789dbb4a27204ffbb68d7ae40f8f5e4a" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 41825 } }
I20260130 03:01:35.671945 24972 leader_election.cc:290] T 563d05e06b1f48b4be140219da5c9dbd P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dd32a35972b347b89ceaaad2455aef5a (127.18.100.66:39781), 789dbb4a27204ffbb68d7ae40f8f5e4a (127.18.100.67:41825)
W20260130 03:01:35.673550 22180 leader_election.cc:343] T 563d05e06b1f48b4be140219da5c9dbd P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dd32a35972b347b89ceaaad2455aef5a (127.18.100.66:39781): Not found: Tablet not found: 563d05e06b1f48b4be140219da5c9dbd
W20260130 03:01:35.673943 22179 leader_election.cc:343] T 563d05e06b1f48b4be140219da5c9dbd P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 789dbb4a27204ffbb68d7ae40f8f5e4a (127.18.100.67:41825): Not found: Tablet not found: 563d05e06b1f48b4be140219da5c9dbd
I20260130 03:01:35.674292 22179 leader_election.cc:304] T 563d05e06b1f48b4be140219da5c9dbd P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 111215cc989f4203a6e1517f2291d66f; no voters: 789dbb4a27204ffbb68d7ae40f8f5e4a, dd32a35972b347b89ceaaad2455aef5a
I20260130 03:01:35.674996 24972 raft_consensus.cc:2749] T 563d05e06b1f48b4be140219da5c9dbd P 111215cc989f4203a6e1517f2291d66f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260130 03:01:35.675587 24972 raft_consensus.cc:493] T 5cc07f1d50624e03b1f5498b12ea680b P 111215cc989f4203a6e1517f2291d66f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 03:01:35.675912 24972 raft_consensus.cc:515] T 5cc07f1d50624e03b1f5498b12ea680b P 111215cc989f4203a6e1517f2291d66f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111215cc989f4203a6e1517f2291d66f" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 39751 } } peers { permanent_uuid: "dd32a35972b347b89ceaaad2455aef5a" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 39781 } } peers { permanent_uuid: "789dbb4a27204ffbb68d7ae40f8f5e4a" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 41825 } }
I20260130 03:01:35.677367 24972 leader_election.cc:290] T 5cc07f1d50624e03b1f5498b12ea680b P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dd32a35972b347b89ceaaad2455aef5a (127.18.100.66:39781), 789dbb4a27204ffbb68d7ae40f8f5e4a (127.18.100.67:41825)
W20260130 03:01:35.678718 22180 leader_election.cc:343] T 5cc07f1d50624e03b1f5498b12ea680b P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dd32a35972b347b89ceaaad2455aef5a (127.18.100.66:39781): Not found: Tablet not found: 5cc07f1d50624e03b1f5498b12ea680b
W20260130 03:01:35.679210 22179 leader_election.cc:343] T 5cc07f1d50624e03b1f5498b12ea680b P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 789dbb4a27204ffbb68d7ae40f8f5e4a (127.18.100.67:41825): Not found: Tablet not found: 5cc07f1d50624e03b1f5498b12ea680b
I20260130 03:01:35.679576 22179 leader_election.cc:304] T 5cc07f1d50624e03b1f5498b12ea680b P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 111215cc989f4203a6e1517f2291d66f; no voters: 789dbb4a27204ffbb68d7ae40f8f5e4a, dd32a35972b347b89ceaaad2455aef5a
I20260130 03:01:35.680192 24972 raft_consensus.cc:2749] T 5cc07f1d50624e03b1f5498b12ea680b P 111215cc989f4203a6e1517f2291d66f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260130 03:01:38.097929 22923 master_runner.cc:186] Error getting master registration for 127.18.100.125:39157: Network error: Client connection negotiation failed: client connection to 127.18.100.125:39157: connect: Connection refused (error 111),
I20260130 03:01:38.098371 22923 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260130 03:01:39.140022 22321 consensus_peers.cc:597] T ce91a17790324cd884a0521ce67dde67 P dd32a35972b347b89ceaaad2455aef5a -> Peer 789dbb4a27204ffbb68d7ae40f8f5e4a (127.18.100.67:41825): Couldn't send request to peer 789dbb4a27204ffbb68d7ae40f8f5e4a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ce91a17790324cd884a0521ce67dde67. This is attempt 71: this message will repeat every 5th retry.
I20260130 03:01:41.182464 25066 raft_consensus.cc:493] T be72107f1e024d5788f5d2459c85015c P 789dbb4a27204ffbb68d7ae40f8f5e4a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 03:01:41.183199 25066 raft_consensus.cc:515] T be72107f1e024d5788f5d2459c85015c P 789dbb4a27204ffbb68d7ae40f8f5e4a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "789dbb4a27204ffbb68d7ae40f8f5e4a" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 41825 } } peers { permanent_uuid: "dd32a35972b347b89ceaaad2455aef5a" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 39781 } } peers { permanent_uuid: "111215cc989f4203a6e1517f2291d66f" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 39751 } }
I20260130 03:01:41.185215 25066 leader_election.cc:290] T be72107f1e024d5788f5d2459c85015c P 789dbb4a27204ffbb68d7ae40f8f5e4a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dd32a35972b347b89ceaaad2455aef5a (127.18.100.66:39781), 111215cc989f4203a6e1517f2291d66f (127.18.100.65:39751)
W20260130 03:01:41.187400 22469 leader_election.cc:343] T be72107f1e024d5788f5d2459c85015c P 789dbb4a27204ffbb68d7ae40f8f5e4a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dd32a35972b347b89ceaaad2455aef5a (127.18.100.66:39781): Not found: Tablet not found: be72107f1e024d5788f5d2459c85015c
W20260130 03:01:41.187948 22468 leader_election.cc:343] T be72107f1e024d5788f5d2459c85015c P 789dbb4a27204ffbb68d7ae40f8f5e4a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 111215cc989f4203a6e1517f2291d66f (127.18.100.65:39751): Not found: Tablet not found: be72107f1e024d5788f5d2459c85015c
I20260130 03:01:41.188328 22468 leader_election.cc:304] T be72107f1e024d5788f5d2459c85015c P 789dbb4a27204ffbb68d7ae40f8f5e4a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 789dbb4a27204ffbb68d7ae40f8f5e4a; no voters: 111215cc989f4203a6e1517f2291d66f, dd32a35972b347b89ceaaad2455aef5a
I20260130 03:01:41.188961 25066 raft_consensus.cc:2749] T be72107f1e024d5788f5d2459c85015c P 789dbb4a27204ffbb68d7ae40f8f5e4a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260130 03:01:41.900362 25077 raft_consensus.cc:493] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 342be83f20a648fd849c7d23454fba08)
I20260130 03:01:41.900920 25077 raft_consensus.cc:515] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22dedcf69e00438999d3a5fedde57324" member_type: VOTER last_known_addr { host: "127.18.100.126" port: 42077 } } peers { permanent_uuid: "b01057b6c5ff4fc4b07253801881fe13" member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39157 } } peers { permanent_uuid: "342be83f20a648fd849c7d23454fba08" member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34987 } }
I20260130 03:01:41.902539 25077 leader_election.cc:290] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers b01057b6c5ff4fc4b07253801881fe13 (127.18.100.125:39157), 342be83f20a648fd849c7d23454fba08 (127.18.100.124:34987)
W20260130 03:01:41.905120 22970 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.18.100.125:39157: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20260130 03:01:41.908082 22968 leader_election.cc:336] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 342be83f20a648fd849c7d23454fba08 (127.18.100.124:34987): Network error: Client connection negotiation failed: client connection to 127.18.100.124:34987: connect: Connection refused (error 111)
W20260130 03:01:41.908581 22970 leader_election.cc:336] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer b01057b6c5ff4fc4b07253801881fe13 (127.18.100.125:39157): Network error: Client connection negotiation failed: client connection to 127.18.100.125:39157: connect: Connection refused (error 111)
I20260130 03:01:41.908890 22970 leader_election.cc:304] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22dedcf69e00438999d3a5fedde57324; no voters: 342be83f20a648fd849c7d23454fba08, b01057b6c5ff4fc4b07253801881fe13
I20260130 03:01:41.909443 25077 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260130 03:01:42.559270 25084 raft_consensus.cc:493] T 563d05e06b1f48b4be140219da5c9dbd P 111215cc989f4203a6e1517f2291d66f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 03:01:42.559793 25084 raft_consensus.cc:515] T 563d05e06b1f48b4be140219da5c9dbd P 111215cc989f4203a6e1517f2291d66f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111215cc989f4203a6e1517f2291d66f" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 39751 } } peers { permanent_uuid: "dd32a35972b347b89ceaaad2455aef5a" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 39781 } } peers { permanent_uuid: "789dbb4a27204ffbb68d7ae40f8f5e4a" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 41825 } }
I20260130 03:01:42.561496 25084 leader_election.cc:290] T 563d05e06b1f48b4be140219da5c9dbd P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dd32a35972b347b89ceaaad2455aef5a (127.18.100.66:39781), 789dbb4a27204ffbb68d7ae40f8f5e4a (127.18.100.67:41825)
W20260130 03:01:42.563017 22180 leader_election.cc:343] T 563d05e06b1f48b4be140219da5c9dbd P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dd32a35972b347b89ceaaad2455aef5a (127.18.100.66:39781): Not found: Tablet not found: 563d05e06b1f48b4be140219da5c9dbd
W20260130 03:01:42.563400 22179 leader_election.cc:343] T 563d05e06b1f48b4be140219da5c9dbd P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 789dbb4a27204ffbb68d7ae40f8f5e4a (127.18.100.67:41825): Not found: Tablet not found: 563d05e06b1f48b4be140219da5c9dbd
I20260130 03:01:42.563746 22179 leader_election.cc:304] T 563d05e06b1f48b4be140219da5c9dbd P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 111215cc989f4203a6e1517f2291d66f; no voters: 789dbb4a27204ffbb68d7ae40f8f5e4a, dd32a35972b347b89ceaaad2455aef5a
I20260130 03:01:42.564275 25084 raft_consensus.cc:2749] T 563d05e06b1f48b4be140219da5c9dbd P 111215cc989f4203a6e1517f2291d66f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260130 03:01:43.103909 22923 master_runner.cc:186] Error getting master registration for 127.18.100.125:39157: Network error: Client connection negotiation failed: client connection to 127.18.100.125:39157: connect: Connection refused (error 111),
I20260130 03:01:43.104332 22923 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260130 03:01:44.980996 22179 consensus_peers.cc:597] T 476f8fd240bc48f1a9c47a0351ad5d27 P 111215cc989f4203a6e1517f2291d66f -> Peer 789dbb4a27204ffbb68d7ae40f8f5e4a (127.18.100.67:41825): Couldn't send request to peer 789dbb4a27204ffbb68d7ae40f8f5e4a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 476f8fd240bc48f1a9c47a0351ad5d27. This is attempt 76: this message will repeat every 5th retry.
I20260130 03:01:46.038641 25149 raft_consensus.cc:493] T 563d05e06b1f48b4be140219da5c9dbd P 111215cc989f4203a6e1517f2291d66f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 03:01:46.039152 25149 raft_consensus.cc:515] T 563d05e06b1f48b4be140219da5c9dbd P 111215cc989f4203a6e1517f2291d66f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111215cc989f4203a6e1517f2291d66f" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 39751 } } peers { permanent_uuid: "dd32a35972b347b89ceaaad2455aef5a" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 39781 } } peers { permanent_uuid: "789dbb4a27204ffbb68d7ae40f8f5e4a" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 41825 } }
I20260130 03:01:46.041424 25149 leader_election.cc:290] T 563d05e06b1f48b4be140219da5c9dbd P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dd32a35972b347b89ceaaad2455aef5a (127.18.100.66:39781), 789dbb4a27204ffbb68d7ae40f8f5e4a (127.18.100.67:41825)
W20260130 03:01:46.043846 22179 leader_election.cc:343] T 563d05e06b1f48b4be140219da5c9dbd P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 789dbb4a27204ffbb68d7ae40f8f5e4a (127.18.100.67:41825): Not found: Tablet not found: 563d05e06b1f48b4be140219da5c9dbd
W20260130 03:01:46.044483 22180 leader_election.cc:343] T 563d05e06b1f48b4be140219da5c9dbd P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dd32a35972b347b89ceaaad2455aef5a (127.18.100.66:39781): Not found: Tablet not found: 563d05e06b1f48b4be140219da5c9dbd
I20260130 03:01:46.044981 22180 leader_election.cc:304] T 563d05e06b1f48b4be140219da5c9dbd P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 111215cc989f4203a6e1517f2291d66f; no voters: 789dbb4a27204ffbb68d7ae40f8f5e4a, dd32a35972b347b89ceaaad2455aef5a
I20260130 03:01:46.045966 25149 raft_consensus.cc:2749] T 563d05e06b1f48b4be140219da5c9dbd P 111215cc989f4203a6e1517f2291d66f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260130 03:01:48.111071 22923 master_runner.cc:186] Error getting master registration for 127.18.100.125:39157: Network error: Client connection negotiation failed: client connection to 127.18.100.125:39157: connect: Connection refused (error 111),
I20260130 03:01:48.111558 22923 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260130 03:01:48.972503 22321 consensus_peers.cc:597] T ce91a17790324cd884a0521ce67dde67 P dd32a35972b347b89ceaaad2455aef5a -> Peer 789dbb4a27204ffbb68d7ae40f8f5e4a (127.18.100.67:41825): Couldn't send request to peer 789dbb4a27204ffbb68d7ae40f8f5e4a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ce91a17790324cd884a0521ce67dde67. This is attempt 76: this message will repeat every 5th retry.
I20260130 03:01:50.349023 25227 raft_consensus.cc:493] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 342be83f20a648fd849c7d23454fba08)
I20260130 03:01:50.349738 25227 raft_consensus.cc:515] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22dedcf69e00438999d3a5fedde57324" member_type: VOTER last_known_addr { host: "127.18.100.126" port: 42077 } } peers { permanent_uuid: "b01057b6c5ff4fc4b07253801881fe13" member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39157 } } peers { permanent_uuid: "342be83f20a648fd849c7d23454fba08" member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34987 } }
I20260130 03:01:50.351805 25227 leader_election.cc:290] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers b01057b6c5ff4fc4b07253801881fe13 (127.18.100.125:39157), 342be83f20a648fd849c7d23454fba08 (127.18.100.124:34987)
W20260130 03:01:50.354032 22970 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.18.100.125:39157: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20260130 03:01:50.356717 22968 leader_election.cc:336] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 342be83f20a648fd849c7d23454fba08 (127.18.100.124:34987): Network error: Client connection negotiation failed: client connection to 127.18.100.124:34987: connect: Connection refused (error 111)
W20260130 03:01:50.357169 22970 leader_election.cc:336] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer b01057b6c5ff4fc4b07253801881fe13 (127.18.100.125:39157): Network error: Client connection negotiation failed: client connection to 127.18.100.125:39157: connect: Connection refused (error 111)
I20260130 03:01:50.357517 22970 leader_election.cc:304] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22dedcf69e00438999d3a5fedde57324; no voters: 342be83f20a648fd849c7d23454fba08, b01057b6c5ff4fc4b07253801881fe13
I20260130 03:01:50.358085 25227 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260130 03:01:51.453088 25246 raft_consensus.cc:493] T 5cc07f1d50624e03b1f5498b12ea680b P 111215cc989f4203a6e1517f2291d66f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 03:01:51.453631 25246 raft_consensus.cc:515] T 5cc07f1d50624e03b1f5498b12ea680b P 111215cc989f4203a6e1517f2291d66f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111215cc989f4203a6e1517f2291d66f" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 39751 } } peers { permanent_uuid: "dd32a35972b347b89ceaaad2455aef5a" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 39781 } } peers { permanent_uuid: "789dbb4a27204ffbb68d7ae40f8f5e4a" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 41825 } }
I20260130 03:01:51.455384 25246 leader_election.cc:290] T 5cc07f1d50624e03b1f5498b12ea680b P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dd32a35972b347b89ceaaad2455aef5a (127.18.100.66:39781), 789dbb4a27204ffbb68d7ae40f8f5e4a (127.18.100.67:41825)
W20260130 03:01:51.457170 22180 leader_election.cc:343] T 5cc07f1d50624e03b1f5498b12ea680b P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dd32a35972b347b89ceaaad2455aef5a (127.18.100.66:39781): Not found: Tablet not found: 5cc07f1d50624e03b1f5498b12ea680b
W20260130 03:01:51.457576 22179 leader_election.cc:343] T 5cc07f1d50624e03b1f5498b12ea680b P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 789dbb4a27204ffbb68d7ae40f8f5e4a (127.18.100.67:41825): Not found: Tablet not found: 5cc07f1d50624e03b1f5498b12ea680b
I20260130 03:01:51.457845 22179 leader_election.cc:304] T 5cc07f1d50624e03b1f5498b12ea680b P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 111215cc989f4203a6e1517f2291d66f; no voters: 789dbb4a27204ffbb68d7ae40f8f5e4a, dd32a35972b347b89ceaaad2455aef5a
I20260130 03:01:51.458379 25246 raft_consensus.cc:2749] T 5cc07f1d50624e03b1f5498b12ea680b P 111215cc989f4203a6e1517f2291d66f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260130 03:01:53.117668 22923 master_runner.cc:186] Error getting master registration for 127.18.100.125:39157: Network error: Client connection negotiation failed: client connection to 127.18.100.125:39157: connect: Connection refused (error 111),
I20260130 03:01:53.118079 22923 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260130 03:01:55.252995 22179 consensus_peers.cc:597] T 476f8fd240bc48f1a9c47a0351ad5d27 P 111215cc989f4203a6e1517f2291d66f -> Peer 789dbb4a27204ffbb68d7ae40f8f5e4a (127.18.100.67:41825): Couldn't send request to peer 789dbb4a27204ffbb68d7ae40f8f5e4a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 476f8fd240bc48f1a9c47a0351ad5d27. This is attempt 81: this message will repeat every 5th retry.
I20260130 03:01:55.781853 25303 raft_consensus.cc:493] T 5cc07f1d50624e03b1f5498b12ea680b P 111215cc989f4203a6e1517f2291d66f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 03:01:55.782357 25303 raft_consensus.cc:515] T 5cc07f1d50624e03b1f5498b12ea680b P 111215cc989f4203a6e1517f2291d66f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111215cc989f4203a6e1517f2291d66f" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 39751 } } peers { permanent_uuid: "dd32a35972b347b89ceaaad2455aef5a" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 39781 } } peers { permanent_uuid: "789dbb4a27204ffbb68d7ae40f8f5e4a" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 41825 } }
I20260130 03:01:55.784178 25303 leader_election.cc:290] T 5cc07f1d50624e03b1f5498b12ea680b P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dd32a35972b347b89ceaaad2455aef5a (127.18.100.66:39781), 789dbb4a27204ffbb68d7ae40f8f5e4a (127.18.100.67:41825)
W20260130 03:01:55.785861 22180 leader_election.cc:343] T 5cc07f1d50624e03b1f5498b12ea680b P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dd32a35972b347b89ceaaad2455aef5a (127.18.100.66:39781): Not found: Tablet not found: 5cc07f1d50624e03b1f5498b12ea680b
W20260130 03:01:55.786249 22179 leader_election.cc:343] T 5cc07f1d50624e03b1f5498b12ea680b P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 789dbb4a27204ffbb68d7ae40f8f5e4a (127.18.100.67:41825): Not found: Tablet not found: 5cc07f1d50624e03b1f5498b12ea680b
I20260130 03:01:55.786656 22179 leader_election.cc:304] T 5cc07f1d50624e03b1f5498b12ea680b P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 111215cc989f4203a6e1517f2291d66f; no voters: 789dbb4a27204ffbb68d7ae40f8f5e4a, dd32a35972b347b89ceaaad2455aef5a
I20260130 03:01:55.787273 25303 raft_consensus.cc:2749] T 5cc07f1d50624e03b1f5498b12ea680b P 111215cc989f4203a6e1517f2291d66f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260130 03:01:57.246021 25338 raft_consensus.cc:493] T be72107f1e024d5788f5d2459c85015c P 789dbb4a27204ffbb68d7ae40f8f5e4a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 03:01:57.246837 25338 raft_consensus.cc:515] T be72107f1e024d5788f5d2459c85015c P 789dbb4a27204ffbb68d7ae40f8f5e4a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "789dbb4a27204ffbb68d7ae40f8f5e4a" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 41825 } } peers { permanent_uuid: "dd32a35972b347b89ceaaad2455aef5a" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 39781 } } peers { permanent_uuid: "111215cc989f4203a6e1517f2291d66f" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 39751 } }
I20260130 03:01:57.249356 25338 leader_election.cc:290] T be72107f1e024d5788f5d2459c85015c P 789dbb4a27204ffbb68d7ae40f8f5e4a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dd32a35972b347b89ceaaad2455aef5a (127.18.100.66:39781), 111215cc989f4203a6e1517f2291d66f (127.18.100.65:39751)
W20260130 03:01:57.251416 22469 leader_election.cc:343] T be72107f1e024d5788f5d2459c85015c P 789dbb4a27204ffbb68d7ae40f8f5e4a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dd32a35972b347b89ceaaad2455aef5a (127.18.100.66:39781): Not found: Tablet not found: be72107f1e024d5788f5d2459c85015c
W20260130 03:01:57.252025 22468 leader_election.cc:343] T be72107f1e024d5788f5d2459c85015c P 789dbb4a27204ffbb68d7ae40f8f5e4a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 111215cc989f4203a6e1517f2291d66f (127.18.100.65:39751): Not found: Tablet not found: be72107f1e024d5788f5d2459c85015c
I20260130 03:01:57.252383 22468 leader_election.cc:304] T be72107f1e024d5788f5d2459c85015c P 789dbb4a27204ffbb68d7ae40f8f5e4a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 789dbb4a27204ffbb68d7ae40f8f5e4a; no voters: 111215cc989f4203a6e1517f2291d66f, dd32a35972b347b89ceaaad2455aef5a
I20260130 03:01:57.253029 25338 raft_consensus.cc:2749] T be72107f1e024d5788f5d2459c85015c P 789dbb4a27204ffbb68d7ae40f8f5e4a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260130 03:01:58.125738 22923 master_runner.cc:186] Error getting master registration for 127.18.100.125:39157: Network error: Client connection negotiation failed: client connection to 127.18.100.125:39157: connect: Connection refused (error 111),
I20260130 03:01:58.126286 22923 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260130 03:01:58.662330 22321 consensus_peers.cc:597] T ce91a17790324cd884a0521ce67dde67 P dd32a35972b347b89ceaaad2455aef5a -> Peer 789dbb4a27204ffbb68d7ae40f8f5e4a (127.18.100.67:41825): Couldn't send request to peer 789dbb4a27204ffbb68d7ae40f8f5e4a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ce91a17790324cd884a0521ce67dde67. This is attempt 81: this message will repeat every 5th retry.
I20260130 03:02:01.169836 25399 raft_consensus.cc:493] T 563d05e06b1f48b4be140219da5c9dbd P 111215cc989f4203a6e1517f2291d66f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 03:02:01.170332 25399 raft_consensus.cc:515] T 563d05e06b1f48b4be140219da5c9dbd P 111215cc989f4203a6e1517f2291d66f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111215cc989f4203a6e1517f2291d66f" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 39751 } } peers { permanent_uuid: "dd32a35972b347b89ceaaad2455aef5a" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 39781 } } peers { permanent_uuid: "789dbb4a27204ffbb68d7ae40f8f5e4a" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 41825 } }
I20260130 03:02:01.172448 25399 leader_election.cc:290] T 563d05e06b1f48b4be140219da5c9dbd P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dd32a35972b347b89ceaaad2455aef5a (127.18.100.66:39781), 789dbb4a27204ffbb68d7ae40f8f5e4a (127.18.100.67:41825)
W20260130 03:02:01.173923 22180 leader_election.cc:343] T 563d05e06b1f48b4be140219da5c9dbd P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dd32a35972b347b89ceaaad2455aef5a (127.18.100.66:39781): Not found: Tablet not found: 563d05e06b1f48b4be140219da5c9dbd
W20260130 03:02:01.174324 22179 leader_election.cc:343] T 563d05e06b1f48b4be140219da5c9dbd P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 789dbb4a27204ffbb68d7ae40f8f5e4a (127.18.100.67:41825): Not found: Tablet not found: 563d05e06b1f48b4be140219da5c9dbd
I20260130 03:02:01.174634 22179 leader_election.cc:304] T 563d05e06b1f48b4be140219da5c9dbd P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 111215cc989f4203a6e1517f2291d66f; no voters: 789dbb4a27204ffbb68d7ae40f8f5e4a, dd32a35972b347b89ceaaad2455aef5a
I20260130 03:02:01.175143 25399 raft_consensus.cc:2749] T 563d05e06b1f48b4be140219da5c9dbd P 111215cc989f4203a6e1517f2291d66f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260130 03:02:01.256057 25399 raft_consensus.cc:493] T 5cc07f1d50624e03b1f5498b12ea680b P 111215cc989f4203a6e1517f2291d66f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 03:02:01.256492 25399 raft_consensus.cc:515] T 5cc07f1d50624e03b1f5498b12ea680b P 111215cc989f4203a6e1517f2291d66f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111215cc989f4203a6e1517f2291d66f" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 39751 } } peers { permanent_uuid: "dd32a35972b347b89ceaaad2455aef5a" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 39781 } } peers { permanent_uuid: "789dbb4a27204ffbb68d7ae40f8f5e4a" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 41825 } }
I20260130 03:02:01.258051 25399 leader_election.cc:290] T 5cc07f1d50624e03b1f5498b12ea680b P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dd32a35972b347b89ceaaad2455aef5a (127.18.100.66:39781), 789dbb4a27204ffbb68d7ae40f8f5e4a (127.18.100.67:41825)
W20260130 03:02:01.259620 22180 leader_election.cc:343] T 5cc07f1d50624e03b1f5498b12ea680b P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dd32a35972b347b89ceaaad2455aef5a (127.18.100.66:39781): Not found: Tablet not found: 5cc07f1d50624e03b1f5498b12ea680b
W20260130 03:02:01.260270 22179 leader_election.cc:343] T 5cc07f1d50624e03b1f5498b12ea680b P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 789dbb4a27204ffbb68d7ae40f8f5e4a (127.18.100.67:41825): Not found: Tablet not found: 5cc07f1d50624e03b1f5498b12ea680b
I20260130 03:02:01.260573 22179 leader_election.cc:304] T 5cc07f1d50624e03b1f5498b12ea680b P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 111215cc989f4203a6e1517f2291d66f; no voters: 789dbb4a27204ffbb68d7ae40f8f5e4a, dd32a35972b347b89ceaaad2455aef5a
I20260130 03:02:01.261165 25399 raft_consensus.cc:2749] T 5cc07f1d50624e03b1f5498b12ea680b P 111215cc989f4203a6e1517f2291d66f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260130 03:02:01.290107 23020 debug-util.cc:398] Leaking SignalData structure 0x7b08000b17e0 after lost signal to thread 22933
W20260130 03:02:01.290735 23020 debug-util.cc:398] Leaking SignalData structure 0x7b0800075c20 after lost signal to thread 23023
I20260130 03:02:03.162870 22923 master_runner.cc:186] Error getting master registration for 127.18.100.125:39157: Network error: Client connection negotiation failed: client connection to 127.18.100.125:39157: connect: Connection refused (error 111),
I20260130 03:02:03.163396 22923 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260130 03:02:05.253283 22179 consensus_peers.cc:597] T 476f8fd240bc48f1a9c47a0351ad5d27 P 111215cc989f4203a6e1517f2291d66f -> Peer 789dbb4a27204ffbb68d7ae40f8f5e4a (127.18.100.67:41825): Couldn't send request to peer 789dbb4a27204ffbb68d7ae40f8f5e4a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 476f8fd240bc48f1a9c47a0351ad5d27. This is attempt 86: this message will repeat every 5th retry.
I20260130 03:02:05.504637 25464 raft_consensus.cc:493] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 342be83f20a648fd849c7d23454fba08)
I20260130 03:02:05.505410 25464 raft_consensus.cc:515] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22dedcf69e00438999d3a5fedde57324" member_type: VOTER last_known_addr { host: "127.18.100.126" port: 42077 } } peers { permanent_uuid: "b01057b6c5ff4fc4b07253801881fe13" member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39157 } } peers { permanent_uuid: "342be83f20a648fd849c7d23454fba08" member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34987 } }
I20260130 03:02:05.507738 25464 leader_election.cc:290] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers b01057b6c5ff4fc4b07253801881fe13 (127.18.100.125:39157), 342be83f20a648fd849c7d23454fba08 (127.18.100.124:34987)
W20260130 03:02:05.510197 22970 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.18.100.125:39157: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20260130 03:02:05.513216 22968 leader_election.cc:336] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 342be83f20a648fd849c7d23454fba08 (127.18.100.124:34987): Network error: Client connection negotiation failed: client connection to 127.18.100.124:34987: connect: Connection refused (error 111)
W20260130 03:02:05.513664 22970 leader_election.cc:336] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer b01057b6c5ff4fc4b07253801881fe13 (127.18.100.125:39157): Network error: Client connection negotiation failed: client connection to 127.18.100.125:39157: connect: Connection refused (error 111)
I20260130 03:02:05.513965 22970 leader_election.cc:304] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22dedcf69e00438999d3a5fedde57324; no voters: 342be83f20a648fd849c7d23454fba08, b01057b6c5ff4fc4b07253801881fe13
I20260130 03:02:05.514593 25464 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260130 03:02:06.771142 25491 raft_consensus.cc:493] T 563d05e06b1f48b4be140219da5c9dbd P 111215cc989f4203a6e1517f2291d66f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 03:02:06.771708 25491 raft_consensus.cc:515] T 563d05e06b1f48b4be140219da5c9dbd P 111215cc989f4203a6e1517f2291d66f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111215cc989f4203a6e1517f2291d66f" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 39751 } } peers { permanent_uuid: "dd32a35972b347b89ceaaad2455aef5a" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 39781 } } peers { permanent_uuid: "789dbb4a27204ffbb68d7ae40f8f5e4a" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 41825 } }
I20260130 03:02:06.773612 25491 leader_election.cc:290] T 563d05e06b1f48b4be140219da5c9dbd P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dd32a35972b347b89ceaaad2455aef5a (127.18.100.66:39781), 789dbb4a27204ffbb68d7ae40f8f5e4a (127.18.100.67:41825)
W20260130 03:02:06.775527 22179 leader_election.cc:343] T 563d05e06b1f48b4be140219da5c9dbd P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 789dbb4a27204ffbb68d7ae40f8f5e4a (127.18.100.67:41825): Not found: Tablet not found: 563d05e06b1f48b4be140219da5c9dbd
W20260130 03:02:06.776124 22180 leader_election.cc:343] T 563d05e06b1f48b4be140219da5c9dbd P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dd32a35972b347b89ceaaad2455aef5a (127.18.100.66:39781): Not found: Tablet not found: 563d05e06b1f48b4be140219da5c9dbd
I20260130 03:02:06.776429 22180 leader_election.cc:304] T 563d05e06b1f48b4be140219da5c9dbd P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 111215cc989f4203a6e1517f2291d66f; no voters: 789dbb4a27204ffbb68d7ae40f8f5e4a, dd32a35972b347b89ceaaad2455aef5a
I20260130 03:02:06.776990 25491 raft_consensus.cc:2749] T 563d05e06b1f48b4be140219da5c9dbd P 111215cc989f4203a6e1517f2291d66f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260130 03:02:08.169816 22923 master_runner.cc:186] Error getting master registration for 127.18.100.125:39157: Network error: Client connection negotiation failed: client connection to 127.18.100.125:39157: connect: Connection refused (error 111),
I20260130 03:02:08.170477 22923 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260130 03:02:09.314028 22321 consensus_peers.cc:597] T ce91a17790324cd884a0521ce67dde67 P dd32a35972b347b89ceaaad2455aef5a -> Peer 789dbb4a27204ffbb68d7ae40f8f5e4a (127.18.100.67:41825): Couldn't send request to peer 789dbb4a27204ffbb68d7ae40f8f5e4a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ce91a17790324cd884a0521ce67dde67. This is attempt 86: this message will repeat every 5th retry.
W20260130 03:02:10.315114 22580 heartbeater.cc:646] Failed to heartbeat to 127.18.100.125:39157 (355 consecutive failures): Network error: Failed to ping master at 127.18.100.125:39157: Client connection negotiation failed: client connection to 127.18.100.125:39157: connect: Connection refused (error 111)
W20260130 03:02:10.330215 22437 heartbeater.cc:646] Failed to heartbeat to 127.18.100.125:39157 (439 consecutive failures): Network error: Failed to ping master at 127.18.100.125:39157: Client connection negotiation failed: client connection to 127.18.100.125:39157: connect: Connection refused (error 111)
W20260130 03:02:10.675567 22293 heartbeater.cc:646] Failed to heartbeat to 127.18.100.125:39157 (443 consecutive failures): Network error: Failed to ping master at 127.18.100.125:39157: Client connection negotiation failed: client connection to 127.18.100.125:39157: connect: Connection refused (error 111)
I20260130 03:02:13.176697 22923 master_runner.cc:186] Error getting master registration for 127.18.100.125:39157: Network error: Client connection negotiation failed: client connection to 127.18.100.125:39157: connect: Connection refused (error 111),
I20260130 03:02:13.177073 22923 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260130 03:02:14.062163 25605 raft_consensus.cc:493] T be72107f1e024d5788f5d2459c85015c P 789dbb4a27204ffbb68d7ae40f8f5e4a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 03:02:14.062884 25605 raft_consensus.cc:515] T be72107f1e024d5788f5d2459c85015c P 789dbb4a27204ffbb68d7ae40f8f5e4a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "789dbb4a27204ffbb68d7ae40f8f5e4a" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 41825 } } peers { permanent_uuid: "dd32a35972b347b89ceaaad2455aef5a" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 39781 } } peers { permanent_uuid: "111215cc989f4203a6e1517f2291d66f" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 39751 } }
I20260130 03:02:14.065173 25605 leader_election.cc:290] T be72107f1e024d5788f5d2459c85015c P 789dbb4a27204ffbb68d7ae40f8f5e4a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dd32a35972b347b89ceaaad2455aef5a (127.18.100.66:39781), 111215cc989f4203a6e1517f2291d66f (127.18.100.65:39751)
W20260130 03:02:14.067251 22469 leader_election.cc:343] T be72107f1e024d5788f5d2459c85015c P 789dbb4a27204ffbb68d7ae40f8f5e4a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dd32a35972b347b89ceaaad2455aef5a (127.18.100.66:39781): Not found: Tablet not found: be72107f1e024d5788f5d2459c85015c
W20260130 03:02:14.068003 22468 leader_election.cc:343] T be72107f1e024d5788f5d2459c85015c P 789dbb4a27204ffbb68d7ae40f8f5e4a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 111215cc989f4203a6e1517f2291d66f (127.18.100.65:39751): Not found: Tablet not found: be72107f1e024d5788f5d2459c85015c
I20260130 03:02:14.068375 22468 leader_election.cc:304] T be72107f1e024d5788f5d2459c85015c P 789dbb4a27204ffbb68d7ae40f8f5e4a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 789dbb4a27204ffbb68d7ae40f8f5e4a; no voters: 111215cc989f4203a6e1517f2291d66f, dd32a35972b347b89ceaaad2455aef5a
I20260130 03:02:14.069061 25605 raft_consensus.cc:2749] T be72107f1e024d5788f5d2459c85015c P 789dbb4a27204ffbb68d7ae40f8f5e4a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260130 03:02:15.363627 22179 consensus_peers.cc:597] T 476f8fd240bc48f1a9c47a0351ad5d27 P 111215cc989f4203a6e1517f2291d66f -> Peer 789dbb4a27204ffbb68d7ae40f8f5e4a (127.18.100.67:41825): Couldn't send request to peer 789dbb4a27204ffbb68d7ae40f8f5e4a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 476f8fd240bc48f1a9c47a0351ad5d27. This is attempt 91: this message will repeat every 5th retry.
I20260130 03:02:15.750530 25631 raft_consensus.cc:493] T 563d05e06b1f48b4be140219da5c9dbd P 111215cc989f4203a6e1517f2291d66f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 03:02:15.751299 25631 raft_consensus.cc:515] T 563d05e06b1f48b4be140219da5c9dbd P 111215cc989f4203a6e1517f2291d66f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111215cc989f4203a6e1517f2291d66f" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 39751 } } peers { permanent_uuid: "dd32a35972b347b89ceaaad2455aef5a" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 39781 } } peers { permanent_uuid: "789dbb4a27204ffbb68d7ae40f8f5e4a" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 41825 } }
I20260130 03:02:15.753158 25631 leader_election.cc:290] T 563d05e06b1f48b4be140219da5c9dbd P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dd32a35972b347b89ceaaad2455aef5a (127.18.100.66:39781), 789dbb4a27204ffbb68d7ae40f8f5e4a (127.18.100.67:41825)
W20260130 03:02:15.755031 22180 leader_election.cc:343] T 563d05e06b1f48b4be140219da5c9dbd P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dd32a35972b347b89ceaaad2455aef5a (127.18.100.66:39781): Not found: Tablet not found: 563d05e06b1f48b4be140219da5c9dbd
W20260130 03:02:15.755404 22179 leader_election.cc:343] T 563d05e06b1f48b4be140219da5c9dbd P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 789dbb4a27204ffbb68d7ae40f8f5e4a (127.18.100.67:41825): Not found: Tablet not found: 563d05e06b1f48b4be140219da5c9dbd
I20260130 03:02:15.755772 22179 leader_election.cc:304] T 563d05e06b1f48b4be140219da5c9dbd P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 111215cc989f4203a6e1517f2291d66f; no voters: 789dbb4a27204ffbb68d7ae40f8f5e4a, dd32a35972b347b89ceaaad2455aef5a
I20260130 03:02:15.756384 25631 raft_consensus.cc:2749] T 563d05e06b1f48b4be140219da5c9dbd P 111215cc989f4203a6e1517f2291d66f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260130 03:02:17.350834 25657 raft_consensus.cc:493] T 5cc07f1d50624e03b1f5498b12ea680b P 111215cc989f4203a6e1517f2291d66f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 03:02:17.351919 25657 raft_consensus.cc:515] T 5cc07f1d50624e03b1f5498b12ea680b P 111215cc989f4203a6e1517f2291d66f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111215cc989f4203a6e1517f2291d66f" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 39751 } } peers { permanent_uuid: "dd32a35972b347b89ceaaad2455aef5a" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 39781 } } peers { permanent_uuid: "789dbb4a27204ffbb68d7ae40f8f5e4a" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 41825 } }
I20260130 03:02:17.354255 25657 leader_election.cc:290] T 5cc07f1d50624e03b1f5498b12ea680b P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dd32a35972b347b89ceaaad2455aef5a (127.18.100.66:39781), 789dbb4a27204ffbb68d7ae40f8f5e4a (127.18.100.67:41825)
W20260130 03:02:17.355540 22180 leader_election.cc:343] T 5cc07f1d50624e03b1f5498b12ea680b P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dd32a35972b347b89ceaaad2455aef5a (127.18.100.66:39781): Not found: Tablet not found: 5cc07f1d50624e03b1f5498b12ea680b
W20260130 03:02:17.355983 22179 leader_election.cc:343] T 5cc07f1d50624e03b1f5498b12ea680b P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 789dbb4a27204ffbb68d7ae40f8f5e4a (127.18.100.67:41825): Not found: Tablet not found: 5cc07f1d50624e03b1f5498b12ea680b
I20260130 03:02:17.356344 22179 leader_election.cc:304] T 5cc07f1d50624e03b1f5498b12ea680b P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 111215cc989f4203a6e1517f2291d66f; no voters: 789dbb4a27204ffbb68d7ae40f8f5e4a, dd32a35972b347b89ceaaad2455aef5a
I20260130 03:02:17.356851 25657 raft_consensus.cc:2749] T 5cc07f1d50624e03b1f5498b12ea680b P 111215cc989f4203a6e1517f2291d66f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260130 03:02:18.183652 22923 master_runner.cc:186] Error getting master registration for 127.18.100.125:39157: Network error: Client connection negotiation failed: client connection to 127.18.100.125:39157: connect: Connection refused (error 111),
I20260130 03:02:18.184191 22923 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260130 03:02:18.588657 22321 consensus_peers.cc:597] T ce91a17790324cd884a0521ce67dde67 P dd32a35972b347b89ceaaad2455aef5a -> Peer 789dbb4a27204ffbb68d7ae40f8f5e4a (127.18.100.67:41825): Couldn't send request to peer 789dbb4a27204ffbb68d7ae40f8f5e4a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ce91a17790324cd884a0521ce67dde67. This is attempt 91: this message will repeat every 5th retry.
I20260130 03:02:22.267752 25736 raft_consensus.cc:493] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 342be83f20a648fd849c7d23454fba08)
I20260130 03:02:22.268352 25736 raft_consensus.cc:515] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22dedcf69e00438999d3a5fedde57324" member_type: VOTER last_known_addr { host: "127.18.100.126" port: 42077 } } peers { permanent_uuid: "b01057b6c5ff4fc4b07253801881fe13" member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39157 } } peers { permanent_uuid: "342be83f20a648fd849c7d23454fba08" member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34987 } }
I20260130 03:02:22.270664 25736 leader_election.cc:290] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers b01057b6c5ff4fc4b07253801881fe13 (127.18.100.125:39157), 342be83f20a648fd849c7d23454fba08 (127.18.100.124:34987)
W20260130 03:02:22.273022 22970 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.18.100.125:39157: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20260130 03:02:22.276314 22968 leader_election.cc:336] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 342be83f20a648fd849c7d23454fba08 (127.18.100.124:34987): Network error: Client connection negotiation failed: client connection to 127.18.100.124:34987: connect: Connection refused (error 111)
W20260130 03:02:22.276880 22970 leader_election.cc:336] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer b01057b6c5ff4fc4b07253801881fe13 (127.18.100.125:39157): Network error: Client connection negotiation failed: client connection to 127.18.100.125:39157: connect: Connection refused (error 111)
I20260130 03:02:22.277223 22970 leader_election.cc:304] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22dedcf69e00438999d3a5fedde57324; no voters: 342be83f20a648fd849c7d23454fba08, b01057b6c5ff4fc4b07253801881fe13
I20260130 03:02:22.278015 25736 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260130 03:02:23.190565 22923 master_runner.cc:186] Error getting master registration for 127.18.100.125:39157: Network error: Client connection negotiation failed: client connection to 127.18.100.125:39157: connect: Connection refused (error 111),
I20260130 03:02:23.190999 22923 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260130 03:02:25.126430 22179 consensus_peers.cc:597] T 476f8fd240bc48f1a9c47a0351ad5d27 P 111215cc989f4203a6e1517f2291d66f -> Peer 789dbb4a27204ffbb68d7ae40f8f5e4a (127.18.100.67:41825): Couldn't send request to peer 789dbb4a27204ffbb68d7ae40f8f5e4a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 476f8fd240bc48f1a9c47a0351ad5d27. This is attempt 96: this message will repeat every 5th retry.
I20260130 03:02:25.253198 25785 raft_consensus.cc:493] T be72107f1e024d5788f5d2459c85015c P 789dbb4a27204ffbb68d7ae40f8f5e4a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 03:02:25.253788 25785 raft_consensus.cc:515] T be72107f1e024d5788f5d2459c85015c P 789dbb4a27204ffbb68d7ae40f8f5e4a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "789dbb4a27204ffbb68d7ae40f8f5e4a" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 41825 } } peers { permanent_uuid: "dd32a35972b347b89ceaaad2455aef5a" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 39781 } } peers { permanent_uuid: "111215cc989f4203a6e1517f2291d66f" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 39751 } }
I20260130 03:02:25.255630 25785 leader_election.cc:290] T be72107f1e024d5788f5d2459c85015c P 789dbb4a27204ffbb68d7ae40f8f5e4a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dd32a35972b347b89ceaaad2455aef5a (127.18.100.66:39781), 111215cc989f4203a6e1517f2291d66f (127.18.100.65:39751)
W20260130 03:02:25.258029 22469 leader_election.cc:343] T be72107f1e024d5788f5d2459c85015c P 789dbb4a27204ffbb68d7ae40f8f5e4a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dd32a35972b347b89ceaaad2455aef5a (127.18.100.66:39781): Not found: Tablet not found: be72107f1e024d5788f5d2459c85015c
W20260130 03:02:25.258641 22468 leader_election.cc:343] T be72107f1e024d5788f5d2459c85015c P 789dbb4a27204ffbb68d7ae40f8f5e4a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 111215cc989f4203a6e1517f2291d66f (127.18.100.65:39751): Not found: Tablet not found: be72107f1e024d5788f5d2459c85015c
I20260130 03:02:25.258922 22468 leader_election.cc:304] T be72107f1e024d5788f5d2459c85015c P 789dbb4a27204ffbb68d7ae40f8f5e4a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 789dbb4a27204ffbb68d7ae40f8f5e4a; no voters: 111215cc989f4203a6e1517f2291d66f, dd32a35972b347b89ceaaad2455aef5a
I20260130 03:02:25.259586 25785 raft_consensus.cc:2749] T be72107f1e024d5788f5d2459c85015c P 789dbb4a27204ffbb68d7ae40f8f5e4a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260130 03:02:27.785977 25816 raft_consensus.cc:493] T 5cc07f1d50624e03b1f5498b12ea680b P 111215cc989f4203a6e1517f2291d66f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 03:02:27.786619 25816 raft_consensus.cc:515] T 5cc07f1d50624e03b1f5498b12ea680b P 111215cc989f4203a6e1517f2291d66f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111215cc989f4203a6e1517f2291d66f" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 39751 } } peers { permanent_uuid: "dd32a35972b347b89ceaaad2455aef5a" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 39781 } } peers { permanent_uuid: "789dbb4a27204ffbb68d7ae40f8f5e4a" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 41825 } }
I20260130 03:02:27.788450 25816 leader_election.cc:290] T 5cc07f1d50624e03b1f5498b12ea680b P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dd32a35972b347b89ceaaad2455aef5a (127.18.100.66:39781), 789dbb4a27204ffbb68d7ae40f8f5e4a (127.18.100.67:41825)
W20260130 03:02:27.790369 22180 leader_election.cc:343] T 5cc07f1d50624e03b1f5498b12ea680b P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dd32a35972b347b89ceaaad2455aef5a (127.18.100.66:39781): Not found: Tablet not found: 5cc07f1d50624e03b1f5498b12ea680b
W20260130 03:02:27.790866 22179 leader_election.cc:343] T 5cc07f1d50624e03b1f5498b12ea680b P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 789dbb4a27204ffbb68d7ae40f8f5e4a (127.18.100.67:41825): Not found: Tablet not found: 5cc07f1d50624e03b1f5498b12ea680b
I20260130 03:02:27.791278 22179 leader_election.cc:304] T 5cc07f1d50624e03b1f5498b12ea680b P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 111215cc989f4203a6e1517f2291d66f; no voters: 789dbb4a27204ffbb68d7ae40f8f5e4a, dd32a35972b347b89ceaaad2455aef5a
I20260130 03:02:27.791987 25816 raft_consensus.cc:2749] T 5cc07f1d50624e03b1f5498b12ea680b P 111215cc989f4203a6e1517f2291d66f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260130 03:02:27.845073 25825 raft_consensus.cc:493] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 342be83f20a648fd849c7d23454fba08)
I20260130 03:02:27.845695 25825 raft_consensus.cc:515] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22dedcf69e00438999d3a5fedde57324" member_type: VOTER last_known_addr { host: "127.18.100.126" port: 42077 } } peers { permanent_uuid: "b01057b6c5ff4fc4b07253801881fe13" member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39157 } } peers { permanent_uuid: "342be83f20a648fd849c7d23454fba08" member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34987 } }
I20260130 03:02:27.847560 25825 leader_election.cc:290] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers b01057b6c5ff4fc4b07253801881fe13 (127.18.100.125:39157), 342be83f20a648fd849c7d23454fba08 (127.18.100.124:34987)
W20260130 03:02:27.850512 22970 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.18.100.125:39157: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20260130 03:02:27.854066 22970 leader_election.cc:336] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer b01057b6c5ff4fc4b07253801881fe13 (127.18.100.125:39157): Network error: Client connection negotiation failed: client connection to 127.18.100.125:39157: connect: Connection refused (error 111)
W20260130 03:02:27.854645 22968 leader_election.cc:336] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 342be83f20a648fd849c7d23454fba08 (127.18.100.124:34987): Network error: Client connection negotiation failed: client connection to 127.18.100.124:34987: connect: Connection refused (error 111)
I20260130 03:02:27.855021 22968 leader_election.cc:304] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22dedcf69e00438999d3a5fedde57324; no voters: 342be83f20a648fd849c7d23454fba08, b01057b6c5ff4fc4b07253801881fe13
I20260130 03:02:27.855615 25825 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260130 03:02:28.196599 22923 master_runner.cc:186] Error getting master registration for 127.18.100.125:39157: Network error: Client connection negotiation failed: client connection to 127.18.100.125:39157: connect: Connection refused (error 111),
I20260130 03:02:28.197018 22923 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260130 03:02:28.731093 22321 consensus_peers.cc:597] T ce91a17790324cd884a0521ce67dde67 P dd32a35972b347b89ceaaad2455aef5a -> Peer 789dbb4a27204ffbb68d7ae40f8f5e4a (127.18.100.67:41825): Couldn't send request to peer 789dbb4a27204ffbb68d7ae40f8f5e4a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ce91a17790324cd884a0521ce67dde67. This is attempt 96: this message will repeat every 5th retry.
I20260130 03:02:33.202745 22923 master_runner.cc:186] Error getting master registration for 127.18.100.125:39157: Network error: Client connection negotiation failed: client connection to 127.18.100.125:39157: connect: Connection refused (error 111),
I20260130 03:02:33.203188 22923 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260130 03:02:33.242300 25913 raft_consensus.cc:493] T 563d05e06b1f48b4be140219da5c9dbd P 111215cc989f4203a6e1517f2291d66f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 03:02:33.243048 25913 raft_consensus.cc:515] T 563d05e06b1f48b4be140219da5c9dbd P 111215cc989f4203a6e1517f2291d66f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111215cc989f4203a6e1517f2291d66f" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 39751 } } peers { permanent_uuid: "dd32a35972b347b89ceaaad2455aef5a" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 39781 } } peers { permanent_uuid: "789dbb4a27204ffbb68d7ae40f8f5e4a" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 41825 } }
I20260130 03:02:33.245779 25913 leader_election.cc:290] T 563d05e06b1f48b4be140219da5c9dbd P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dd32a35972b347b89ceaaad2455aef5a (127.18.100.66:39781), 789dbb4a27204ffbb68d7ae40f8f5e4a (127.18.100.67:41825)
W20260130 03:02:33.247491 22180 leader_election.cc:343] T 563d05e06b1f48b4be140219da5c9dbd P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dd32a35972b347b89ceaaad2455aef5a (127.18.100.66:39781): Not found: Tablet not found: 563d05e06b1f48b4be140219da5c9dbd
W20260130 03:02:33.248046 22179 leader_election.cc:343] T 563d05e06b1f48b4be140219da5c9dbd P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 789dbb4a27204ffbb68d7ae40f8f5e4a (127.18.100.67:41825): Not found: Tablet not found: 563d05e06b1f48b4be140219da5c9dbd
I20260130 03:02:33.248304 22179 leader_election.cc:304] T 563d05e06b1f48b4be140219da5c9dbd P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 111215cc989f4203a6e1517f2291d66f; no voters: 789dbb4a27204ffbb68d7ae40f8f5e4a, dd32a35972b347b89ceaaad2455aef5a
I20260130 03:02:33.248898 25913 raft_consensus.cc:2749] T 563d05e06b1f48b4be140219da5c9dbd P 111215cc989f4203a6e1517f2291d66f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260130 03:02:33.279431 25916 raft_consensus.cc:493] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 342be83f20a648fd849c7d23454fba08)
I20260130 03:02:33.279928 25916 raft_consensus.cc:515] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22dedcf69e00438999d3a5fedde57324" member_type: VOTER last_known_addr { host: "127.18.100.126" port: 42077 } } peers { permanent_uuid: "b01057b6c5ff4fc4b07253801881fe13" member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39157 } } peers { permanent_uuid: "342be83f20a648fd849c7d23454fba08" member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34987 } }
I20260130 03:02:33.281718 25916 leader_election.cc:290] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers b01057b6c5ff4fc4b07253801881fe13 (127.18.100.125:39157), 342be83f20a648fd849c7d23454fba08 (127.18.100.124:34987)
W20260130 03:02:33.283324 22970 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.18.100.125:39157: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20260130 03:02:33.286562 22970 leader_election.cc:336] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer b01057b6c5ff4fc4b07253801881fe13 (127.18.100.125:39157): Network error: Client connection negotiation failed: client connection to 127.18.100.125:39157: connect: Connection refused (error 111)
W20260130 03:02:33.287025 22968 leader_election.cc:336] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 342be83f20a648fd849c7d23454fba08 (127.18.100.124:34987): Network error: Client connection negotiation failed: client connection to 127.18.100.124:34987: connect: Connection refused (error 111)
I20260130 03:02:33.287334 22968 leader_election.cc:304] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22dedcf69e00438999d3a5fedde57324; no voters: 342be83f20a648fd849c7d23454fba08, b01057b6c5ff4fc4b07253801881fe13
I20260130 03:02:33.287868 25916 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260130 03:02:34.928841 25942 raft_consensus.cc:493] T be72107f1e024d5788f5d2459c85015c P 789dbb4a27204ffbb68d7ae40f8f5e4a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 03:02:34.929541 25942 raft_consensus.cc:515] T be72107f1e024d5788f5d2459c85015c P 789dbb4a27204ffbb68d7ae40f8f5e4a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "789dbb4a27204ffbb68d7ae40f8f5e4a" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 41825 } } peers { permanent_uuid: "dd32a35972b347b89ceaaad2455aef5a" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 39781 } } peers { permanent_uuid: "111215cc989f4203a6e1517f2291d66f" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 39751 } }
I20260130 03:02:34.932049 25942 leader_election.cc:290] T be72107f1e024d5788f5d2459c85015c P 789dbb4a27204ffbb68d7ae40f8f5e4a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dd32a35972b347b89ceaaad2455aef5a (127.18.100.66:39781), 111215cc989f4203a6e1517f2291d66f (127.18.100.65:39751)
W20260130 03:02:34.933951 22469 leader_election.cc:343] T be72107f1e024d5788f5d2459c85015c P 789dbb4a27204ffbb68d7ae40f8f5e4a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dd32a35972b347b89ceaaad2455aef5a (127.18.100.66:39781): Not found: Tablet not found: be72107f1e024d5788f5d2459c85015c
W20260130 03:02:34.934556 22468 leader_election.cc:343] T be72107f1e024d5788f5d2459c85015c P 789dbb4a27204ffbb68d7ae40f8f5e4a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 111215cc989f4203a6e1517f2291d66f (127.18.100.65:39751): Not found: Tablet not found: be72107f1e024d5788f5d2459c85015c
I20260130 03:02:34.934870 22468 leader_election.cc:304] T be72107f1e024d5788f5d2459c85015c P 789dbb4a27204ffbb68d7ae40f8f5e4a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 789dbb4a27204ffbb68d7ae40f8f5e4a; no voters: 111215cc989f4203a6e1517f2291d66f, dd32a35972b347b89ceaaad2455aef5a
I20260130 03:02:34.935488 25942 raft_consensus.cc:2749] T be72107f1e024d5788f5d2459c85015c P 789dbb4a27204ffbb68d7ae40f8f5e4a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260130 03:02:35.216145 22179 consensus_peers.cc:597] T 476f8fd240bc48f1a9c47a0351ad5d27 P 111215cc989f4203a6e1517f2291d66f -> Peer 789dbb4a27204ffbb68d7ae40f8f5e4a (127.18.100.67:41825): Couldn't send request to peer 789dbb4a27204ffbb68d7ae40f8f5e4a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 476f8fd240bc48f1a9c47a0351ad5d27. This is attempt 101: this message will repeat every 5th retry.
I20260130 03:02:35.370615 25952 raft_consensus.cc:493] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 342be83f20a648fd849c7d23454fba08)
I20260130 03:02:35.371102 25952 raft_consensus.cc:515] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22dedcf69e00438999d3a5fedde57324" member_type: VOTER last_known_addr { host: "127.18.100.126" port: 42077 } } peers { permanent_uuid: "b01057b6c5ff4fc4b07253801881fe13" member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39157 } } peers { permanent_uuid: "342be83f20a648fd849c7d23454fba08" member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34987 } }
I20260130 03:02:35.372771 25952 leader_election.cc:290] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers b01057b6c5ff4fc4b07253801881fe13 (127.18.100.125:39157), 342be83f20a648fd849c7d23454fba08 (127.18.100.124:34987)
W20260130 03:02:35.377564 22970 leader_election.cc:336] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer b01057b6c5ff4fc4b07253801881fe13 (127.18.100.125:39157): Network error: Client connection negotiation failed: client connection to 127.18.100.125:39157: connect: Connection refused (error 111)
W20260130 03:02:35.378052 22968 leader_election.cc:336] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 342be83f20a648fd849c7d23454fba08 (127.18.100.124:34987): Network error: Client connection negotiation failed: client connection to 127.18.100.124:34987: connect: Connection refused (error 111)
I20260130 03:02:35.378415 22968 leader_election.cc:304] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22dedcf69e00438999d3a5fedde57324; no voters: 342be83f20a648fd849c7d23454fba08, b01057b6c5ff4fc4b07253801881fe13
I20260130 03:02:35.379103 25952 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260130 03:02:35.658519 22289 debug-util.cc:398] Leaking SignalData structure 0x7b08000b7160 after lost signal to thread 22155
W20260130 03:02:35.659997 22289 debug-util.cc:398] Leaking SignalData structure 0x7b08000cda80 after lost signal to thread 22292
I20260130 03:02:38.210095 22923 master_runner.cc:186] Error getting master registration for 127.18.100.125:39157: Network error: Client connection negotiation failed: client connection to 127.18.100.125:39157: connect: Connection refused (error 111),
I20260130 03:02:38.210673 22923 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260130 03:02:39.434343 22321 consensus_peers.cc:597] T ce91a17790324cd884a0521ce67dde67 P dd32a35972b347b89ceaaad2455aef5a -> Peer 789dbb4a27204ffbb68d7ae40f8f5e4a (127.18.100.67:41825): Couldn't send request to peer 789dbb4a27204ffbb68d7ae40f8f5e4a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ce91a17790324cd884a0521ce67dde67. This is attempt 101: this message will repeat every 5th retry.
I20260130 03:02:43.216488 22923 master_runner.cc:186] Error getting master registration for 127.18.100.125:39157: Network error: Client connection negotiation failed: client connection to 127.18.100.125:39157: connect: Connection refused (error 111),
I20260130 03:02:43.216920 22923 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260130 03:02:43.511322 26080 raft_consensus.cc:493] T be72107f1e024d5788f5d2459c85015c P 789dbb4a27204ffbb68d7ae40f8f5e4a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 03:02:43.511906 26080 raft_consensus.cc:515] T be72107f1e024d5788f5d2459c85015c P 789dbb4a27204ffbb68d7ae40f8f5e4a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "789dbb4a27204ffbb68d7ae40f8f5e4a" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 41825 } } peers { permanent_uuid: "dd32a35972b347b89ceaaad2455aef5a" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 39781 } } peers { permanent_uuid: "111215cc989f4203a6e1517f2291d66f" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 39751 } }
I20260130 03:02:43.513631 26080 leader_election.cc:290] T be72107f1e024d5788f5d2459c85015c P 789dbb4a27204ffbb68d7ae40f8f5e4a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dd32a35972b347b89ceaaad2455aef5a (127.18.100.66:39781), 111215cc989f4203a6e1517f2291d66f (127.18.100.65:39751)
W20260130 03:02:43.515247 22469 leader_election.cc:343] T be72107f1e024d5788f5d2459c85015c P 789dbb4a27204ffbb68d7ae40f8f5e4a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dd32a35972b347b89ceaaad2455aef5a (127.18.100.66:39781): Not found: Tablet not found: be72107f1e024d5788f5d2459c85015c
W20260130 03:02:43.515877 22468 leader_election.cc:343] T be72107f1e024d5788f5d2459c85015c P 789dbb4a27204ffbb68d7ae40f8f5e4a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 111215cc989f4203a6e1517f2291d66f (127.18.100.65:39751): Not found: Tablet not found: be72107f1e024d5788f5d2459c85015c
I20260130 03:02:43.516175 22468 leader_election.cc:304] T be72107f1e024d5788f5d2459c85015c P 789dbb4a27204ffbb68d7ae40f8f5e4a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 789dbb4a27204ffbb68d7ae40f8f5e4a; no voters: 111215cc989f4203a6e1517f2291d66f, dd32a35972b347b89ceaaad2455aef5a
I20260130 03:02:43.516817 26080 raft_consensus.cc:2749] T be72107f1e024d5788f5d2459c85015c P 789dbb4a27204ffbb68d7ae40f8f5e4a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260130 03:02:43.888084 26087 raft_consensus.cc:493] T 5cc07f1d50624e03b1f5498b12ea680b P 111215cc989f4203a6e1517f2291d66f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 03:02:43.888677 26087 raft_consensus.cc:515] T 5cc07f1d50624e03b1f5498b12ea680b P 111215cc989f4203a6e1517f2291d66f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111215cc989f4203a6e1517f2291d66f" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 39751 } } peers { permanent_uuid: "dd32a35972b347b89ceaaad2455aef5a" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 39781 } } peers { permanent_uuid: "789dbb4a27204ffbb68d7ae40f8f5e4a" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 41825 } }
I20260130 03:02:43.890594 26087 leader_election.cc:290] T 5cc07f1d50624e03b1f5498b12ea680b P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dd32a35972b347b89ceaaad2455aef5a (127.18.100.66:39781), 789dbb4a27204ffbb68d7ae40f8f5e4a (127.18.100.67:41825)
W20260130 03:02:43.892118 22180 leader_election.cc:343] T 5cc07f1d50624e03b1f5498b12ea680b P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dd32a35972b347b89ceaaad2455aef5a (127.18.100.66:39781): Not found: Tablet not found: 5cc07f1d50624e03b1f5498b12ea680b
W20260130 03:02:43.892645 22179 leader_election.cc:343] T 5cc07f1d50624e03b1f5498b12ea680b P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 789dbb4a27204ffbb68d7ae40f8f5e4a (127.18.100.67:41825): Not found: Tablet not found: 5cc07f1d50624e03b1f5498b12ea680b
I20260130 03:02:43.892915 22179 leader_election.cc:304] T 5cc07f1d50624e03b1f5498b12ea680b P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 111215cc989f4203a6e1517f2291d66f; no voters: 789dbb4a27204ffbb68d7ae40f8f5e4a, dd32a35972b347b89ceaaad2455aef5a
I20260130 03:02:43.893424 26087 raft_consensus.cc:2749] T 5cc07f1d50624e03b1f5498b12ea680b P 111215cc989f4203a6e1517f2291d66f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260130 03:02:44.274194 26091 raft_consensus.cc:493] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 342be83f20a648fd849c7d23454fba08)
I20260130 03:02:44.274772 26091 raft_consensus.cc:515] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22dedcf69e00438999d3a5fedde57324" member_type: VOTER last_known_addr { host: "127.18.100.126" port: 42077 } } peers { permanent_uuid: "b01057b6c5ff4fc4b07253801881fe13" member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39157 } } peers { permanent_uuid: "342be83f20a648fd849c7d23454fba08" member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34987 } }
I20260130 03:02:44.283965 26091 leader_election.cc:290] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers b01057b6c5ff4fc4b07253801881fe13 (127.18.100.125:39157), 342be83f20a648fd849c7d23454fba08 (127.18.100.124:34987)
W20260130 03:02:44.285714 22970 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.18.100.125:39157: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20260130 03:02:44.289657 22970 leader_election.cc:336] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer b01057b6c5ff4fc4b07253801881fe13 (127.18.100.125:39157): Network error: Client connection negotiation failed: client connection to 127.18.100.125:39157: connect: Connection refused (error 111)
W20260130 03:02:44.290630 22968 leader_election.cc:336] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 342be83f20a648fd849c7d23454fba08 (127.18.100.124:34987): Network error: Client connection negotiation failed: client connection to 127.18.100.124:34987: connect: Connection refused (error 111)
I20260130 03:02:44.290977 22968 leader_election.cc:304] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22dedcf69e00438999d3a5fedde57324; no voters: 342be83f20a648fd849c7d23454fba08, b01057b6c5ff4fc4b07253801881fe13
I20260130 03:02:44.291719 26091 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260130 03:02:44.627769 22576 debug-util.cc:398] Leaking SignalData structure 0x7b08000cafe0 after lost signal to thread 22448
W20260130 03:02:44.628926 22576 debug-util.cc:398] Leaking SignalData structure 0x7b08000dbe20 after lost signal to thread 22579
W20260130 03:02:44.831434 22179 consensus_peers.cc:597] T 476f8fd240bc48f1a9c47a0351ad5d27 P 111215cc989f4203a6e1517f2291d66f -> Peer 789dbb4a27204ffbb68d7ae40f8f5e4a (127.18.100.67:41825): Couldn't send request to peer 789dbb4a27204ffbb68d7ae40f8f5e4a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 476f8fd240bc48f1a9c47a0351ad5d27. This is attempt 106: this message will repeat every 5th retry.
I20260130 03:02:46.722340 26130 raft_consensus.cc:493] T 563d05e06b1f48b4be140219da5c9dbd P 111215cc989f4203a6e1517f2291d66f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 03:02:46.722919 26130 raft_consensus.cc:515] T 563d05e06b1f48b4be140219da5c9dbd P 111215cc989f4203a6e1517f2291d66f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111215cc989f4203a6e1517f2291d66f" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 39751 } } peers { permanent_uuid: "dd32a35972b347b89ceaaad2455aef5a" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 39781 } } peers { permanent_uuid: "789dbb4a27204ffbb68d7ae40f8f5e4a" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 41825 } }
I20260130 03:02:46.724637 26130 leader_election.cc:290] T 563d05e06b1f48b4be140219da5c9dbd P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dd32a35972b347b89ceaaad2455aef5a (127.18.100.66:39781), 789dbb4a27204ffbb68d7ae40f8f5e4a (127.18.100.67:41825)
W20260130 03:02:46.726362 22180 leader_election.cc:343] T 563d05e06b1f48b4be140219da5c9dbd P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dd32a35972b347b89ceaaad2455aef5a (127.18.100.66:39781): Not found: Tablet not found: 563d05e06b1f48b4be140219da5c9dbd
W20260130 03:02:46.726768 22179 leader_election.cc:343] T 563d05e06b1f48b4be140219da5c9dbd P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 789dbb4a27204ffbb68d7ae40f8f5e4a (127.18.100.67:41825): Not found: Tablet not found: 563d05e06b1f48b4be140219da5c9dbd
I20260130 03:02:46.727054 22179 leader_election.cc:304] T 563d05e06b1f48b4be140219da5c9dbd P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 111215cc989f4203a6e1517f2291d66f; no voters: 789dbb4a27204ffbb68d7ae40f8f5e4a, dd32a35972b347b89ceaaad2455aef5a
I20260130 03:02:46.727578 26130 raft_consensus.cc:2749] T 563d05e06b1f48b4be140219da5c9dbd P 111215cc989f4203a6e1517f2291d66f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260130 03:02:48.224200 22923 master_runner.cc:186] Error getting master registration for 127.18.100.125:39157: Network error: Client connection negotiation failed: client connection to 127.18.100.125:39157: connect: Connection refused (error 111),
I20260130 03:02:48.224730 22923 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260130 03:02:49.774955 22321 consensus_peers.cc:597] T ce91a17790324cd884a0521ce67dde67 P dd32a35972b347b89ceaaad2455aef5a -> Peer 789dbb4a27204ffbb68d7ae40f8f5e4a (127.18.100.67:41825): Couldn't send request to peer 789dbb4a27204ffbb68d7ae40f8f5e4a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ce91a17790324cd884a0521ce67dde67. This is attempt 106: this message will repeat every 5th retry.
I20260130 03:02:53.230383 22923 master_runner.cc:186] Error getting master registration for 127.18.100.125:39157: Network error: Client connection negotiation failed: client connection to 127.18.100.125:39157: connect: Connection refused (error 111),
I20260130 03:02:53.230854 22923 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260130 03:02:54.318799 22179 consensus_peers.cc:597] T 476f8fd240bc48f1a9c47a0351ad5d27 P 111215cc989f4203a6e1517f2291d66f -> Peer 789dbb4a27204ffbb68d7ae40f8f5e4a (127.18.100.67:41825): Couldn't send request to peer 789dbb4a27204ffbb68d7ae40f8f5e4a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 476f8fd240bc48f1a9c47a0351ad5d27. This is attempt 111: this message will repeat every 5th retry.
I20260130 03:02:55.060509 26269 raft_consensus.cc:493] T be72107f1e024d5788f5d2459c85015c P 789dbb4a27204ffbb68d7ae40f8f5e4a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 03:02:55.061105 26269 raft_consensus.cc:515] T be72107f1e024d5788f5d2459c85015c P 789dbb4a27204ffbb68d7ae40f8f5e4a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "789dbb4a27204ffbb68d7ae40f8f5e4a" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 41825 } } peers { permanent_uuid: "dd32a35972b347b89ceaaad2455aef5a" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 39781 } } peers { permanent_uuid: "111215cc989f4203a6e1517f2291d66f" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 39751 } }
I20260130 03:02:55.063330 26269 leader_election.cc:290] T be72107f1e024d5788f5d2459c85015c P 789dbb4a27204ffbb68d7ae40f8f5e4a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dd32a35972b347b89ceaaad2455aef5a (127.18.100.66:39781), 111215cc989f4203a6e1517f2291d66f (127.18.100.65:39751)
W20260130 03:02:55.065311 22469 leader_election.cc:343] T be72107f1e024d5788f5d2459c85015c P 789dbb4a27204ffbb68d7ae40f8f5e4a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dd32a35972b347b89ceaaad2455aef5a (127.18.100.66:39781): Not found: Tablet not found: be72107f1e024d5788f5d2459c85015c
W20260130 03:02:55.066150 22468 leader_election.cc:343] T be72107f1e024d5788f5d2459c85015c P 789dbb4a27204ffbb68d7ae40f8f5e4a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 111215cc989f4203a6e1517f2291d66f (127.18.100.65:39751): Not found: Tablet not found: be72107f1e024d5788f5d2459c85015c
I20260130 03:02:55.066488 22468 leader_election.cc:304] T be72107f1e024d5788f5d2459c85015c P 789dbb4a27204ffbb68d7ae40f8f5e4a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 789dbb4a27204ffbb68d7ae40f8f5e4a; no voters: 111215cc989f4203a6e1517f2291d66f, dd32a35972b347b89ceaaad2455aef5a
I20260130 03:02:55.067122 26269 raft_consensus.cc:2749] T be72107f1e024d5788f5d2459c85015c P 789dbb4a27204ffbb68d7ae40f8f5e4a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260130 03:02:58.237270 22923 master_runner.cc:186] Error getting master registration for 127.18.100.125:39157: Network error: Client connection negotiation failed: client connection to 127.18.100.125:39157: connect: Connection refused (error 111),
I20260130 03:02:58.237803 22923 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260130 03:02:59.381415 22321 consensus_peers.cc:597] T ce91a17790324cd884a0521ce67dde67 P dd32a35972b347b89ceaaad2455aef5a -> Peer 789dbb4a27204ffbb68d7ae40f8f5e4a (127.18.100.67:41825): Couldn't send request to peer 789dbb4a27204ffbb68d7ae40f8f5e4a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ce91a17790324cd884a0521ce67dde67. This is attempt 111: this message will repeat every 5th retry.
I20260130 03:03:01.155646 26368 raft_consensus.cc:493] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 342be83f20a648fd849c7d23454fba08)
I20260130 03:03:01.156358 26368 raft_consensus.cc:515] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22dedcf69e00438999d3a5fedde57324" member_type: VOTER last_known_addr { host: "127.18.100.126" port: 42077 } } peers { permanent_uuid: "b01057b6c5ff4fc4b07253801881fe13" member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39157 } } peers { permanent_uuid: "342be83f20a648fd849c7d23454fba08" member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34987 } }
I20260130 03:03:01.159952 26368 leader_election.cc:290] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers b01057b6c5ff4fc4b07253801881fe13 (127.18.100.125:39157), 342be83f20a648fd849c7d23454fba08 (127.18.100.124:34987)
W20260130 03:03:01.161484 22970 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.18.100.125:39157: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20260130 03:03:01.165092 22968 leader_election.cc:336] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 342be83f20a648fd849c7d23454fba08 (127.18.100.124:34987): Network error: Client connection negotiation failed: client connection to 127.18.100.124:34987: connect: Connection refused (error 111)
W20260130 03:03:01.165556 22970 leader_election.cc:336] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer b01057b6c5ff4fc4b07253801881fe13 (127.18.100.125:39157): Network error: Client connection negotiation failed: client connection to 127.18.100.125:39157: connect: Connection refused (error 111)
I20260130 03:03:01.165939 22970 leader_election.cc:304] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22dedcf69e00438999d3a5fedde57324; no voters: 342be83f20a648fd849c7d23454fba08, b01057b6c5ff4fc4b07253801881fe13
I20260130 03:03:01.166533 26368 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260130 03:03:01.885237 26378 raft_consensus.cc:493] T 5cc07f1d50624e03b1f5498b12ea680b P 111215cc989f4203a6e1517f2291d66f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 03:03:01.885807 26378 raft_consensus.cc:515] T 5cc07f1d50624e03b1f5498b12ea680b P 111215cc989f4203a6e1517f2291d66f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111215cc989f4203a6e1517f2291d66f" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 39751 } } peers { permanent_uuid: "dd32a35972b347b89ceaaad2455aef5a" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 39781 } } peers { permanent_uuid: "789dbb4a27204ffbb68d7ae40f8f5e4a" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 41825 } }
I20260130 03:03:01.888031 26378 leader_election.cc:290] T 5cc07f1d50624e03b1f5498b12ea680b P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dd32a35972b347b89ceaaad2455aef5a (127.18.100.66:39781), 789dbb4a27204ffbb68d7ae40f8f5e4a (127.18.100.67:41825)
W20260130 03:03:01.889876 22180 leader_election.cc:343] T 5cc07f1d50624e03b1f5498b12ea680b P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dd32a35972b347b89ceaaad2455aef5a (127.18.100.66:39781): Not found: Tablet not found: 5cc07f1d50624e03b1f5498b12ea680b
W20260130 03:03:01.890287 22179 leader_election.cc:343] T 5cc07f1d50624e03b1f5498b12ea680b P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 789dbb4a27204ffbb68d7ae40f8f5e4a (127.18.100.67:41825): Not found: Tablet not found: 5cc07f1d50624e03b1f5498b12ea680b
I20260130 03:03:01.890604 22179 leader_election.cc:304] T 5cc07f1d50624e03b1f5498b12ea680b P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 111215cc989f4203a6e1517f2291d66f; no voters: 789dbb4a27204ffbb68d7ae40f8f5e4a, dd32a35972b347b89ceaaad2455aef5a
I20260130 03:03:01.891191 26378 raft_consensus.cc:2749] T 5cc07f1d50624e03b1f5498b12ea680b P 111215cc989f4203a6e1517f2291d66f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260130 03:03:03.243479 22923 master_runner.cc:186] Error getting master registration for 127.18.100.125:39157: Network error: Client connection negotiation failed: client connection to 127.18.100.125:39157: connect: Connection refused (error 111),
I20260130 03:03:03.243903 22923 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260130 03:03:03.956945 22179 consensus_peers.cc:597] T 476f8fd240bc48f1a9c47a0351ad5d27 P 111215cc989f4203a6e1517f2291d66f -> Peer 789dbb4a27204ffbb68d7ae40f8f5e4a (127.18.100.67:41825): Couldn't send request to peer 789dbb4a27204ffbb68d7ae40f8f5e4a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 476f8fd240bc48f1a9c47a0351ad5d27. This is attempt 116: this message will repeat every 5th retry.
I20260130 03:03:04.808895 26423 raft_consensus.cc:493] T 563d05e06b1f48b4be140219da5c9dbd P 111215cc989f4203a6e1517f2291d66f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 03:03:04.809509 26423 raft_consensus.cc:515] T 563d05e06b1f48b4be140219da5c9dbd P 111215cc989f4203a6e1517f2291d66f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111215cc989f4203a6e1517f2291d66f" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 39751 } } peers { permanent_uuid: "dd32a35972b347b89ceaaad2455aef5a" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 39781 } } peers { permanent_uuid: "789dbb4a27204ffbb68d7ae40f8f5e4a" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 41825 } }
I20260130 03:03:04.811657 26423 leader_election.cc:290] T 563d05e06b1f48b4be140219da5c9dbd P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dd32a35972b347b89ceaaad2455aef5a (127.18.100.66:39781), 789dbb4a27204ffbb68d7ae40f8f5e4a (127.18.100.67:41825)
W20260130 03:03:04.813585 22180 leader_election.cc:343] T 563d05e06b1f48b4be140219da5c9dbd P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dd32a35972b347b89ceaaad2455aef5a (127.18.100.66:39781): Not found: Tablet not found: 563d05e06b1f48b4be140219da5c9dbd
W20260130 03:03:04.814076 22179 leader_election.cc:343] T 563d05e06b1f48b4be140219da5c9dbd P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 789dbb4a27204ffbb68d7ae40f8f5e4a (127.18.100.67:41825): Not found: Tablet not found: 563d05e06b1f48b4be140219da5c9dbd
I20260130 03:03:04.814363 22179 leader_election.cc:304] T 563d05e06b1f48b4be140219da5c9dbd P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 111215cc989f4203a6e1517f2291d66f; no voters: 789dbb4a27204ffbb68d7ae40f8f5e4a, dd32a35972b347b89ceaaad2455aef5a
I20260130 03:03:04.815012 26423 raft_consensus.cc:2749] T 563d05e06b1f48b4be140219da5c9dbd P 111215cc989f4203a6e1517f2291d66f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260130 03:03:07.122659 23020 debug-util.cc:398] Leaking SignalData structure 0x7b080006f160 after lost signal to thread 22933
W20260130 03:03:07.123701 23020 debug-util.cc:398] Leaking SignalData structure 0x7b0800074f80 after lost signal to thread 23023
I20260130 03:03:08.116657 26476 raft_consensus.cc:493] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 342be83f20a648fd849c7d23454fba08)
I20260130 03:03:08.117249 26476 raft_consensus.cc:515] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22dedcf69e00438999d3a5fedde57324" member_type: VOTER last_known_addr { host: "127.18.100.126" port: 42077 } } peers { permanent_uuid: "b01057b6c5ff4fc4b07253801881fe13" member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39157 } } peers { permanent_uuid: "342be83f20a648fd849c7d23454fba08" member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34987 } }
I20260130 03:03:08.119246 26476 leader_election.cc:290] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers b01057b6c5ff4fc4b07253801881fe13 (127.18.100.125:39157), 342be83f20a648fd849c7d23454fba08 (127.18.100.124:34987)
W20260130 03:03:08.122054 22968 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.18.100.124:34987: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20260130 03:03:08.125332 22970 leader_election.cc:336] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer b01057b6c5ff4fc4b07253801881fe13 (127.18.100.125:39157): Network error: Client connection negotiation failed: client connection to 127.18.100.125:39157: connect: Connection refused (error 111)
W20260130 03:03:08.125798 22968 leader_election.cc:336] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 342be83f20a648fd849c7d23454fba08 (127.18.100.124:34987): Network error: Client connection negotiation failed: client connection to 127.18.100.124:34987: connect: Connection refused (error 111)
I20260130 03:03:08.126169 22968 leader_election.cc:304] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22dedcf69e00438999d3a5fedde57324; no voters: 342be83f20a648fd849c7d23454fba08, b01057b6c5ff4fc4b07253801881fe13
I20260130 03:03:08.126837 26476 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260130 03:03:08.298643 22923 master_runner.cc:186] Error getting master registration for 127.18.100.125:39157: Network error: Client connection negotiation failed: client connection to 127.18.100.125:39157: connect: Connection refused (error 111),
I20260130 03:03:08.299048 22923 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260130 03:03:09.386896 22321 consensus_peers.cc:597] T ce91a17790324cd884a0521ce67dde67 P dd32a35972b347b89ceaaad2455aef5a -> Peer 789dbb4a27204ffbb68d7ae40f8f5e4a (127.18.100.67:41825): Couldn't send request to peer 789dbb4a27204ffbb68d7ae40f8f5e4a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ce91a17790324cd884a0521ce67dde67. This is attempt 116: this message will repeat every 5th retry.
I20260130 03:03:10.505060 26520 raft_consensus.cc:493] T be72107f1e024d5788f5d2459c85015c P 789dbb4a27204ffbb68d7ae40f8f5e4a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 03:03:10.505738 26520 raft_consensus.cc:515] T be72107f1e024d5788f5d2459c85015c P 789dbb4a27204ffbb68d7ae40f8f5e4a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "789dbb4a27204ffbb68d7ae40f8f5e4a" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 41825 } } peers { permanent_uuid: "dd32a35972b347b89ceaaad2455aef5a" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 39781 } } peers { permanent_uuid: "111215cc989f4203a6e1517f2291d66f" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 39751 } }
I20260130 03:03:10.508241 26520 leader_election.cc:290] T be72107f1e024d5788f5d2459c85015c P 789dbb4a27204ffbb68d7ae40f8f5e4a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dd32a35972b347b89ceaaad2455aef5a (127.18.100.66:39781), 111215cc989f4203a6e1517f2291d66f (127.18.100.65:39751)
W20260130 03:03:10.509953 22469 leader_election.cc:343] T be72107f1e024d5788f5d2459c85015c P 789dbb4a27204ffbb68d7ae40f8f5e4a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dd32a35972b347b89ceaaad2455aef5a (127.18.100.66:39781): Not found: Tablet not found: be72107f1e024d5788f5d2459c85015c
W20260130 03:03:10.510530 22468 leader_election.cc:343] T be72107f1e024d5788f5d2459c85015c P 789dbb4a27204ffbb68d7ae40f8f5e4a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 111215cc989f4203a6e1517f2291d66f (127.18.100.65:39751): Not found: Tablet not found: be72107f1e024d5788f5d2459c85015c
I20260130 03:03:10.510814 22468 leader_election.cc:304] T be72107f1e024d5788f5d2459c85015c P 789dbb4a27204ffbb68d7ae40f8f5e4a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 789dbb4a27204ffbb68d7ae40f8f5e4a; no voters: 111215cc989f4203a6e1517f2291d66f, dd32a35972b347b89ceaaad2455aef5a
I20260130 03:03:10.511359 26520 raft_consensus.cc:2749] T be72107f1e024d5788f5d2459c85015c P 789dbb4a27204ffbb68d7ae40f8f5e4a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260130 03:03:10.587908 22580 heartbeater.cc:646] Failed to heartbeat to 127.18.100.125:39157 (475 consecutive failures): Network error: Failed to ping master at 127.18.100.125:39157: Client connection negotiation failed: client connection to 127.18.100.125:39157: connect: Connection refused (error 111)
W20260130 03:03:10.709035 22438 heartbeater.cc:646] Failed to heartbeat to 127.18.100.124:34987 (574 consecutive failures): Network error: Failed to ping master at 127.18.100.124:34987: Client connection negotiation failed: client connection to 127.18.100.124:34987: connect: Connection refused (error 111)
W20260130 03:03:10.890151 22294 heartbeater.cc:646] Failed to heartbeat to 127.18.100.124:34987 (578 consecutive failures): Network error: Failed to ping master at 127.18.100.124:34987: Client connection negotiation failed: client connection to 127.18.100.124:34987: connect: Connection refused (error 111)
I20260130 03:03:11.660086 26538 raft_consensus.cc:493] T 5cc07f1d50624e03b1f5498b12ea680b P 111215cc989f4203a6e1517f2291d66f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 03:03:11.660557 26538 raft_consensus.cc:515] T 5cc07f1d50624e03b1f5498b12ea680b P 111215cc989f4203a6e1517f2291d66f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111215cc989f4203a6e1517f2291d66f" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 39751 } } peers { permanent_uuid: "dd32a35972b347b89ceaaad2455aef5a" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 39781 } } peers { permanent_uuid: "789dbb4a27204ffbb68d7ae40f8f5e4a" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 41825 } }
I20260130 03:03:11.662045 26538 leader_election.cc:290] T 5cc07f1d50624e03b1f5498b12ea680b P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dd32a35972b347b89ceaaad2455aef5a (127.18.100.66:39781), 789dbb4a27204ffbb68d7ae40f8f5e4a (127.18.100.67:41825)
W20260130 03:03:11.663988 22179 leader_election.cc:343] T 5cc07f1d50624e03b1f5498b12ea680b P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 789dbb4a27204ffbb68d7ae40f8f5e4a (127.18.100.67:41825): Not found: Tablet not found: 5cc07f1d50624e03b1f5498b12ea680b
W20260130 03:03:11.664389 22180 leader_election.cc:343] T 5cc07f1d50624e03b1f5498b12ea680b P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dd32a35972b347b89ceaaad2455aef5a (127.18.100.66:39781): Not found: Tablet not found: 5cc07f1d50624e03b1f5498b12ea680b
I20260130 03:03:11.664673 22180 leader_election.cc:304] T 5cc07f1d50624e03b1f5498b12ea680b P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 111215cc989f4203a6e1517f2291d66f; no voters: 789dbb4a27204ffbb68d7ae40f8f5e4a, dd32a35972b347b89ceaaad2455aef5a
I20260130 03:03:11.665320 26538 raft_consensus.cc:2749] T 5cc07f1d50624e03b1f5498b12ea680b P 111215cc989f4203a6e1517f2291d66f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260130 03:03:13.110280 26559 raft_consensus.cc:493] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 342be83f20a648fd849c7d23454fba08)
I20260130 03:03:13.110754 26559 raft_consensus.cc:515] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22dedcf69e00438999d3a5fedde57324" member_type: VOTER last_known_addr { host: "127.18.100.126" port: 42077 } } peers { permanent_uuid: "b01057b6c5ff4fc4b07253801881fe13" member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39157 } } peers { permanent_uuid: "342be83f20a648fd849c7d23454fba08" member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34987 } }
I20260130 03:03:13.112423 26559 leader_election.cc:290] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers b01057b6c5ff4fc4b07253801881fe13 (127.18.100.125:39157), 342be83f20a648fd849c7d23454fba08 (127.18.100.124:34987)
W20260130 03:03:13.117446 22970 leader_election.cc:336] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer b01057b6c5ff4fc4b07253801881fe13 (127.18.100.125:39157): Network error: Client connection negotiation failed: client connection to 127.18.100.125:39157: connect: Connection refused (error 111)
W20260130 03:03:13.117915 22968 leader_election.cc:336] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 342be83f20a648fd849c7d23454fba08 (127.18.100.124:34987): Network error: Client connection negotiation failed: client connection to 127.18.100.124:34987: connect: Connection refused (error 111)
I20260130 03:03:13.118227 22968 leader_election.cc:304] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22dedcf69e00438999d3a5fedde57324; no voters: 342be83f20a648fd849c7d23454fba08, b01057b6c5ff4fc4b07253801881fe13
I20260130 03:03:13.118819 26559 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260130 03:03:13.303831 22923 master_runner.cc:186] Error getting master registration for 127.18.100.125:39157: Network error: Client connection negotiation failed: client connection to 127.18.100.125:39157: connect: Connection refused (error 111),
I20260130 03:03:13.304291 22923 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260130 03:03:13.901157 22179 consensus_peers.cc:597] T 476f8fd240bc48f1a9c47a0351ad5d27 P 111215cc989f4203a6e1517f2291d66f -> Peer 789dbb4a27204ffbb68d7ae40f8f5e4a (127.18.100.67:41825): Couldn't send request to peer 789dbb4a27204ffbb68d7ae40f8f5e4a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 476f8fd240bc48f1a9c47a0351ad5d27. This is attempt 121: this message will repeat every 5th retry.
I20260130 03:03:14.204708 26576 raft_consensus.cc:493] T 563d05e06b1f48b4be140219da5c9dbd P 111215cc989f4203a6e1517f2291d66f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 03:03:14.205302 26576 raft_consensus.cc:515] T 563d05e06b1f48b4be140219da5c9dbd P 111215cc989f4203a6e1517f2291d66f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111215cc989f4203a6e1517f2291d66f" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 39751 } } peers { permanent_uuid: "dd32a35972b347b89ceaaad2455aef5a" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 39781 } } peers { permanent_uuid: "789dbb4a27204ffbb68d7ae40f8f5e4a" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 41825 } }
I20260130 03:03:14.207275 26576 leader_election.cc:290] T 563d05e06b1f48b4be140219da5c9dbd P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dd32a35972b347b89ceaaad2455aef5a (127.18.100.66:39781), 789dbb4a27204ffbb68d7ae40f8f5e4a (127.18.100.67:41825)
W20260130 03:03:14.208961 22180 leader_election.cc:343] T 563d05e06b1f48b4be140219da5c9dbd P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dd32a35972b347b89ceaaad2455aef5a (127.18.100.66:39781): Not found: Tablet not found: 563d05e06b1f48b4be140219da5c9dbd
W20260130 03:03:14.209334 22179 leader_election.cc:343] T 563d05e06b1f48b4be140219da5c9dbd P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 789dbb4a27204ffbb68d7ae40f8f5e4a (127.18.100.67:41825): Not found: Tablet not found: 563d05e06b1f48b4be140219da5c9dbd
I20260130 03:03:14.209656 22179 leader_election.cc:304] T 563d05e06b1f48b4be140219da5c9dbd P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 111215cc989f4203a6e1517f2291d66f; no voters: 789dbb4a27204ffbb68d7ae40f8f5e4a, dd32a35972b347b89ceaaad2455aef5a
I20260130 03:03:14.210199 26576 raft_consensus.cc:2749] T 563d05e06b1f48b4be140219da5c9dbd P 111215cc989f4203a6e1517f2291d66f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260130 03:03:18.103472 26640 raft_consensus.cc:493] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 342be83f20a648fd849c7d23454fba08)
I20260130 03:03:18.104144 26640 raft_consensus.cc:515] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22dedcf69e00438999d3a5fedde57324" member_type: VOTER last_known_addr { host: "127.18.100.126" port: 42077 } } peers { permanent_uuid: "b01057b6c5ff4fc4b07253801881fe13" member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39157 } } peers { permanent_uuid: "342be83f20a648fd849c7d23454fba08" member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34987 } }
I20260130 03:03:18.106094 26640 leader_election.cc:290] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers b01057b6c5ff4fc4b07253801881fe13 (127.18.100.125:39157), 342be83f20a648fd849c7d23454fba08 (127.18.100.124:34987)
W20260130 03:03:18.108824 22970 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.18.100.125:39157: connect: Connection refused (error 111) [suppressed 4 similar messages]
W20260130 03:03:18.112138 22968 leader_election.cc:336] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 342be83f20a648fd849c7d23454fba08 (127.18.100.124:34987): Network error: Client connection negotiation failed: client connection to 127.18.100.124:34987: connect: Connection refused (error 111)
W20260130 03:03:18.112598 22970 leader_election.cc:336] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer b01057b6c5ff4fc4b07253801881fe13 (127.18.100.125:39157): Network error: Client connection negotiation failed: client connection to 127.18.100.125:39157: connect: Connection refused (error 111)
I20260130 03:03:18.112937 22970 leader_election.cc:304] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22dedcf69e00438999d3a5fedde57324; no voters: 342be83f20a648fd849c7d23454fba08, b01057b6c5ff4fc4b07253801881fe13
I20260130 03:03:18.113521 26640 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260130 03:03:18.309762 22923 master_runner.cc:186] Error getting master registration for 127.18.100.125:39157: Network error: Client connection negotiation failed: client connection to 127.18.100.125:39157: connect: Connection refused (error 111),
I20260130 03:03:18.310312 22923 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260130 03:03:18.603343 26654 raft_consensus.cc:493] T be72107f1e024d5788f5d2459c85015c P 789dbb4a27204ffbb68d7ae40f8f5e4a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 03:03:18.604172 26654 raft_consensus.cc:515] T be72107f1e024d5788f5d2459c85015c P 789dbb4a27204ffbb68d7ae40f8f5e4a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "789dbb4a27204ffbb68d7ae40f8f5e4a" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 41825 } } peers { permanent_uuid: "dd32a35972b347b89ceaaad2455aef5a" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 39781 } } peers { permanent_uuid: "111215cc989f4203a6e1517f2291d66f" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 39751 } }
I20260130 03:03:18.606207 26654 leader_election.cc:290] T be72107f1e024d5788f5d2459c85015c P 789dbb4a27204ffbb68d7ae40f8f5e4a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dd32a35972b347b89ceaaad2455aef5a (127.18.100.66:39781), 111215cc989f4203a6e1517f2291d66f (127.18.100.65:39751)
W20260130 03:03:18.608184 22469 leader_election.cc:343] T be72107f1e024d5788f5d2459c85015c P 789dbb4a27204ffbb68d7ae40f8f5e4a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dd32a35972b347b89ceaaad2455aef5a (127.18.100.66:39781): Not found: Tablet not found: be72107f1e024d5788f5d2459c85015c
W20260130 03:03:18.608698 22468 leader_election.cc:343] T be72107f1e024d5788f5d2459c85015c P 789dbb4a27204ffbb68d7ae40f8f5e4a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 111215cc989f4203a6e1517f2291d66f (127.18.100.65:39751): Not found: Tablet not found: be72107f1e024d5788f5d2459c85015c
I20260130 03:03:18.609057 22468 leader_election.cc:304] T be72107f1e024d5788f5d2459c85015c P 789dbb4a27204ffbb68d7ae40f8f5e4a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 789dbb4a27204ffbb68d7ae40f8f5e4a; no voters: 111215cc989f4203a6e1517f2291d66f, dd32a35972b347b89ceaaad2455aef5a
I20260130 03:03:18.609733 26654 raft_consensus.cc:2749] T be72107f1e024d5788f5d2459c85015c P 789dbb4a27204ffbb68d7ae40f8f5e4a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260130 03:03:18.986514 22321 consensus_peers.cc:597] T ce91a17790324cd884a0521ce67dde67 P dd32a35972b347b89ceaaad2455aef5a -> Peer 789dbb4a27204ffbb68d7ae40f8f5e4a (127.18.100.67:41825): Couldn't send request to peer 789dbb4a27204ffbb68d7ae40f8f5e4a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ce91a17790324cd884a0521ce67dde67. This is attempt 121: this message will repeat every 5th retry.
I20260130 03:03:23.316128 22923 master_runner.cc:186] Error getting master registration for 127.18.100.125:39157: Network error: Client connection negotiation failed: client connection to 127.18.100.125:39157: connect: Connection refused (error 111),
I20260130 03:03:23.316624 22923 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260130 03:03:23.426288 22179 consensus_peers.cc:597] T 476f8fd240bc48f1a9c47a0351ad5d27 P 111215cc989f4203a6e1517f2291d66f -> Peer 789dbb4a27204ffbb68d7ae40f8f5e4a (127.18.100.67:41825): Couldn't send request to peer 789dbb4a27204ffbb68d7ae40f8f5e4a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 476f8fd240bc48f1a9c47a0351ad5d27. This is attempt 126: this message will repeat every 5th retry.
W20260130 03:03:26.308315 22433 debug-util.cc:398] Leaking SignalData structure 0x7b08000cc880 after lost signal to thread 22304
W20260130 03:03:26.309274 22433 debug-util.cc:398] Leaking SignalData structure 0x7b08000d0b00 after lost signal to thread 22436
I20260130 03:03:26.631805 26787 raft_consensus.cc:493] T 5cc07f1d50624e03b1f5498b12ea680b P 111215cc989f4203a6e1517f2291d66f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 03:03:26.632537 26787 raft_consensus.cc:515] T 5cc07f1d50624e03b1f5498b12ea680b P 111215cc989f4203a6e1517f2291d66f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111215cc989f4203a6e1517f2291d66f" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 39751 } } peers { permanent_uuid: "dd32a35972b347b89ceaaad2455aef5a" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 39781 } } peers { permanent_uuid: "789dbb4a27204ffbb68d7ae40f8f5e4a" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 41825 } }
I20260130 03:03:26.635643 26787 leader_election.cc:290] T 5cc07f1d50624e03b1f5498b12ea680b P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dd32a35972b347b89ceaaad2455aef5a (127.18.100.66:39781), 789dbb4a27204ffbb68d7ae40f8f5e4a (127.18.100.67:41825)
W20260130 03:03:26.637529 22179 leader_election.cc:343] T 5cc07f1d50624e03b1f5498b12ea680b P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 789dbb4a27204ffbb68d7ae40f8f5e4a (127.18.100.67:41825): Not found: Tablet not found: 5cc07f1d50624e03b1f5498b12ea680b
W20260130 03:03:26.638219 22180 leader_election.cc:343] T 5cc07f1d50624e03b1f5498b12ea680b P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dd32a35972b347b89ceaaad2455aef5a (127.18.100.66:39781): Not found: Tablet not found: 5cc07f1d50624e03b1f5498b12ea680b
I20260130 03:03:26.638543 22180 leader_election.cc:304] T 5cc07f1d50624e03b1f5498b12ea680b P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 111215cc989f4203a6e1517f2291d66f; no voters: 789dbb4a27204ffbb68d7ae40f8f5e4a, dd32a35972b347b89ceaaad2455aef5a
I20260130 03:03:26.639221 26787 raft_consensus.cc:2749] T 5cc07f1d50624e03b1f5498b12ea680b P 111215cc989f4203a6e1517f2291d66f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260130 03:03:28.324311 22923 master_runner.cc:186] Error getting master registration for 127.18.100.125:39157: Network error: Client connection negotiation failed: client connection to 127.18.100.125:39157: connect: Connection refused (error 111),
I20260130 03:03:28.324867 22923 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260130 03:03:28.517531 22321 consensus_peers.cc:597] T ce91a17790324cd884a0521ce67dde67 P dd32a35972b347b89ceaaad2455aef5a -> Peer 789dbb4a27204ffbb68d7ae40f8f5e4a (127.18.100.67:41825): Couldn't send request to peer 789dbb4a27204ffbb68d7ae40f8f5e4a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ce91a17790324cd884a0521ce67dde67. This is attempt 126: this message will repeat every 5th retry.
I20260130 03:03:31.612010 26854 raft_consensus.cc:493] T be72107f1e024d5788f5d2459c85015c P 789dbb4a27204ffbb68d7ae40f8f5e4a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 03:03:31.612644 26854 raft_consensus.cc:515] T be72107f1e024d5788f5d2459c85015c P 789dbb4a27204ffbb68d7ae40f8f5e4a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "789dbb4a27204ffbb68d7ae40f8f5e4a" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 41825 } } peers { permanent_uuid: "dd32a35972b347b89ceaaad2455aef5a" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 39781 } } peers { permanent_uuid: "111215cc989f4203a6e1517f2291d66f" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 39751 } }
I20260130 03:03:31.615280 26854 leader_election.cc:290] T be72107f1e024d5788f5d2459c85015c P 789dbb4a27204ffbb68d7ae40f8f5e4a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dd32a35972b347b89ceaaad2455aef5a (127.18.100.66:39781), 111215cc989f4203a6e1517f2291d66f (127.18.100.65:39751)
W20260130 03:03:31.617031 22469 leader_election.cc:343] T be72107f1e024d5788f5d2459c85015c P 789dbb4a27204ffbb68d7ae40f8f5e4a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dd32a35972b347b89ceaaad2455aef5a (127.18.100.66:39781): Not found: Tablet not found: be72107f1e024d5788f5d2459c85015c
W20260130 03:03:31.617642 22468 leader_election.cc:343] T be72107f1e024d5788f5d2459c85015c P 789dbb4a27204ffbb68d7ae40f8f5e4a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 111215cc989f4203a6e1517f2291d66f (127.18.100.65:39751): Not found: Tablet not found: be72107f1e024d5788f5d2459c85015c
I20260130 03:03:31.617916 22468 leader_election.cc:304] T be72107f1e024d5788f5d2459c85015c P 789dbb4a27204ffbb68d7ae40f8f5e4a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 789dbb4a27204ffbb68d7ae40f8f5e4a; no voters: 111215cc989f4203a6e1517f2291d66f, dd32a35972b347b89ceaaad2455aef5a
I20260130 03:03:31.618508 26854 raft_consensus.cc:2749] T be72107f1e024d5788f5d2459c85015c P 789dbb4a27204ffbb68d7ae40f8f5e4a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260130 03:03:32.659854 26870 raft_consensus.cc:493] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 342be83f20a648fd849c7d23454fba08)
I20260130 03:03:32.660614 26870 raft_consensus.cc:515] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22dedcf69e00438999d3a5fedde57324" member_type: VOTER last_known_addr { host: "127.18.100.126" port: 42077 } } peers { permanent_uuid: "b01057b6c5ff4fc4b07253801881fe13" member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39157 } } peers { permanent_uuid: "342be83f20a648fd849c7d23454fba08" member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34987 } }
I20260130 03:03:32.662387 26870 leader_election.cc:290] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers b01057b6c5ff4fc4b07253801881fe13 (127.18.100.125:39157), 342be83f20a648fd849c7d23454fba08 (127.18.100.124:34987)
W20260130 03:03:32.665095 22970 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.18.100.125:39157: connect: Connection refused (error 111)
W20260130 03:03:32.668998 22970 leader_election.cc:336] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer b01057b6c5ff4fc4b07253801881fe13 (127.18.100.125:39157): Network error: Client connection negotiation failed: client connection to 127.18.100.125:39157: connect: Connection refused (error 111)
W20260130 03:03:32.669586 22968 leader_election.cc:336] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 342be83f20a648fd849c7d23454fba08 (127.18.100.124:34987): Network error: Client connection negotiation failed: client connection to 127.18.100.124:34987: connect: Connection refused (error 111)
I20260130 03:03:32.669952 22968 leader_election.cc:304] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22dedcf69e00438999d3a5fedde57324; no voters: 342be83f20a648fd849c7d23454fba08, b01057b6c5ff4fc4b07253801881fe13
I20260130 03:03:32.670606 26870 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260130 03:03:33.331063 22923 master_runner.cc:186] Error getting master registration for 127.18.100.125:39157: Network error: Client connection negotiation failed: client connection to 127.18.100.125:39157: connect: Connection refused (error 111),
I20260130 03:03:33.331476 22923 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260130 03:03:33.750401 26894 raft_consensus.cc:493] T 563d05e06b1f48b4be140219da5c9dbd P 111215cc989f4203a6e1517f2291d66f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 03:03:33.750993 26894 raft_consensus.cc:515] T 563d05e06b1f48b4be140219da5c9dbd P 111215cc989f4203a6e1517f2291d66f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111215cc989f4203a6e1517f2291d66f" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 39751 } } peers { permanent_uuid: "dd32a35972b347b89ceaaad2455aef5a" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 39781 } } peers { permanent_uuid: "789dbb4a27204ffbb68d7ae40f8f5e4a" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 41825 } }
I20260130 03:03:33.752791 26894 leader_election.cc:290] T 563d05e06b1f48b4be140219da5c9dbd P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dd32a35972b347b89ceaaad2455aef5a (127.18.100.66:39781), 789dbb4a27204ffbb68d7ae40f8f5e4a (127.18.100.67:41825)
W20260130 03:03:33.754601 22180 leader_election.cc:343] T 563d05e06b1f48b4be140219da5c9dbd P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dd32a35972b347b89ceaaad2455aef5a (127.18.100.66:39781): Not found: Tablet not found: 563d05e06b1f48b4be140219da5c9dbd
W20260130 03:03:33.755321 22179 leader_election.cc:343] T 563d05e06b1f48b4be140219da5c9dbd P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 789dbb4a27204ffbb68d7ae40f8f5e4a (127.18.100.67:41825): Not found: Tablet not found: 563d05e06b1f48b4be140219da5c9dbd
I20260130 03:03:33.755677 22179 leader_election.cc:304] T 563d05e06b1f48b4be140219da5c9dbd P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 111215cc989f4203a6e1517f2291d66f; no voters: 789dbb4a27204ffbb68d7ae40f8f5e4a, dd32a35972b347b89ceaaad2455aef5a
I20260130 03:03:33.756330 26894 raft_consensus.cc:2749] T 563d05e06b1f48b4be140219da5c9dbd P 111215cc989f4203a6e1517f2291d66f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260130 03:03:34.156085 22179 consensus_peers.cc:597] T 476f8fd240bc48f1a9c47a0351ad5d27 P 111215cc989f4203a6e1517f2291d66f -> Peer 789dbb4a27204ffbb68d7ae40f8f5e4a (127.18.100.67:41825): Couldn't send request to peer 789dbb4a27204ffbb68d7ae40f8f5e4a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 476f8fd240bc48f1a9c47a0351ad5d27. This is attempt 131: this message will repeat every 5th retry.
I20260130 03:03:35.897253 26924 raft_consensus.cc:493] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 342be83f20a648fd849c7d23454fba08)
I20260130 03:03:35.897759 26924 raft_consensus.cc:515] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22dedcf69e00438999d3a5fedde57324" member_type: VOTER last_known_addr { host: "127.18.100.126" port: 42077 } } peers { permanent_uuid: "b01057b6c5ff4fc4b07253801881fe13" member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39157 } } peers { permanent_uuid: "342be83f20a648fd849c7d23454fba08" member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34987 } }
I20260130 03:03:35.899410 26924 leader_election.cc:290] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers b01057b6c5ff4fc4b07253801881fe13 (127.18.100.125:39157), 342be83f20a648fd849c7d23454fba08 (127.18.100.124:34987)
W20260130 03:03:35.904496 22970 leader_election.cc:336] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer b01057b6c5ff4fc4b07253801881fe13 (127.18.100.125:39157): Network error: Client connection negotiation failed: client connection to 127.18.100.125:39157: connect: Connection refused (error 111)
W20260130 03:03:35.904958 22968 leader_election.cc:336] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 342be83f20a648fd849c7d23454fba08 (127.18.100.124:34987): Network error: Client connection negotiation failed: client connection to 127.18.100.124:34987: connect: Connection refused (error 111)
I20260130 03:03:35.905262 22968 leader_election.cc:304] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22dedcf69e00438999d3a5fedde57324; no voters: 342be83f20a648fd849c7d23454fba08, b01057b6c5ff4fc4b07253801881fe13
I20260130 03:03:35.905761 26924 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260130 03:03:38.340664 22923 master_runner.cc:186] Error getting master registration for 127.18.100.125:39157: Network error: Client connection negotiation failed: client connection to 127.18.100.125:39157: connect: Connection refused (error 111),
I20260130 03:03:38.341117 22923 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260130 03:03:38.475144 22321 consensus_peers.cc:597] T ce91a17790324cd884a0521ce67dde67 P dd32a35972b347b89ceaaad2455aef5a -> Peer 789dbb4a27204ffbb68d7ae40f8f5e4a (127.18.100.67:41825): Couldn't send request to peer 789dbb4a27204ffbb68d7ae40f8f5e4a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ce91a17790324cd884a0521ce67dde67. This is attempt 131: this message will repeat every 5th retry.
I20260130 03:03:43.353271 22923 master_runner.cc:186] Error getting master registration for 127.18.100.125:39157: Network error: Client connection negotiation failed: client connection to 127.18.100.125:39157: connect: Connection refused (error 111),
I20260130 03:03:43.353636 22923 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260130 03:03:44.148489 22179 consensus_peers.cc:597] T 476f8fd240bc48f1a9c47a0351ad5d27 P 111215cc989f4203a6e1517f2291d66f -> Peer 789dbb4a27204ffbb68d7ae40f8f5e4a (127.18.100.67:41825): Couldn't send request to peer 789dbb4a27204ffbb68d7ae40f8f5e4a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 476f8fd240bc48f1a9c47a0351ad5d27. This is attempt 136: this message will repeat every 5th retry.
I20260130 03:03:44.295616 27056 raft_consensus.cc:493] T 563d05e06b1f48b4be140219da5c9dbd P 111215cc989f4203a6e1517f2291d66f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 03:03:44.296232 27056 raft_consensus.cc:515] T 563d05e06b1f48b4be140219da5c9dbd P 111215cc989f4203a6e1517f2291d66f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111215cc989f4203a6e1517f2291d66f" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 39751 } } peers { permanent_uuid: "dd32a35972b347b89ceaaad2455aef5a" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 39781 } } peers { permanent_uuid: "789dbb4a27204ffbb68d7ae40f8f5e4a" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 41825 } }
I20260130 03:03:44.298893 27056 leader_election.cc:290] T 563d05e06b1f48b4be140219da5c9dbd P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dd32a35972b347b89ceaaad2455aef5a (127.18.100.66:39781), 789dbb4a27204ffbb68d7ae40f8f5e4a (127.18.100.67:41825)
W20260130 03:03:44.300707 22180 leader_election.cc:343] T 563d05e06b1f48b4be140219da5c9dbd P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dd32a35972b347b89ceaaad2455aef5a (127.18.100.66:39781): Not found: Tablet not found: 563d05e06b1f48b4be140219da5c9dbd
W20260130 03:03:44.301173 22179 leader_election.cc:343] T 563d05e06b1f48b4be140219da5c9dbd P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 789dbb4a27204ffbb68d7ae40f8f5e4a (127.18.100.67:41825): Not found: Tablet not found: 563d05e06b1f48b4be140219da5c9dbd
I20260130 03:03:44.301568 22179 leader_election.cc:304] T 563d05e06b1f48b4be140219da5c9dbd P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 111215cc989f4203a6e1517f2291d66f; no voters: 789dbb4a27204ffbb68d7ae40f8f5e4a, dd32a35972b347b89ceaaad2455aef5a
I20260130 03:03:44.302175 27056 raft_consensus.cc:2749] T 563d05e06b1f48b4be140219da5c9dbd P 111215cc989f4203a6e1517f2291d66f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260130 03:03:44.879514 27056 raft_consensus.cc:493] T 5cc07f1d50624e03b1f5498b12ea680b P 111215cc989f4203a6e1517f2291d66f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 03:03:44.880023 27056 raft_consensus.cc:515] T 5cc07f1d50624e03b1f5498b12ea680b P 111215cc989f4203a6e1517f2291d66f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111215cc989f4203a6e1517f2291d66f" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 39751 } } peers { permanent_uuid: "dd32a35972b347b89ceaaad2455aef5a" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 39781 } } peers { permanent_uuid: "789dbb4a27204ffbb68d7ae40f8f5e4a" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 41825 } }
I20260130 03:03:44.881851 27056 leader_election.cc:290] T 5cc07f1d50624e03b1f5498b12ea680b P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dd32a35972b347b89ceaaad2455aef5a (127.18.100.66:39781), 789dbb4a27204ffbb68d7ae40f8f5e4a (127.18.100.67:41825)
W20260130 03:03:44.883293 22179 leader_election.cc:343] T 5cc07f1d50624e03b1f5498b12ea680b P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 789dbb4a27204ffbb68d7ae40f8f5e4a (127.18.100.67:41825): Not found: Tablet not found: 5cc07f1d50624e03b1f5498b12ea680b
W20260130 03:03:44.883764 22180 leader_election.cc:343] T 5cc07f1d50624e03b1f5498b12ea680b P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dd32a35972b347b89ceaaad2455aef5a (127.18.100.66:39781): Not found: Tablet not found: 5cc07f1d50624e03b1f5498b12ea680b
I20260130 03:03:44.884083 22180 leader_election.cc:304] T 5cc07f1d50624e03b1f5498b12ea680b P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 111215cc989f4203a6e1517f2291d66f; no voters: 789dbb4a27204ffbb68d7ae40f8f5e4a, dd32a35972b347b89ceaaad2455aef5a
I20260130 03:03:44.884788 27056 raft_consensus.cc:2749] T 5cc07f1d50624e03b1f5498b12ea680b P 111215cc989f4203a6e1517f2291d66f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260130 03:03:47.174858 27098 raft_consensus.cc:493] T be72107f1e024d5788f5d2459c85015c P 789dbb4a27204ffbb68d7ae40f8f5e4a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 03:03:47.175428 27098 raft_consensus.cc:515] T be72107f1e024d5788f5d2459c85015c P 789dbb4a27204ffbb68d7ae40f8f5e4a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "789dbb4a27204ffbb68d7ae40f8f5e4a" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 41825 } } peers { permanent_uuid: "dd32a35972b347b89ceaaad2455aef5a" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 39781 } } peers { permanent_uuid: "111215cc989f4203a6e1517f2291d66f" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 39751 } }
I20260130 03:03:47.177543 27098 leader_election.cc:290] T be72107f1e024d5788f5d2459c85015c P 789dbb4a27204ffbb68d7ae40f8f5e4a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dd32a35972b347b89ceaaad2455aef5a (127.18.100.66:39781), 111215cc989f4203a6e1517f2291d66f (127.18.100.65:39751)
W20260130 03:03:47.179780 22468 leader_election.cc:343] T be72107f1e024d5788f5d2459c85015c P 789dbb4a27204ffbb68d7ae40f8f5e4a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 111215cc989f4203a6e1517f2291d66f (127.18.100.65:39751): Not found: Tablet not found: be72107f1e024d5788f5d2459c85015c
W20260130 03:03:47.180298 22469 leader_election.cc:343] T be72107f1e024d5788f5d2459c85015c P 789dbb4a27204ffbb68d7ae40f8f5e4a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dd32a35972b347b89ceaaad2455aef5a (127.18.100.66:39781): Not found: Tablet not found: be72107f1e024d5788f5d2459c85015c
I20260130 03:03:47.180573 22469 leader_election.cc:304] T be72107f1e024d5788f5d2459c85015c P 789dbb4a27204ffbb68d7ae40f8f5e4a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 789dbb4a27204ffbb68d7ae40f8f5e4a; no voters: 111215cc989f4203a6e1517f2291d66f, dd32a35972b347b89ceaaad2455aef5a
I20260130 03:03:47.181353 27098 raft_consensus.cc:2749] T be72107f1e024d5788f5d2459c85015c P 789dbb4a27204ffbb68d7ae40f8f5e4a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260130 03:03:48.359535 22923 master_runner.cc:186] Error getting master registration for 127.18.100.125:39157: Network error: Client connection negotiation failed: client connection to 127.18.100.125:39157: connect: Connection refused (error 111),
I20260130 03:03:48.359956 22923 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260130 03:03:48.919744 22321 consensus_peers.cc:597] T ce91a17790324cd884a0521ce67dde67 P dd32a35972b347b89ceaaad2455aef5a -> Peer 789dbb4a27204ffbb68d7ae40f8f5e4a (127.18.100.67:41825): Couldn't send request to peer 789dbb4a27204ffbb68d7ae40f8f5e4a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ce91a17790324cd884a0521ce67dde67. This is attempt 136: this message will repeat every 5th retry.
I20260130 03:03:53.366808 22923 master_runner.cc:186] Error getting master registration for 127.18.100.125:39157: Network error: Client connection negotiation failed: client connection to 127.18.100.125:39157: connect: Connection refused (error 111),
I20260130 03:03:53.367173 22923 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260130 03:03:53.390941 22179 consensus_peers.cc:597] T 476f8fd240bc48f1a9c47a0351ad5d27 P 111215cc989f4203a6e1517f2291d66f -> Peer 789dbb4a27204ffbb68d7ae40f8f5e4a (127.18.100.67:41825): Couldn't send request to peer 789dbb4a27204ffbb68d7ae40f8f5e4a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 476f8fd240bc48f1a9c47a0351ad5d27. This is attempt 141: this message will repeat every 5th retry.
I20260130 03:03:53.900707 27202 raft_consensus.cc:493] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 342be83f20a648fd849c7d23454fba08)
I20260130 03:03:53.901224 27202 raft_consensus.cc:515] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22dedcf69e00438999d3a5fedde57324" member_type: VOTER last_known_addr { host: "127.18.100.126" port: 42077 } } peers { permanent_uuid: "b01057b6c5ff4fc4b07253801881fe13" member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39157 } } peers { permanent_uuid: "342be83f20a648fd849c7d23454fba08" member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34987 } }
I20260130 03:03:53.910753 27202 leader_election.cc:290] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers b01057b6c5ff4fc4b07253801881fe13 (127.18.100.125:39157), 342be83f20a648fd849c7d23454fba08 (127.18.100.124:34987)
W20260130 03:03:53.916401 22968 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.18.100.124:34987: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20260130 03:03:53.920802 22970 leader_election.cc:336] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer b01057b6c5ff4fc4b07253801881fe13 (127.18.100.125:39157): Network error: Client connection negotiation failed: client connection to 127.18.100.125:39157: connect: Connection refused (error 111)
W20260130 03:03:53.921316 22968 leader_election.cc:336] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 342be83f20a648fd849c7d23454fba08 (127.18.100.124:34987): Network error: Client connection negotiation failed: client connection to 127.18.100.124:34987: connect: Connection refused (error 111)
I20260130 03:03:53.921670 22968 leader_election.cc:304] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22dedcf69e00438999d3a5fedde57324; no voters: 342be83f20a648fd849c7d23454fba08, b01057b6c5ff4fc4b07253801881fe13
I20260130 03:03:53.922294 27202 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260130 03:03:54.610860 27216 raft_consensus.cc:493] T 563d05e06b1f48b4be140219da5c9dbd P 111215cc989f4203a6e1517f2291d66f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 03:03:54.611343 27216 raft_consensus.cc:515] T 563d05e06b1f48b4be140219da5c9dbd P 111215cc989f4203a6e1517f2291d66f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111215cc989f4203a6e1517f2291d66f" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 39751 } } peers { permanent_uuid: "dd32a35972b347b89ceaaad2455aef5a" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 39781 } } peers { permanent_uuid: "789dbb4a27204ffbb68d7ae40f8f5e4a" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 41825 } }
I20260130 03:03:54.613200 27216 leader_election.cc:290] T 563d05e06b1f48b4be140219da5c9dbd P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dd32a35972b347b89ceaaad2455aef5a (127.18.100.66:39781), 789dbb4a27204ffbb68d7ae40f8f5e4a (127.18.100.67:41825)
W20260130 03:03:54.614957 22180 leader_election.cc:343] T 563d05e06b1f48b4be140219da5c9dbd P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dd32a35972b347b89ceaaad2455aef5a (127.18.100.66:39781): Not found: Tablet not found: 563d05e06b1f48b4be140219da5c9dbd
W20260130 03:03:54.615299 22179 leader_election.cc:343] T 563d05e06b1f48b4be140219da5c9dbd P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 789dbb4a27204ffbb68d7ae40f8f5e4a (127.18.100.67:41825): Not found: Tablet not found: 563d05e06b1f48b4be140219da5c9dbd
I20260130 03:03:54.615720 22179 leader_election.cc:304] T 563d05e06b1f48b4be140219da5c9dbd P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 111215cc989f4203a6e1517f2291d66f; no voters: 789dbb4a27204ffbb68d7ae40f8f5e4a, dd32a35972b347b89ceaaad2455aef5a
I20260130 03:03:54.616348 27216 raft_consensus.cc:2749] T 563d05e06b1f48b4be140219da5c9dbd P 111215cc989f4203a6e1517f2291d66f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260130 03:03:56.493547 23020 debug-util.cc:398] Leaking SignalData structure 0x7b08000b20c0 after lost signal to thread 22933
W20260130 03:03:56.494319 23020 debug-util.cc:398] Leaking SignalData structure 0x7b0800074f40 after lost signal to thread 23023
I20260130 03:03:56.970709 27245 raft_consensus.cc:493] T 5cc07f1d50624e03b1f5498b12ea680b P 111215cc989f4203a6e1517f2291d66f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 03:03:56.971236 27245 raft_consensus.cc:515] T 5cc07f1d50624e03b1f5498b12ea680b P 111215cc989f4203a6e1517f2291d66f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111215cc989f4203a6e1517f2291d66f" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 39751 } } peers { permanent_uuid: "dd32a35972b347b89ceaaad2455aef5a" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 39781 } } peers { permanent_uuid: "789dbb4a27204ffbb68d7ae40f8f5e4a" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 41825 } }
I20260130 03:03:56.973006 27245 leader_election.cc:290] T 5cc07f1d50624e03b1f5498b12ea680b P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dd32a35972b347b89ceaaad2455aef5a (127.18.100.66:39781), 789dbb4a27204ffbb68d7ae40f8f5e4a (127.18.100.67:41825)
W20260130 03:03:56.974660 22180 leader_election.cc:343] T 5cc07f1d50624e03b1f5498b12ea680b P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dd32a35972b347b89ceaaad2455aef5a (127.18.100.66:39781): Not found: Tablet not found: 5cc07f1d50624e03b1f5498b12ea680b
W20260130 03:03:56.975186 22179 leader_election.cc:343] T 5cc07f1d50624e03b1f5498b12ea680b P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 789dbb4a27204ffbb68d7ae40f8f5e4a (127.18.100.67:41825): Not found: Tablet not found: 5cc07f1d50624e03b1f5498b12ea680b
I20260130 03:03:56.975498 22179 leader_election.cc:304] T 5cc07f1d50624e03b1f5498b12ea680b P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 111215cc989f4203a6e1517f2291d66f; no voters: 789dbb4a27204ffbb68d7ae40f8f5e4a, dd32a35972b347b89ceaaad2455aef5a
I20260130 03:03:56.976121 27245 raft_consensus.cc:2749] T 5cc07f1d50624e03b1f5498b12ea680b P 111215cc989f4203a6e1517f2291d66f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260130 03:03:58.368546 22321 consensus_peers.cc:597] T ce91a17790324cd884a0521ce67dde67 P dd32a35972b347b89ceaaad2455aef5a -> Peer 789dbb4a27204ffbb68d7ae40f8f5e4a (127.18.100.67:41825): Couldn't send request to peer 789dbb4a27204ffbb68d7ae40f8f5e4a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ce91a17790324cd884a0521ce67dde67. This is attempt 141: this message will repeat every 5th retry.
I20260130 03:03:58.396627 22923 master_runner.cc:186] Error getting master registration for 127.18.100.125:39157: Network error: Client connection negotiation failed: client connection to 127.18.100.125:39157: connect: Connection refused (error 111),
I20260130 03:03:58.397209 22923 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260130 03:03:58.551162 27273 raft_consensus.cc:493] T 563d05e06b1f48b4be140219da5c9dbd P 111215cc989f4203a6e1517f2291d66f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 03:03:58.551739 27273 raft_consensus.cc:515] T 563d05e06b1f48b4be140219da5c9dbd P 111215cc989f4203a6e1517f2291d66f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111215cc989f4203a6e1517f2291d66f" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 39751 } } peers { permanent_uuid: "dd32a35972b347b89ceaaad2455aef5a" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 39781 } } peers { permanent_uuid: "789dbb4a27204ffbb68d7ae40f8f5e4a" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 41825 } }
I20260130 03:03:58.553993 27273 leader_election.cc:290] T 563d05e06b1f48b4be140219da5c9dbd P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dd32a35972b347b89ceaaad2455aef5a (127.18.100.66:39781), 789dbb4a27204ffbb68d7ae40f8f5e4a (127.18.100.67:41825)
W20260130 03:03:58.556336 22179 leader_election.cc:343] T 563d05e06b1f48b4be140219da5c9dbd P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 789dbb4a27204ffbb68d7ae40f8f5e4a (127.18.100.67:41825): Not found: Tablet not found: 563d05e06b1f48b4be140219da5c9dbd
W20260130 03:03:58.556762 22180 leader_election.cc:343] T 563d05e06b1f48b4be140219da5c9dbd P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dd32a35972b347b89ceaaad2455aef5a (127.18.100.66:39781): Not found: Tablet not found: 563d05e06b1f48b4be140219da5c9dbd
I20260130 03:03:58.557157 22180 leader_election.cc:304] T 563d05e06b1f48b4be140219da5c9dbd P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 111215cc989f4203a6e1517f2291d66f; no voters: 789dbb4a27204ffbb68d7ae40f8f5e4a, dd32a35972b347b89ceaaad2455aef5a
I20260130 03:03:58.557889 27273 raft_consensus.cc:2749] T 563d05e06b1f48b4be140219da5c9dbd P 111215cc989f4203a6e1517f2291d66f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260130 03:04:03.197636 27342 raft_consensus.cc:493] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 342be83f20a648fd849c7d23454fba08)
I20260130 03:04:03.198295 27342 raft_consensus.cc:515] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22dedcf69e00438999d3a5fedde57324" member_type: VOTER last_known_addr { host: "127.18.100.126" port: 42077 } } peers { permanent_uuid: "b01057b6c5ff4fc4b07253801881fe13" member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39157 } } peers { permanent_uuid: "342be83f20a648fd849c7d23454fba08" member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34987 } }
I20260130 03:04:03.200070 27342 leader_election.cc:290] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers b01057b6c5ff4fc4b07253801881fe13 (127.18.100.125:39157), 342be83f20a648fd849c7d23454fba08 (127.18.100.124:34987)
W20260130 03:04:03.202119 22970 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.18.100.125:39157: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20260130 03:04:03.205583 22970 leader_election.cc:336] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer b01057b6c5ff4fc4b07253801881fe13 (127.18.100.125:39157): Network error: Client connection negotiation failed: client connection to 127.18.100.125:39157: connect: Connection refused (error 111)
W20260130 03:04:03.206231 22968 leader_election.cc:336] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 342be83f20a648fd849c7d23454fba08 (127.18.100.124:34987): Network error: Client connection negotiation failed: client connection to 127.18.100.124:34987: connect: Connection refused (error 111)
I20260130 03:04:03.206516 22968 leader_election.cc:304] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22dedcf69e00438999d3a5fedde57324; no voters: 342be83f20a648fd849c7d23454fba08, b01057b6c5ff4fc4b07253801881fe13
I20260130 03:04:03.207150 27342 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260130 03:04:03.397984 22179 consensus_peers.cc:597] T 476f8fd240bc48f1a9c47a0351ad5d27 P 111215cc989f4203a6e1517f2291d66f -> Peer 789dbb4a27204ffbb68d7ae40f8f5e4a (127.18.100.67:41825): Couldn't send request to peer 789dbb4a27204ffbb68d7ae40f8f5e4a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 476f8fd240bc48f1a9c47a0351ad5d27. This is attempt 146: this message will repeat every 5th retry.
I20260130 03:04:03.403977 22923 master_runner.cc:186] Error getting master registration for 127.18.100.125:39157: Network error: Client connection negotiation failed: client connection to 127.18.100.125:39157: connect: Connection refused (error 111),
I20260130 03:04:03.404422 22923 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260130 03:04:05.334151 22289 debug-util.cc:398] Leaking SignalData structure 0x7b08000b7400 after lost signal to thread 22155
W20260130 03:04:05.335282 22289 debug-util.cc:398] Leaking SignalData structure 0x7b08000d1aa0 after lost signal to thread 22292
I20260130 03:04:05.376387 27376 raft_consensus.cc:493] T 5cc07f1d50624e03b1f5498b12ea680b P 111215cc989f4203a6e1517f2291d66f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 03:04:05.377190 27376 raft_consensus.cc:515] T 5cc07f1d50624e03b1f5498b12ea680b P 111215cc989f4203a6e1517f2291d66f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111215cc989f4203a6e1517f2291d66f" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 39751 } } peers { permanent_uuid: "dd32a35972b347b89ceaaad2455aef5a" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 39781 } } peers { permanent_uuid: "789dbb4a27204ffbb68d7ae40f8f5e4a" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 41825 } }
I20260130 03:04:05.380582 27376 leader_election.cc:290] T 5cc07f1d50624e03b1f5498b12ea680b P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dd32a35972b347b89ceaaad2455aef5a (127.18.100.66:39781), 789dbb4a27204ffbb68d7ae40f8f5e4a (127.18.100.67:41825)
W20260130 03:04:05.382229 22179 leader_election.cc:343] T 5cc07f1d50624e03b1f5498b12ea680b P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 789dbb4a27204ffbb68d7ae40f8f5e4a (127.18.100.67:41825): Not found: Tablet not found: 5cc07f1d50624e03b1f5498b12ea680b
W20260130 03:04:05.382750 22180 leader_election.cc:343] T 5cc07f1d50624e03b1f5498b12ea680b P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dd32a35972b347b89ceaaad2455aef5a (127.18.100.66:39781): Not found: Tablet not found: 5cc07f1d50624e03b1f5498b12ea680b
I20260130 03:04:05.383163 22180 leader_election.cc:304] T 5cc07f1d50624e03b1f5498b12ea680b P 111215cc989f4203a6e1517f2291d66f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 111215cc989f4203a6e1517f2291d66f; no voters: 789dbb4a27204ffbb68d7ae40f8f5e4a, dd32a35972b347b89ceaaad2455aef5a
I20260130 03:04:05.383737 27376 raft_consensus.cc:2749] T 5cc07f1d50624e03b1f5498b12ea680b P 111215cc989f4203a6e1517f2291d66f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260130 03:04:05.583489 27381 raft_consensus.cc:493] T be72107f1e024d5788f5d2459c85015c P 789dbb4a27204ffbb68d7ae40f8f5e4a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260130 03:04:05.584170 27381 raft_consensus.cc:515] T be72107f1e024d5788f5d2459c85015c P 789dbb4a27204ffbb68d7ae40f8f5e4a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "789dbb4a27204ffbb68d7ae40f8f5e4a" member_type: VOTER last_known_addr { host: "127.18.100.67" port: 41825 } } peers { permanent_uuid: "dd32a35972b347b89ceaaad2455aef5a" member_type: VOTER last_known_addr { host: "127.18.100.66" port: 39781 } } peers { permanent_uuid: "111215cc989f4203a6e1517f2291d66f" member_type: VOTER last_known_addr { host: "127.18.100.65" port: 39751 } }
I20260130 03:04:05.586879 27381 leader_election.cc:290] T be72107f1e024d5788f5d2459c85015c P 789dbb4a27204ffbb68d7ae40f8f5e4a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dd32a35972b347b89ceaaad2455aef5a (127.18.100.66:39781), 111215cc989f4203a6e1517f2291d66f (127.18.100.65:39751)
W20260130 03:04:05.588568 22469 leader_election.cc:343] T be72107f1e024d5788f5d2459c85015c P 789dbb4a27204ffbb68d7ae40f8f5e4a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dd32a35972b347b89ceaaad2455aef5a (127.18.100.66:39781): Not found: Tablet not found: be72107f1e024d5788f5d2459c85015c
W20260130 03:04:05.589110 22468 leader_election.cc:343] T be72107f1e024d5788f5d2459c85015c P 789dbb4a27204ffbb68d7ae40f8f5e4a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 111215cc989f4203a6e1517f2291d66f (127.18.100.65:39751): Not found: Tablet not found: be72107f1e024d5788f5d2459c85015c
I20260130 03:04:05.589464 22468 leader_election.cc:304] T be72107f1e024d5788f5d2459c85015c P 789dbb4a27204ffbb68d7ae40f8f5e4a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 789dbb4a27204ffbb68d7ae40f8f5e4a; no voters: 111215cc989f4203a6e1517f2291d66f, dd32a35972b347b89ceaaad2455aef5a
I20260130 03:04:05.590067 27381 raft_consensus.cc:2749] T be72107f1e024d5788f5d2459c85015c P 789dbb4a27204ffbb68d7ae40f8f5e4a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260130 03:04:05.848112 27382 raft_consensus.cc:493] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 342be83f20a648fd849c7d23454fba08)
I20260130 03:04:05.848567 27382 raft_consensus.cc:515] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22dedcf69e00438999d3a5fedde57324" member_type: VOTER last_known_addr { host: "127.18.100.126" port: 42077 } } peers { permanent_uuid: "b01057b6c5ff4fc4b07253801881fe13" member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39157 } } peers { permanent_uuid: "342be83f20a648fd849c7d23454fba08" member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34987 } }
I20260130 03:04:05.850257 27382 leader_election.cc:290] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers b01057b6c5ff4fc4b07253801881fe13 (127.18.100.125:39157), 342be83f20a648fd849c7d23454fba08 (127.18.100.124:34987)
W20260130 03:04:05.855379 22970 leader_election.cc:336] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer b01057b6c5ff4fc4b07253801881fe13 (127.18.100.125:39157): Network error: Client connection negotiation failed: client connection to 127.18.100.125:39157: connect: Connection refused (error 111)
W20260130 03:04:05.856082 22968 leader_election.cc:336] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 342be83f20a648fd849c7d23454fba08 (127.18.100.124:34987): Network error: Client connection negotiation failed: client connection to 127.18.100.124:34987: connect: Connection refused (error 111)
I20260130 03:04:05.856745 22968 leader_election.cc:304] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22dedcf69e00438999d3a5fedde57324; no voters: 342be83f20a648fd849c7d23454fba08, b01057b6c5ff4fc4b07253801881fe13
I20260130 03:04:05.857352 27382 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260130 03:04:07.507826 27413 raft_consensus.cc:493] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 342be83f20a648fd849c7d23454fba08)
I20260130 03:04:07.508795 27413 raft_consensus.cc:515] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22dedcf69e00438999d3a5fedde57324" member_type: VOTER last_known_addr { host: "127.18.100.126" port: 42077 } } peers { permanent_uuid: "b01057b6c5ff4fc4b07253801881fe13" member_type: VOTER last_known_addr { host: "127.18.100.125" port: 39157 } } peers { permanent_uuid: "342be83f20a648fd849c7d23454fba08" member_type: VOTER last_known_addr { host: "127.18.100.124" port: 34987 } }
I20260130 03:04:07.518436 27413 leader_election.cc:290] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers b01057b6c5ff4fc4b07253801881fe13 (127.18.100.125:39157), 342be83f20a648fd849c7d23454fba08 (127.18.100.124:34987)
W20260130 03:04:07.537413 22970 leader_election.cc:336] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer b01057b6c5ff4fc4b07253801881fe13 (127.18.100.125:39157): Network error: Client connection negotiation failed: client connection to 127.18.100.125:39157: connect: Connection refused (error 111)
W20260130 03:04:07.538210 22968 leader_election.cc:336] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 342be83f20a648fd849c7d23454fba08 (127.18.100.124:34987): Network error: Client connection negotiation failed: client connection to 127.18.100.124:34987: connect: Connection refused (error 111)
I20260130 03:04:07.538676 22968 leader_election.cc:304] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22dedcf69e00438999d3a5fedde57324; no voters: 342be83f20a648fd849c7d23454fba08, b01057b6c5ff4fc4b07253801881fe13
I20260130 03:04:07.539494 27413 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 22dedcf69e00438999d3a5fedde57324 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260130 03:04:08.410418 22923 master_runner.cc:186] Error getting master registration for 127.18.100.125:39157: Network error: Client connection negotiation failed: client connection to 127.18.100.125:39157: connect: Connection refused (error 111),
I20260130 03:04:08.410921 22923 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260130 03:04:08.566790 22321 consensus_peers.cc:597] T ce91a17790324cd884a0521ce67dde67 P dd32a35972b347b89ceaaad2455aef5a -> Peer 789dbb4a27204ffbb68d7ae40f8f5e4a (127.18.100.67:41825): Couldn't send request to peer 789dbb4a27204ffbb68d7ae40f8f5e4a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ce91a17790324cd884a0521ce67dde67. This is attempt 146: this message will repeat every 5th retry.
W20260130 03:04:10.794452 22438 heartbeater.cc:646] Failed to heartbeat to 127.18.100.124:34987 (725 consecutive failures): Network error: Failed to ping master at 127.18.100.124:34987: Client connection negotiation failed: client connection to 127.18.100.124:34987: connect: Connection refused (error 111)
W20260130 03:04:10.886147 22580 heartbeater.cc:646] Failed to heartbeat to 127.18.100.125:39157 (595 consecutive failures): Network error: Failed to ping master at 127.18.100.125:39157: Client connection negotiation failed: client connection to 127.18.100.125:39157: connect: Connection refused (error 111)
W20260130 03:04:10.927305 22294 heartbeater.cc:646] Failed to heartbeat to 127.18.100.124:34987 (725 consecutive failures): Network error: Failed to ping master at 127.18.100.124:34987: Client connection negotiation failed: client connection to 127.18.100.124:34987: connect: Connection refused (error 111)
W20260130 03:04:11.243050 22592 master_proxy_rpc.cc:177] Failed to determine new leader Master: Timed out: ConnectToClusterRpc(addrs: 127.18.100.126:42077,127.18.100.125:39157,127.18.100.124:34987, num_attempts: 758) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.18.100.124:34987: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260130 03:04:11.243664 22592 master_proxy_rpc.cc:177] Failed to determine new leader Master: Timed out: ConnectToClusterRpc(addrs: 127.18.100.126:42077,127.18.100.125:39157,127.18.100.124:34987, num_attempts: 758) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.18.100.124:34987: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260130 03:04:11.244035 22592 master_proxy_rpc.cc:177] Failed to determine new leader Master: Timed out: ConnectToClusterRpc(addrs: 127.18.100.126:42077,127.18.100.125:39157,127.18.100.124:34987, num_attempts: 758) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.18.100.124:34987: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260130 03:04:11.244282 22592 master_proxy_rpc.cc:177] Failed to determine new leader Master: Timed out: ConnectToClusterRpc(addrs: 127.18.100.126:42077,127.18.100.125:39157,127.18.100.124:34987, num_attempts: 758) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.18.100.124:34987: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
F20260130 03:04:11.259337 22602 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_2dde477c96844e51bc37ab8d6bd13248 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.18.100.126:42077,127.18.100.125:39157,127.18.100.124:34987, num_attempts: 758) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.18.100.124:34987: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
*** Check failure stack trace: ***
W20260130 03:04:11.262406 22592 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.18.100.124:34987)
*** Aborted at 1769742251 (unix time) try "date -d @1769742251" if you are using GNU date ***
F20260130 03:04:11.266278 22603 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_53c1c4a7b44f4bb095c3167a66cf3315 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.18.100.126:42077,127.18.100.125:39157,127.18.100.124:34987, num_attempts: 758) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.18.100.124:34987: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
PC: @ 0x0 (unknown)
*** SIGABRT (@0x3e800004991) received by PID 18833 (TID 0x7fc993dfb700) from PID 18833; stack trace: ***
@ 0x2de460 __tsan::CallUserSignalHandler() at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:1915
@ 0x2e0943 rtl_sigaction() at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:2005
@ 0x7fc9a6f6a980 (unknown) at ??:0
@ 0x7fc9a2117fb7 gsignal at ??:0
*** Check failure stack trace: ***
@ 0x7fc9a2119921 abort at ??:0
@ 0x2e4456 abort at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:263
@ 0x7fc9a51afda4 google::LogMessage::SendToLog() at ??:0
@ 0x7fc9a51b0950 google::LogMessage::Flush() at ??:0
@ 0x7fc9a51b5a8b google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x360594 kudu::MasterStressTest::CreateTableThread() at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/integration-tests/master-stress-test.cc:?
@ 0x38e45e _ZNSt3__18__invokeIMN4kudu16MasterStressTestEFvvEPNS1_26MasterStressTest_Test_TestEJEvEEDTcldsdeclsr3std3__1E7forwardIT0_Efp0_Efp_spclsr3std3__1E7forwardIT1_Efp1_EEEOT_OS7_DpOS8_ at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/type_traits:3846
@ 0x38e311 _ZNSt3__116__thread_executeINS_10unique_ptrINS_15__thread_structENS_14default_deleteIS2_EEEEMN4kudu16MasterStressTestEFvvEJPNS6_26MasterStressTest_Test_TestEEJLm2EEEEvRNS_5tupleIJT_T0_DpT1_EEENS_15__tuple_indicesIJXspT2_EEEE at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/thread:281
@ 0x38da5b std::__1::__thread_proxy<>() at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/thread:?
@ 0x2de320 __tsan_thread_start_func at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:959
@ 0x7fc9a6f5f6db start_thread at ??:0
@ 0x7fc9a21fa71f clone at ??:0